Server time out, take me to school, what is it...

[dutch]

Can someone school me on "server time out" issues.
In the past I had issues running power search...it always timed out. I checked with support, they changed my 'global time out'...power search ran ok
Today I tried importing new items, file was 200k worth of new products, 83mb, constant time out errors. I sent support ticket, they said they changed my global time out again, ran ok.

What is it? Why does it seem to sometimes work after support changes the global time out and other time doesn't work
Thanks

[ILoveHostasaurus]

The issue comes down to having to intentionally defeat security/availability measures to allow things to run to completion because of a software issue with Power Search.

There are two timeouts in play:

• There is a web server connection timeout. This is a counter where the web server software will close a connection to a visitor if the timeout is reached, whether the request has completed or not. This typically shows as an error along the lines of "Something went wrong" or "Server connection reset" displayed by the browser, rather than an error from the server. The purpose of this timeout is to not allow incredibly slow connections to hang around, ultimately exhausting the available connections and denying legitimate traffic. The longer the value, the easier it is for a remote attacker to simply open up as many connections as they can and let them sit there doing nothing while the server happily lets it happen.
  
  This timeout MUST be longer than the Miva Empresa (underlying engine software) global timeout, because you need time for the engine to do whatever it is doing and send back a complete response before the web server connection is closed.
  
  The default value is 65 seconds.
• There is the Miva Empresa 'global timeout'; this is an execution timeout the Miva Empresa engine software employs mostly to stop runaway processes. The default value is 90 seconds, because in most cases, who would ever want to click on a website they were visiting and wait 90+ seconds for a response. If an application, such as Miva Merchant or a module within it, actually needs more than 90 seconds of time on the processor to complete its task, then the default limit would be too low and must be raised, along with the previously mentioned web server timeout.
  
  This is a limit designed to prevent problems because it's unusual for something to need more than that amount of processing time. For example, let's say a remote attacker figures out a request they can send to your site that consumes minutes of processing time; well they just send a bunch of those requests and overload your site, taking it down. If the timeout would have otherwise cut them off, it did its job. So raising it increases risk.

In the case of Power Search, most of its code dates back a decade or more when Bill Weiland originally released it, so it was not written using methods designed to behave nicely when handling large amounts of data. It was written to do all of its work with one HTTP request, from start to finish, and if that doesn't happen, what you were trying to do ends in error. With small stores, that's fine, but with larger stores, it has an awful lot to do and the only choice to make it finish is to keep increasing the timeout until you find the magic number that lets it complete.

Some of the other data processing tasks in Miva Merchant, such as imports/exports using the core software, have been written to use javascript and asynchronous HTTP requests to make little bits of processing happen per HTTP request. So a large import looks like you just click the button once and it chugs on through, but in reality your browser is sending new requests every few seconds to process bits of the import at a time while the progress meter advances, so there is no need for a long timeout. The browser and store keep track of the progress so it can work its way through. We acquired the power search code from Bill before his passing, but I don't know if there are any plans to rewrite it as it would be mostly from the ground up.

I also want to note that in some cases, we don't have a way to get around the timeout issue. For example, customers who use Cloudflare's CDN in front of their website for security or caching of content wouldn't be able to get around this issue. Cloudflare has a fixed 65 second timeout you can't change on their lower plans, so even if we extended things on the server side it would still kill the connection before completion.

[Mark Hood]

I'd be curious to know a couple things... Is your module up to date (i.e. do you have a checkbox that says turbo rebuild? If so does it work? My other question would be to make sure you haven't created any new custom fields that break the rules in the power search instructions.

[dutch]

Very helpful David thank you