Letting a php program run for hours

I have a massive amount of data that needs to be read from mysql, analyzed and based on the results split up and stored in new records. five record takes about 20 seconds, but the records vary in length so I can't really estimate how long the program will take, however have calculated that the process should not take longer much longer than 5 hours, so I'd like to run it over night and feel quite sure that when I come back to the office the next morning the program is done.

Assuming the code is fail safe (I know right ;) how should set up Apache / PHP /Mysql settings so that when I execute the script so that I can be sure that开发者_如何学JAVA the program will not time out and/or not run out of ram?

(it is basically running in a loop fetching sets of 100 rows until it can't anymore a loop so, I am hoping the fact that the variables are being reset at the beginning of each iteration will keep the memory usage constant.)

The actual size of the database when dumped is 14mb, so the volume of the data is not so high

(on a side note, it might also be that I haven't assigned the maximum resources to the server settings, so maybe that's why it takes 20 seconds to run 5 records)

Make sure you have removed any max_execution_time limits by setting this to 0 (unlimited) in your PHP.ini or by calling set_time_limit(0). This will ensure that PHP doesn't stop the script mid-execution.
If it all possible, you should run the script from the CLI so that you don't have to worry about Apache timing your request out (it shouldn't, but it might).
Since you are working with only 15 MB of data I wouldn't worry about memory usage (128 MB is the default in PHP). If you are really worried you can remove memory limits in PHP by modifying the memory_limit to be either a higher number of -1 (infinite memory).

Keep in mind modifying the PHP.ini will affect all scripts that are interpreted by that installation. I prefer to use the appropriate ini setting functions at the top of my scripts to prevent dangerous global changes.

On a side note: This doesn't really sound like a job for PHP. I'm not trying to discourage your use of PHP here, but there are other languages that are better suited for command line usage.

Better make your script exit the execution, and then restart it. Store the point where it left last time. This will ensure you do not have memory leaks and script does not run out of memory due to some error in garbage collection,and that the execution continues if there is unexpected failure. A simple shell command would be :

while [ 1 ]; do php myPhpScript.php  a; done

you can make other checks to ensure proper running.

I'd like to point out, by default scripts run via a CLI in PHP, default to having no time limit, unlike scripts run through CGI, mod_php etc.

And as stated avoid running this via Apache.

However if you MUST do this, consider breaking it down. You can make a page that could process 5-10 results, appends the dump file, then prints out either a meta refresh, or some JavaScript to reload the page with with a parameter telling it where it's up too, and to continue until done.

Not recommended though.

adding to some of the other good options here you might want to look at http://www.electrictoolbox.com/article/php/process-forking/ and also sending some requests to dev/null if you dont need them to give back feedback.

Don't do this using a web interface. Run it from the command line; but look to see if your code can be optimised, or set break points and do it in "chunks"

First of all http://php.net/manual/en/function.set-time-limit.php put set_time_limit(0); at the beginning of the script.

As for the memory you should take care of that by unsetting any variables, array, pointers that you do not need on each iteration.

Better run the script from the shell (CLI) or as cronjob.

As far as I know MySQL connections do not time out, so you should be safe by setting:

php_value max_execution_time X

in a .htaccess file or placing set_time_limit(X) at the beginning of your script where X is a comfortable value in seconds.