running really long scripts - how to keep them running and start them again if they fail?

I need to run a bunch of long running processes on a CENTOS server.

If I l开发者_C百科eave the processes (python/php scripts) to run sometimes the processes will stop running because of trivial errors eg. string encoding issues or sometimes because the process seems to get killed by the server.

I try to use nohup and fire the jobs from the crontab

Is there any way to keep these processes running in such a way that all the variables are saved and I can restart the script from where it stopped?

I know I can program this into the code but would prefer a generalised utility which could just keep these things running so that the script completed even if there were trivial errors.

Perhaps I need some sort of process-management tool?

Many thanks for any suggestions

is there any way to keep these processes running in such a way that all the variables are saved and i can restart the script from where it stopped?

Yes. It's called creating a "checkpoint" or "memento".

i know i can program this

Good. Get started. Each problem is unique, so you have to create, save, and reload the mementos.

but would prefer a generalised utility which could just keep these things running so that the script completed even if there were trivial errors.

It doesn't generalize well. Not all variables can be saved. Only you know what's required to restart your process in a meaningful way.

perhaps i need some sort of process-management tool?

Not really.

trivial errors eg. string encoding issues

Usually, we find these by unit testing. That saves a lot of programming to work around the error. An ounce of prevention is worth a pound of silly work-arounds.

sometimes because the process seems to get killed by the server.

What? You'd better find out why. An ounce of prevention is worth a pound of silly work-arounds.