Should I be using message queuing for this?

I have a PHP application that currently has 5k users and will keep increasing for the forseeable future. Once a week I run a script that:

• fetches all the users from the database
• loops through the users, and performs some upkeep for each one (this includes adding new DB records)

The last time this script ran, it only processed 1400 users before dieing due to a 30 second maximum execute time error. One solution I thought of was to have the main script still fetch all the users, but instead of performing the upkeep process itself, it would make an asynchronous cURL call (1 for each user) to a new script that will perform the upkeep for that particular user.

My concern here is that 5k+ cURL calls could bring down the server. Is this something that could be remedied by using a messaging queue instead of cURL calls? I have no experience using one, but from what I've read it seems like this might help. If so, which message queuing system would you recommend?

Some background info:

• this is a Symfony project, using Doctrine as my ORM and MySQL as开发者_StackOverflow中文版 my DB
• the server is a Windows machine, and I'm using Windows' task scheduler and wget to run this script automatically once per week.

Any advice and help is greatly appreciated.

If it's possible, I would make a scheduled task (cron job) that would run more often and use LIMIT 100 (or some other number) to process a limited number of users at a time.

A few ideas:

1. Increase the Script Execution time-limit - set_time_limit()
   Don't go overboard, but more than 30 seconds would be a start.
2. Track Upkeep against Users
   Maybe add a field for each user, last_check and have that field set to the date/time of the last successful "Upkeep" action performed against that user.
3. Process Smaller Batches
   Better to run smaller batches more often. Think of it as being the PHP equivalent of "all of your eggs in more than one basket". With the last_check field above, it would be easy to identify those with the longest period since the last update, and also set a threshold for how often to process them.
4. Run More Often
   Set a cronjob and process, say 100 records every 2 minutes or something like that.
5. Log and Review your Performance
   Have logfiles and record stats. How many records were processed, how long was it since they were last processed, how long did the script take. These metrics will allow you to tweak the batch sizes, cronjob settings, time-limits, etc. to ensure that the maximum checks are performed in a stable fashion.

Setting all this may sound like alot of work compared to a single process, but it will allow you to handle increased user volumes, and would form a strong foundation for any further maintenance tasks you might be looking at down the track.

Why don't you still use the cURL idea, but instead of processing only one user for each, send a bunch of users to one by splitting them into groups of 1000 or something.

Have you considered changing your logic to commit changes as you process each user? It sounds like you may be running a single transaction to process all users, which may not be necessary.

How about just increasing the execution time limit of PHP?

Also, looking into if you can improve your upkeep-procedure to make it faster can help too. Depending on what exactly you are doing, you could also look into spreading it out a bit. Do a couple once in a while rather than everyone at once. But depends on what exactly you're doing of course.