Running some massive task during off-peak period?
Background
In one of our project, we need to run some massive task occasionally, e.g., generate reports, send numbers of notification emails. Sometimes it causes noticeable lag when such massive task is being run. So we are thinking of one possible solution.
Some thoughts
- Set crontab to run a backend script every 10 minutes.
- Collect the cpu usage info, I found http://phpsysinfo.sourceforge.net/phpsysinfo/index.php?disp=dynamic , but I'm not sure if there is a better way?
- If there are contiguous usage lower than a specific value开发者_JAVA技巧, or the first task in the queue reaches its deadline, the script will get a certain number of tasks from the queue and run.
There are different types of massive task: e.g.,
- User can request certain type of report
- Notification emails
- Cleaning data in database
- ...
I am wondering if this idea is worth trying? Is there any problem, or is there some other better solution?
This works up to a point but struggles if you are running anything where access is required 24 hours a day (like an internationally used site).
You may wish to replicate your database and then run your heavy queries off of that - or investigate a form of data warehousing.
What is a data warehouse?
精彩评论