A background web-script for APM server

I'm using Twitter API with PHP, developing sort of tests. Those test are one like "who's your best friend", "what are your frequent keywords", "how often do you tweet" But since it takes forever to get a response from the Twitter API server, the PHP script runs for 30 seconds or more. This is definitely a waste of resource, and I would like to improve this task into the followi开发者_开发知识库ng sequence.

"Run your test" → click → PHP script initiates background script → autorefresh by ajax and keep checking → PHP page checks if the background script has finished running and returned its values (maybe to a file or to the DB)

Would there be a way to make a background script? I was considering Python, but I'm sure you guys could give me a better idea.

I think an ajax interface would be your best bet, the user can fire a script through AJAX that would start the processing in the background. I would then use an ajax-auto-refresh on the page that pings a simple page to see if a job is being processed (to display a waiting message) and if it has finished it can either reload the entire page, or just get the relevant data back from the server for viewing.

Your web server becomes your background thread manager (just make sure it doesn't stop processing if the client moves on) and the clients browser is the mechanism that is pinging for new information every 5 seconds or so. If you point that ajax-auto-refresh to a very simple page then you can keep server load very low over thousands of pinging connections. Should be really easy to make a proof of concept for something like this!

If you don't need partial results, you can use regular AJAX and just keep the connection open until the result is in (this is sometimes called Reverse Ajax). You can implement this in PHP, no need to use Python (although Python is certainly up to the job as well).

More important than the language is a web server that supports a high number of concurrent threads – most web servers work with a limited pool of threads. Longstanding connections use up this pool. Therefore, either use a highly concurrent platform like node.js or simply cache the result and thereby limit the number of concurrent, long-running threads.

Take a look at node.js. It's a server side JavaScript library built on top of googles chrome engine with lots of networking support baked in