Creating Server Processing Queue

I am not even quite sure how to ask this question as it is sort of an overall conceptual issue but I figured I'd give it a try.

I am trying to create a process conceptually similar to many of the video hosting websites out there like Vimeo. Using Vimeo as an example, the user uploads a video to the main server, the video is added to a processing queue, and the user is free to go do other things while the video is processing. Sometime later, the user receives an email saying that the processing is complete and the video can now be accessed and the user can do whatever they want with it.

My question is this, how does a system like this work? My experience is primarily with PHP where all the processing is tied directly to the code in that page (or via includes). How do you deal with things like this processing while the user navigates away from the page?

This is my guess... Please correct me. The page submitted by the user adds th开发者_运维问答eir file to the processing software queue via some sort of API Hook. The processor which is probably written in some other language like C or Java or whatever, does its thing and then when it is finished, it fires a call to some PHP page that then sends the user an email and does whatever needs to be done with the video.

I guess I am just trying to understand how the "black boxes" of some of the big websites out there work, at least conceptually. Can anyone help me out here? Sorry it is such a big, vague question.

One solution to your question would be to start a new process fork which handles the video processing and mailing the user, and then the main process just continues as usual. This will prevent the main process to stoping and forcing the user to wait for the processing to finish.

You can read about creating process forks in php here: http://php.net/manual/en/book.pcntl.php

Then there are applications which handels video processing such as ffmpeg: http://www.ffmpeg.org/

Instead of having to use a new programming language you just execute a cli application such as ffmpeg using the exec function: http://php.net/manual/en/function.exec.php

You need to figure out a way to que the processes so that the server won't overload. Maybe there needs to be some functionality which checks the database for active processes and waits until the table have less active processes than which is allowed.