Recommendation for big binary chunk transport protocol based on TCP/IP

I'm planning to develop a kind of video transcoding system.

One machine has frame grabber and will receive audio/video signal from various sources.

Several transcoding machines will be connected to this 开发者_JAVA百科source machine through gigabit ethernet.

The source machine will send zipped audio/video frame data to transcoding machines.

Because this is simple one way transport, I thought I can use HTTP. But network bandwidth is the problem.

With simple search I found a post in the Superuser.

This real world gigabit ethernet example shows only 340Mbps throughput.

My target system should be capable of multiple simultaneous transcoding of Full HD video.

Data rate of 1080P full HD video could be 712Mbps without compress. Even with compress, this could easily saturate gigabit network at only 1 or 2 channels.

Let's say 3 is the target. What protocol do I use to accomplish 3 simultaneous transport of full HD video data? Can I use HTTP? Do I have to devise special purpose multicast protocol? Is there any open source and/or open specification protocol for this purpose?

Thanks in advance.

If you want to use multicast and you can switch to UDP protocol, then I'll suggest you to take in consideration RTP protocol. But you seem not to have streaming purposes and , in addition , you are sending zipped file. HTTP should be ok for this , but if you want to avoid overhead then you could use 3 simple TCP connections without HTTP.

TCP/IP throughput is something that is difficult to quantify without actually trying it under your own conditions. That person might only see 340Mbps of throughput, but the simple way to saturate high bandwidth connections to maximum speed is multiple parallel transfers over TCP... for that purpose HTTP will work as well.

The real question is twofold... first does your video need to arrive at your transcoder within a certain amount of time? If so, what is your timing window? Second, how much bandwidth does a single gzipped stream take up?

Finally, remember you can use LACP to bundle your ethernet connections together with two or more NICs, so your server could pump more data out if you find that you're limited by a single GigE connection. Talk to your network administrator to see if that is a possibility...

EDIT RESPONDING TO DISCUSSION IN COMMENTS:

So you have about 30 milliseconds to send the data for a single frame.... just to help you with budgeting, if that does not include transcoding delays, be sure to subtract those from the 1/30th of a second figure... with that kind of timing, I would keep a constant TCP socket open and just incrementally send the files over the same TCP socket... that will reduce your overhead from TCP establishment and teardown... you might even be able to work something out with plain-ole FTP... just don't close the FTP session until the video program is over... configuring your switch and hosts to use jumbo frames (ethernet MTU over 1522 bytes, including the headers, crc, and vlan tag...) could shave a few more milliseconds off your file transfer numbers, but the administrative overhead of jumbos can be a pain... for instance when someone upgrades the switches or routers, they often forget to check vendor interface support for jumbos... compounding the problem are vendor sales reps who "make assumptions" about their jumbo frame support.