Best format/Compression/protocol to transport data from client to server?

We have multiple clients asking for work from server - a list of 10000 urls .= 1MB

and sending back process开发者_JS百科ed data back to server - around 1KB per URL i.e. 10 MB

Each client will do that every 5-10 minutes.

I have requirements of extensibility, high performance and platform independence in that order.

1. Should i use XML or custom binary format for transfer ?

2. Best compression type to use ? gzip, i see .NET folks using ICSHARP library. What are trade-offs ?

3. Should client send data via POST ? or any other better method ?

4. Any other suggestions to keep in mind for such scenario ?

1. I would look at BSON, only because it's getting a lot of attention now, and you should find the serializers well supported. It's also a fairly tight wire protocol, which is why its the focus of a few high-profile nosql dbms's.

2. As far as compression, use whatever your web servers support (probably gzip). The tradeoffs with compression almost always comes down to cpu cycles consumed vs compactness of the compression. But in the end, gzip is a good middle of the road, and all you have to do to use it is turn it on.

3. Yes, POST is the appropriate method.

4. If performance or concurrency become bottlenecks, you can move gzip ( and SSL and authentication and session, and many other things) to a tier in front of your web servers. But my gut is you'll be I/O constrained on the server side. Good luck.

have you considered json? same information as in an xml file but using less text to describe it.