Distributing Real-Time Market Data Using ZeroMQ / NFS?

Suppose that you have a machine that gets fed with real-time stock prices from the exchange. These prices need to b开发者_JS百科e transferred to 50 other machines in your network in the fastest possible way, so that each of them can run its own processing on the data.

What would be the best / fastest way to send the data over to the other 50 machines?

I am looking for a solution that would work on linux, using python as the programming language. Some ideas that I had are:

(1) Send it to the other machines using python's ZeroMQ module

(2) Save the data to a shared folder and have the 50 machines read it using NFS

Any other ideas?

I would go with zeromq with pub/sub sockets..

in your 2 option, your "clients" will have to refresh in order to get your file modifications.. like polling.. if you have some write error, you will have to handle this by hand, which won't be easy as well..

zeromq is simple, reliable and powerful.. i think that perfectly fit your case..

I'm pretty sure sending with ZeroMQ will be substantially quicker than saving and loading files.

There are other ways to send information over the network, such as raw sockets (lower level), AMQP implementations like RabbitMQ (more structured/complicated), HTTP requests/replies, and so on. ZeroMQ is a pretty good option, but it probably depends on your situation.

You could also look at frameworks for distributed computing, such as that in IPython.

Definatly do NOT use the file system. ZeroMQ is a great solution wiht bindings in Py. I have some examples here: www.coastrd.com. Contact me if you need more help.