Dividing file to be processed partly, then batching results [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.

开发者_Go百科

Closed 4 years ago.

Improve this question

Please,

We have the following situation:

Component X that divides a request file into parts, sending each part to an independent processing component Y -through a network- that will reply with a result to component Z , component Z collects all the results of the file parts into a batch result file.

Note:- Request file: file contains N number of data records that need to be processed.

What is the best practice for this situation? Is there any protocol for that?, Is there any library that can help? design patterns ??

thanx in advance.

---

We use the command pattern to queue up units of work to process very large files.

The commands themselves are stored in a database (command entries are things like "process File X, line 1-100", "process File X, line 101-200", etc.). Any number of servers from a server farm can pick up one command, indicate they are working on it, and write back their result. A controller looks for abandoned work (picked up but no result written within X minutes) and can reset the work to be eligible for pickup again.

---

A message service queue/bus like RabbitMQ might be of help. Using this service, you can connect all the distributed components together & dispatch/collect results in a reliable manner.

Whilst the service bus won't solve all your "problems", it probably would address the distributed & reliable communication bits.