Sequential vs multi-threaded Accounting Events Processing

We are working on an event-driven accounting engine and so far we are doing everything in a batch/sequential manner.

Problem is there are thousands of events created per day and processing everything sequentially makes it slow.

Is there a safe way we can implement开发者_如何学Go a multi-thread event processing accounting engine without worrying about financial data integrity and consistency?

Or is it just better to play it safe and allows follow a batch/sequential approach?

Perhaps you might want to look into software transactional memory model. The concept has been discussed in this paper

You already have managed building an accounting engine satisfying your financial data integrity and consistency requirements. With this, it should be feasible to parallelize processing.

You would need

• a new dispatcher component which would be responsible for feeding multiple threads with to-be-processed event data while maintaining consistency (i.e. bill each event exactly once) and
• a new aggregator component for muxing the rated events on the accounts.

Typically the aggregator component will become the bottleneck in your parallel architecture.

Since you explicitly intend to build a multi-thread application: Depending on requirements and environment it might be feasible to build a multi-process design, i.e. have several independent processes running in parallel. You would gain a simpler concurrency model versus the evemtual need for interprocess communication.

Virtually all telecom billing systems scale by multi-threading or multi-processing so this is definitelly a sane way forward.

You can use multiple concurrent transactions if you follow the rules

check out
en.wikipedia.org/wiki/Atomicity_(database_systems)
en.wikipedia.org/wiki/ACID
en.wikipedia.org/wiki/Record_locking

or read all relevant articles in this section
http://en.wikipedia.org/wiki/Category:Transaction_processing

with the correct level of locking for what you are doing