NoSQL as storage for publish-subscribe/multi-reader queue?

Looking for a storage solution for the following problem, preferably with some NoSQL-like speed and scalability:

• Events. Lots of them, little data per event. This is what we need to store.

  • Not necessary to exactly keep the order in which the events arrive.

It would be nice not to store multiple copies of each event (as in separate storage for each observer).

• Observers. A few of them (< 50) They need to read the events开发者_运维问答

  • At their own pace (pull model)

  • Preferably with a "get me the next chunk of unread events" API

  • Each observer needs to read every event (eventually)

  • No guarantees on how often they will pull the changes. It might be necessary to store lots of events before they are read.

In an RDBMS you'd probably just number the events sequentially and remember the "last read no" for every observer. Is it possible to implement something similar while trading some of the ACID for speed & scalability?

So far Redis with its lists looks good - anything better I should look at?

I think Redis lists are a good choice. I'd go with a list for each observer though - that way you have O(1) read and write with RPUSH/LPOP, and events automatically disappear from the system when all observers have received them.

You can reduce the storage required for each observer by just storing an event id in each list, though then you will need to keep a counter for each event to determine when it can be removed from the system.

To implement with a single list, set up a counter that is incremented every time an event is added to the head of list. Also set up a counter for each client indicating how many events they have received. The difference between those is the number of items you need to get from the list.

The disadvantage of this approach is that new items can be added to the list after you check the counters. You can get around this by counting from the tail of the list, but that is O(N) rather than O(1). You can reduce N by trimming received events from the list and maintaining a counter for tail position also - how well that works will depend on how many events can accumulate when an observer is offline.

You could take a look at how it's done in Tarantool, with a Lua procedure to keep a ring buffer for events: https://github.com/mailru/tntlua/blob/master/notifications.lua