Data store for sequential key-value, time limited event "journal"

by shevron   Last Updated August 12, 2018 14:05 PM

I'm designing a system in which a key component acts like a "journal", in the sense that it stores a list of sequential events that can change the state of the entire system. Upon request, any other component in the system can ask this component for the full list of events that happened since event XXX.

One can think of the sequential event ID as the key, and of the event itself (a JSON-encoded object of a few kb) as the value.

The system needs to keep a history of a couple of days worth of data, or alternatively, a few million records, but may need to scale to 10s of millions. Past events should be easy to automatically remove from storage.

Preferably, it should be easy to scale this storage to multiple nodes using master-master or even master-replica replication.

Any suggestions for such storage would be helpful.

Related Questions

SQL database with only one table

Updated September 18, 2017 14:05 PM

Which database strategy should I choose?

Updated September 18, 2017 15:05 PM

Database design, processor vs inconsistency

Updated April 28, 2016 08:02 AM