Using Riak for time-series data
siculars at gmail.com
Tue Jan 29 18:28:48 EST 2013
Size wise, riak can handle. Leveldb will compress natively. Search (particularly non composite queries) and materialized keys will help you access your data. The main concern as you eluded to would be merging multiple sets. Riak doesn't like to do that. You would have to bring two sets into application space and do your magic there. Maybe check an hbase based solution lik http://opentsdb.net/ or Cassandra. Overall the fact that your data is immutable works in your favor and opens a lot of solutions up.
Let us know which way you go. Sounds fun.
Sent from my iRotaryPhone
On Jan 29, 2013, at 16:53, Boris Solovyov <boris.solovyov at gmail.com> wrote:
> Is Riak good for time-series data like stock trade information? Data I consider is:
> Millions, maybe billions or more of series, each has name, like stock ticker symbol
> One-per-second measurement (occasionally will have missing values, rarely)
> Many measurement are zero or static, unchanging
> Append-only, never update, never write in past
> Data probably high compressible, good if it can be compress well
> Query requirements are probably typical, for graphing and analyze,
> Get all measurements for [serie1,serie2,...serieN] for a time range
> Find series with similar names, maybe with wildcards matching
> Get all measurements aggregated for similar series, e.g. get a time range of two series, but add their measurements together to produce a single series of output
> In reading on Riak, seems to have many nice feature not needed for this purpose, and maybe too costly for some of suggested operations. Too much IO, maybe too expensive to add two serie together, etc. Also, features like conflict detection, versioning, extra metadata etc not needed. And if Riak store timestamp with every value, will reduce compressibility, i.e. no need store timestamps with every measure if serie is just long set of 0's.
> If Riak is not right, what suggestions do you have?
> riak-users mailing list
> riak-users at lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users