Storing Sets in a new backend

Anthony Molinaro anthonym at alumni.caltech.edu
Tue Dec 21 20:02:09 EST 2010


Hi,

Just wondering if anyone had an opinions on this idea?

Thanks,

-Anthony

On Fri, Dec 17, 2010 at 03:20:58PM -0800, Anthony Molinaro wrote:
> Hi,
> 
>   I have a situation where I need to store sets of integers as efficiently
> as possible.  In order to do so I was going to implement a custom backend
> and alter the semantics of get/put/delete, such that
> 
> put (K, [{i,V}])    - insert one or more V into list at K
> put (K, [{d,V}])    - delete one or more V from list at K
> get (K)             - returns list at K
> delete (K)          - delete entire list
> 
> I'll probably keep the lists in sorted order via insertion sort so that
> a quorum read can easily determine equality.
> 
> My backend would probably be a linked in driver which mmaps a file so
> I can keep this stuff as tightly packed in memory as possible.
> 
> We did something similar using just riak_core for a different system,
> but for that system we had a custom thrift frontend and didn't adhere
> to the riak_kv api.
> 
> So I was wondering, what sort of issues might I see with this sort of
> use of riak?  Is there another way I'm missing which would allow me
> to efficiently do this?
> 
> I expect in the end to have approximately 500 million lists of integers
> with most in the ~10 integer range with a few up to the ~200 integer
> range.  Also, I'll probably have a timestamp associated with each
> integer and want to use map-reduce to expire entries which are too old.
> 
> Anway, I realize this might be an odd usage, and I can always fallback
> to what we did before which was a thrift server on top up riak_core with
> a custom datastore, but I figured if I could use the same API as riak_kv
> with a custom backend it might save me a little time.
> 
> -Anthony
> 
> -- 
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym at alumni.caltech.edu>

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym at alumni.caltech.edu>



More information about the riak-users mailing list