Large numbers of keys

Dietrich Featherston dietrich.featherston at gmail.com
Wed Jun 27 11:12:18 EDT 2012


LevelDB is a nice option with a key space that will not fit in memory. Whether or not bitcask will work for you depends on total memory capacity of the cluster and N value. Recommend using the bitcask capacity planner to see if it is a suitable backed for your hardware+data combination.

http://wiki.basho.com/Bitcask-Capacity-Planning.html

D



On Jun 27, 2012, at 7:19 AM, Eric Anderson <anderson at copperegg.com> wrote:

> Hi all,
> 
> I did a quick computation, and figured that I need to be able to manage a minimum 5 billion keys roughly 80bytes each.  Since Bitcask needs all keys in memory, that's a lot of memory. How does it handle swapping?  I typically only need a portion of those keys available at a given time, so swapping them out would be fine if it was intelligent about that and loading it back in.  Meaning that if it has to pull in the whole index to add a new key, that would be an issue for me.  I typically need to add new keys pretty quickly and then access the most recent ones.  The older ones could swap out without too much worry.
> 
> Would LevelDB help with this?
> 
> Thanks!
> 
> Eric Anderson
> Co-Founder
> http://CopperEgg.com
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120627/15c81422/attachment.html>


More information about the riak-users mailing list