Storing large collections.

Jeremiah Peschka jeremiah.peschka at gmail.com
Mon Mar 5 23:27:41 EST 2012


On Mar 5, 2012, at 7:09 PM, Eric Siegel wrote:

> Originally, I had planned to map each of my items to their own key.  
> This was foolish as I estimate that I'll have around 6 billion keys, and this simply won't fit into memory.

This is only an issue if you're using bitcask (which is the default back end for Riak). If you're willing to consider one of the alternative storage backends (LevelDB or InnoDB), then you can store as much data as you want, as long as you have disk space to hold it.

> 
> My next plan of attack is store a collection of items to a given key, approximately 1million keys each with 6000 values.

This sounds cumbersome.

> 
> I was wondering how much performance will degrade as the value size gets larger.  Also, I'm worried about
> having to merge the new value into the collection.

I would too - there's no guarantee that two writes to the same collection won't happen enough together to cause a conflict.

> 
> Does anyone have any experience with this problem and perhaps some advice?
> 
> eric
> 


---
Jeremiah Peschka - Managing Director, Brent Ozar PLF, LLC
Microsoft SQL Server MVP



More information about the riak-users mailing list