Issues with capacity planning pages on wiki

Anthony Molinaro anthonym at alumni.caltech.edu
Wed May 25 13:50:17 EDT 2011


Hi Nico,

  Thanks for the awesome analysis, the one part I was at first confused
about was this part

On Wed, May 25, 2011 at 06:22:53PM +0200, Nico Meyer wrote:
> Your actual RAM consumption of 70 GB seems to be at odd with the
> output of erlang:memory/0 that you sent:
> 
> {total,7281790968} =>   RAM: 7281790968 * 8 = 54.3 GB

Because at first I thought your were multiplying by wordsize which is 8 as
well, but then I realized you were multiplying by number of nodes (also 8).

Anyway, things make a lot more sense now, and I'm thinking I may need 
to fork bitcask and get rid of some of that extra overhead.  For instance
13 bytes of overhead to store a tuple of binaries seems unnecessary, it's
probably better to just have a single binary with the bucket size as a
prefix, so something like

<<BucketSize:16,Bucket,Key>>

That way you turn 13 bytes of overhead to 2.

Of course I'd need some way to work with old data, but a one time migration
shouldn't be too bad.

It also seems like there should be some way to trim down some of that on
disk usage.  I mean 300+ bytes to store 36 bytes is a lot.

Again, many thanks this is an awesome analysis.

-Anthony

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym at alumni.caltech.edu>




More information about the riak-users mailing list