Parameter Planning (eleveldb)

Simon Effenberg seffenberg at team.mobile.de
Sun Feb 3 15:44:09 EST 2013


Hi,

I'm not sure if I understand this all well to calculate the memory
usage per file and other stuff.

The webpage tells me some steps but I'm completly unsure if I understand all parameters.

"Step 1: Calculate Available Working Memory"

taking the example:

leveldb_working_memory = 32G * (1 - .50) = 16G

"Step 2: Calculate Working Memory per vnode"

vnode_working_memory = leveldb_working_memory / vnode_count

vnode_count = 256

=> vnode_working_memory = 16G / 256 = 64MB/vnode

also easy

"Step 3: Estimate Memory Used by Open Files"

open_file_memory =
   (max_open_files-10) * (
     184 + (average_sst_filesize/2048) *
     (8 + ((average_key_size+average_value_size)/2048 +1) *
     0.6
   )

so how do I know the average_sst_filesize (and what is this value exactly)
(and is 2048 for both /2048 true or 4096 in riak 1.2?) and how do I know
the max_open_files?


average_key_size could be 16byte (I have to ask someone but taking it for now)
average_value_size will be 14kbyte 

so for now

open_file_memory =
   (max_open_files-10) * (
     184 + (average_sst_filesize/2048) *
     (8 + ((16+14336)/2048 +1) *
     0.6
   )

(side question: should I increase the block_size because of the big average value size?
and also should I leave the cache_size at the default value like it was recommended?)

"Step 4: Calculate Average Write Buffer"

should I increase these values or not? If only two are held in memory and I have, as an
example, 32GB or RAM like in this scenario, shouldn't I increase it to something else?

"Step 5: Calculate vnode Memory Used"

memory/vnode = average_write_buffer_size + cache_size + open_file_memory + 20 MB

So for now I miss almost all 3 values :(.

To get an Idea:

- 3 buckets
- overall ~ 343347732 keys (but only 2/3 have 14kbyte in average)


Thx for help!
Simon




More information about the riak-users mailing list