Riak memory usage higher than expected

Nico Meyer nico.meyer at adition.com
Thu Aug 18 06:31:06 EDT 2011


Have you taken into account the size of the bucket name? is it 12 bytes 
for key+bucket?

Also there is some additional overhead that is due to memory alignment 
requirements, depending on your CPU architecture.
I have done some tests on this for x86_64 a while ago:

size=55: overhead=9
size=56: overhead=8
size=57: overhead=23
size=58: overhead=22
size=59: overhead=21
size=60: overhead=20
size=61: overhead=19
size=62: overhead=18
size=63: overhead=17
size=64: overhead=16
size=65: overhead=15
size=66: overhead=14
size=67: overhead=13
size=68: overhead=12
size=69: overhead=11
size=70: overhead=10
size=71: overhead=9
size=72: overhead=8
Not much you can do about that, unless you want to use unaligned memory, 
which is super complicated (replace malloc with your own allocator) and 
has a huge performance penalty.

So lets assume your 12 bytes are include bucket and key:

43 byte overhead + 12 byte keydata = 55 bytes, this is the number of 
bytes allocated per entry. In reality malloc(55) allocates 64 bytes 
internally, so you total memory consumption is:

64byte*3*450Mio = 80 Gb

But if the bucket size is not included in the 12 bytes, and your bucket 
name(s) has at least a size of 2 bytes we get:

(43 + 12 + 2) bytes  = 57 bytes, and in this case malloc(57) really 
allocates 80 bytes inernally. Therefore:

80byte*3*450Mio = 100Gb


Cheers,
Nico


Am 17.08.2011 19:43, schrieb Jacques:
>
> So if my math is correct, consumption should be around 70-80 for me. 
> Yet we see a usage over 100. I'm also notclear what the step changes 
> are. If you have any insights on these that would be great. Thanks!
>
> On Aug 17, 2011 12:07 AM, "Nico Meyer" <nico.meyer at adition.com 
> <mailto:nico.meyer at adition.com>> wrote:
> > Hi Jacques,
> >
> > please read my reply back in May, which should clear things up:
> >
> > 
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/004292.html
> >
> > Cheers,
> > Nico
> >
> > On 16.08.2011 21:09, Jacques wrote:
> >> We're utilizing Riak 14.2 and we're seeing higher memory consumption
> >> than we expect.
> >>
> >> We're running on a 4 node cluster with each node housing 32gb of memory
> >> and are utilizing bitcask with a 3x write replication factor. We're
> >> seeing faster growth than we expect and also seeing weird bounces 
> update.
> >>
> >> You can see an example chart <http://picturepush.com/public/6331935>.
> >> (note, there are sometimes where we've had to stop the job for short
> >> periods of time-- you can see these as flat spots).
> >>
> >> We are doing a large throttled import that has key sizes of
> >> approximately 12 bytes. We're currently around 450mm unique items and
> >> riak memory consumption is ~110gb. The input job is probably 95% new
> >> puts and 5% overwriting puts.
> >>
> >> According to the capacity planner tools, our key space should probably
> >> be about half what are actual memory consumption is.
> >>
> >> As you can see in the chart, we're also seeing jumps in memory size at
> >> random intervals. What might these be? Nothing interesting in the logs
> >> that I can see. Regular merges.
> >>
> >> A close up <http://picturepush.com/public/6332084> of a recent jump in
> >> memory consumption for one of the nodes (they all look the same). There
> >> are no corresponding distinct patterns within the cpu chart. Things are
> >> pretty flat although we have more wait sometimes than we like (need 
> more
> >> spindles clearly).
> >>
> >>
> >> Any helpful thoughts?
> >>
> >> Thanks,
> >> Jacques
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> riak-users mailing list
> >> riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110818/b2071a11/attachment.html>


More information about the riak-users mailing list