Riak memory usage higher than expected

Jacques whshub at gmail.com
Sun Aug 21 15:55:27 EDT 2011


Thanks for the help. We've decided to expand the cluster and plan a future
analysis.

Thanks again!
On Aug 19, 2011 1:57 AM, "Nico Meyer" <nico.meyer at adition.com> wrote:
> Hi Jaques,
>
> at least it is within a factor of 1.5 or so, could be worse :-).
>
> I believe some of the additional overhead is due to some intricacies in
> Erlang memory management (or possibly memory management in general, like
> fragmentation). There was some discussion about a similar issue on the
> Erlang mailing list a while ago
> (
http://erlang.2086793.n4.nabble.com/discrepancy-of-memory-usage-figures-td3618939.html),

> but no conclusion.
>
> I just checked one of our riak nodes, and also saw a similar
> discrepancy: 'erlang:memory()' (from the erlang shell of the node)
> reported about 10Gb used, but ps reported about 13Gb of resident memory
> for the beam.smp process.
> After restarting the riak node the discrepancy became much smaller, 9Gb
> to 10Gb.
>
> Cheers,
> Nico
>
>
> Am 19.08.2011 01:40, schrieb Jacques:
>> This is very helpful. Thank you.
>>
>> We actually have two different key sizes, 9 key + 1 bucket = 10 total
>> And 11 key + 1 bucket = 12 total.
>>
>> Sounds like both of those will get stored on a 64 byte boundary.
>>
>> That puts us at ~80gb of expected usage. We're currently at 110gb.
>>
>> Coming from the java world, I'm wondering if there are any
>> intermediate objects that need to be collected?
>>
>> It sounds like you're generally saying that "it is what it is" for now
>> and just capacity plan around it.
>>
>> thanks again,
>> Jacques
>>
>> On Thu, Aug 18, 2011 at 3:31 AM, Nico Meyer <nico.meyer at adition.com
>> <mailto:nico.meyer at adition.com>> wrote:
>>
>> Have you taken into account the size of the bucket name? is it 12
>> bytes for key+bucket?
>>
>> Also there is some additional overhead that is due to memory
>> alignment requirements, depending on your CPU architecture.
>> I have done some tests on this for x86_64 a while ago:
>>
>> size=55: overhead=9
>> size=56: overhead=8
>> size=57: overhead=23
>> size=58: overhead=22
>> size=59: overhead=21
>> size=60: overhead=20
>> size=61: overhead=19
>> size=62: overhead=18
>> size=63: overhead=17
>> size=64: overhead=16
>> size=65: overhead=15
>> size=66: overhead=14
>> size=67: overhead=13
>> size=68: overhead=12
>> size=69: overhead=11
>> size=70: overhead=10
>> size=71: overhead=9
>> size=72: overhead=8
>> Not much you can do about that, unless you want to use unaligned
>> memory, which is super complicated (replace malloc with your own
>> allocator) and has a huge performance penalty.
>>
>> So lets assume your 12 bytes are include bucket and key:
>>
>> 43 byte overhead + 12 byte keydata = 55 bytes, this is the number
>> of bytes allocated per entry. In reality malloc(55) allocates 64
>> bytes internally, so you total memory consumption is:
>>
>> 64byte*3*450Mio = 80 Gb
>>
>> But if the bucket size is not included in the 12 bytes, and your
>> bucket name(s) has at least a size of 2 bytes we get:
>>
>> (43 + 12 + 2) bytes = 57 bytes, and in this case malloc(57)
>> really allocates 80 bytes inernally. Therefore:
>>
>> 80byte*3*450Mio = 100Gb
>>
>>
>> Cheers,
>> Nico
>>
>>
>> Am 17.08.2011 19:43, schrieb Jacques:
>>>
>>> So if my math is correct, consumption should be around 70-80 for
>>> me. Yet we see a usage over 100. I'm also notclear what the step
>>> changes are. If you have any insights on these that would be
>>> great. Thanks!
>>>
>>> On Aug 17, 2011 12:07 AM, "Nico Meyer" <nico.meyer at adition.com
>>> <mailto:nico.meyer at adition.com>> wrote:
>>> > Hi Jacques,
>>> >
>>> > please read my reply back in May, which should clear things up:
>>> >
>>> >
>>>
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-May/004292.html
>>> >
>>> > Cheers,
>>> > Nico
>>> >
>>> > On 16.08.2011 21:09, Jacques wrote:
>>> >> We're utilizing Riak 14.2 and we're seeing higher memory
>>> consumption
>>> >> than we expect.
>>> >>
>>> >> We're running on a 4 node cluster with each node housing 32gb
>>> of memory
>>> >> and are utilizing bitcask with a 3x write replication factor.
>>> We're
>>> >> seeing faster growth than we expect and also seeing weird
>>> bounces update.
>>> >>
>>> >> You can see an example chart
>>> <http://picturepush.com/public/6331935>.
>>> >> (note, there are sometimes where we've had to stop the job for
>>> short
>>> >> periods of time-- you can see these as flat spots).
>>> >>
>>> >> We are doing a large throttled import that has key sizes of
>>> >> approximately 12 bytes. We're currently around 450mm unique
>>> items and
>>> >> riak memory consumption is ~110gb. The input job is probably
>>> 95% new
>>> >> puts and 5% overwriting puts.
>>> >>
>>> >> According to the capacity planner tools, our key space should
>>> probably
>>> >> be about half what are actual memory consumption is.
>>> >>
>>> >> As you can see in the chart, we're also seeing jumps in memory
>>> size at
>>> >> random intervals. What might these be? Nothing interesting in
>>> the logs
>>> >> that I can see. Regular merges.
>>> >>
>>> >> A close up <http://picturepush.com/public/6332084> of a recent
>>> jump in
>>> >> memory consumption for one of the nodes (they all look the
>>> same). There
>>> >> are no corresponding distinct patterns within the cpu chart.
>>> Things are
>>> >> pretty flat although we have more wait sometimes than we like
>>> (need more
>>> >> spindles clearly).
>>> >>
>>> >>
>>> >> Any helpful thoughts?
>>> >>
>>> >> Thanks,
>>> >> Jacques
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> riak-users mailing list
>>> >> riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110821/13027ada/attachment.html>


More information about the riak-users mailing list