Riak Memory Usage Constantly Growing

John E. Vincent lusis.org+riak-users at gmail.com
Tue Oct 2 11:19:59 EDT 2012


On Tue, Oct 2, 2012 at 7:55 AM, Kelly McLaughlin <kelly at basho.com> wrote:
> John and Shane,
>
> I have been looking into some memory issues lately and I would be very interested in more
> information about your particular problems. If either of you are able to get some output
> from etop using the -sort memory option when you are having elevated memory usage it
> would be very helpful to see. I know that sometimes you get the connection_lost message
> when trying to use etop, but I have found that sometimes if you keep trying it may succeed
> after a few attempts.
>
> Are either of you using MapReduce? I see that John is using 2I. Shane, do you also use 2I?
> Finally, do you notice a lot of messages to the console or console log that have the either the
> phrase 'monitor large_heap' or 'monitor long_gc'?
>
> Kelly
>

Kelly,

As it stands now, we're not seeing the problem anymore. If it crops up
again, I'll be ready to go. To answer your original question r.e. M/R,
internally we consider it a bug if our in-house ODM generates a MR
call but that's the fall back path it takes. And right now we ARE
generating considerable MR volume. When we upgraded to 1.2, we didn't
change any application code. We still get the following in the logs (I
haven't had the time to open a ticket on it) pretty frequently:

2012-10-02 15:12:37.190 [error]
<0.344.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
failed:fitting was gone before startup
2012-10-02 15:12:37.190 [error]
<0.356.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
failed:fitting was gone before startup
2012-10-02 15:12:37.191 [error]
<0.354.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
failed:fitting was gone before startup
2012-10-02 15:12:37.236 [error]
<0.344.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
failed:fitting was gone before startup
2012-10-02 15:13:13.718 [error]
<0.340.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
failed:fitting was gone before startup
2012-10-02 15:13:13.719 [error]
<0.332.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
failed:fitting was gone before startup

And that hasn't changed with the upgrade. There's other debugging
information as well that I'll need to do via ticket. I'd love to have
something to post back to the group though afterwards.

<snip>




More information about the riak-users mailing list