node_put_fsm_active maxing out

Doug Rohrer drohrer at basho.com
Thu Nov 5 18:49:20 EST 2015


Chris:

First, I'd like to thank you for going through the effort of identifying the difference that is causing you issues. Narrowing down the change to the Erlang version is incredibly helpful, and I doubt many other people would have found this correlation.

Second, while several of us from Basho are currently traveling for conferences and some meetings next week, we will definitely look into the changes between -basho5 and -basho6 as they relate to this issue.

I'll create an issue in our internal tracker that's specific to our patched OTP, but could you create something at https://github.com/basho/riak/issues <https://github.com/basho/riak/issues> so we have somewhere to post information as we work through troubleshooting the issue that you (and others) can see publicly?

Also, would it be acceptable for me to reach out to you individually to discuss the issue in more detail?

Thanks,

Doug Rohrer



> On Nov 5, 2015, at 3:21 PM, Chris Read <chris.read at gmail.com> wrote:
> 
> Anyone out there?
> 
> Here's some more detail on the Erlang builds:
> 
> This one works as expected:
> 
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:64] [hipe] [kernel-poll:true]">>
> 
> This one has the problem:
> 
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_system_version : <<"Erlang R16B02_basho6 (erts-5.10.3) [source-bcd8abb] [64-bit] [smp:24:24] [async-threads:64] [kernel-poll:true] [frame-pointer]">>
> 
> Chris
> 
> 
> On Tue, Nov 3, 2015 at 12:47 PM, Chris Read <chris.read at gmail.com <mailto:chris.read at gmail.com>> wrote:
> Greetings all...
> 
> We've been building riak from source for a while, but I've had trouble getting the 2.1 lines built reliably and so would like to revert back to using the .deb package. The problem I have is that in our test environment we always manage to max out node_put_fsm_active under sustained write loads, and they never drop.
> 
> When running riak 2.0.4 on R16B02-basho5 (our current prod version) everything is working as expected. 
> 
> Using the .deb package of 2.0.4 pushes us to R16B02_basho6, which is where we see the problem arrive of node_puts_fsm_active going up and never dropping back own again, even after the write load stops.
> 
> Further testing with the riak 2.0.6 2.1.1 .deb packages (both contain R16B02_basho8) show the same problem.
> 
> Questions I have are:
> 
> 1) Anyone else seen this?
> 2) Is there any way I can see why these FSM's appear to be deadlocked?
> 
> Thanks,
> 
> Chris
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20151105/488e94b8/attachment-0002.html>


More information about the riak-users mailing list