Undiagnosed High FSM Time
richard.jonas at erlang-solutions.com
Wed Jan 27 04:22:12 EST 2016
You may want to monitor
- node_put_fsm_rejected (and get, too) to if overload protection rejected
put or get requests
- node_put_fsm_active, node_put_fsm_active_60s (to see how many active put
requests are being served now and in the last minute) there is metrics for
get, too. A spike of those metrics can show you a hint that something
A big object can cause a lot of put fsms, since big object can cause
head-of-line-blocking inside the cluster. So the problem is not with
serving the get/put for the big objects, but the puts/gets coming after
that. Like during driving pushing a hard brake won't cause you traffic jam,
but the others behind you.
On Tue, Jan 26, 2016 at 10:43 PM, Alex Wolfe <alex at activeprospect.com>
> Thanks for your reply.
> We are. We sort of expected an anomaly in the object size, but there was
> none. We found the root cause. It was a large number of additions to a
> single set. It’s not clear to me which metric reveals that problem, but it
> appears as though object size doesn’t.
> > On Jan 26, 2016, at 3:40 PM, Luke Bakken <lbakken at basho.com> wrote:
> > Hi Alex -
> > Are you monitoring any of Riak's statistics? Specifically object size
> > and sibling count, though all of the stats are useful.
> > --
> > Luke Bakken
> > Engineer
> > lbakken at basho.com
> > On Tue, Jan 26, 2016 at 11:40 AM, Alex Wolfe <alex at activeprospect.com>
> >> We have a 5 node Riak cluster running 2.1.1. This morning FSM Time
> (99th percentile) went way up. We couldn't find any clear signs of trouble
> with the cluster and ultimately chose to move the data files and restart
> the nodes. Once we started with an empty DB, the FSM Time normalized. But
> now it's headed back up again. We're stumped on how to trouble shoot this
> issue. Any suggestions?
> >> _______________________________________________
> >> riak-users mailing list
> >> riak-users at lists.basho.com
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> riak-users mailing list
> riak-users at lists.basho.com
Erlang Solutions Hungary Kft
Riverpark Office K.32
Közraktár street 32. 3/1.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users