Undiagnosed High FSM Time

Richard Jonas richard.jonas at erlang-solutions.com
Wed Jan 27 04:22:12 EST 2016


You may want to monitor
- node_put_fsm_rejected (and get, too) to if overload protection rejected
put or get requests
- node_put_fsm_active, node_put_fsm_active_60s (to see how many active put
requests are being served now and in the last minute) there is metrics for
get, too. A spike of those metrics can show you a hint that something
unusual happens.

A big object can cause a lot of put fsms, since big object can cause
head-of-line-blocking inside the cluster. So the problem is not with
serving the get/put for the big objects, but the puts/gets coming after
that. Like during driving pushing a hard brake won't cause you traffic jam,
but the others behind you.

On Tue, Jan 26, 2016 at 10:43 PM, Alex Wolfe <alex at activeprospect.com>
wrote:

> Thanks for your reply.
>
> We are. We sort of expected an anomaly in the object size, but there was
> none. We found the root cause. It was a large number of additions to a
> single set. It’s not clear to me which metric reveals that problem, but it
> appears as though object size doesn’t.
>
> Alex
>
>
> > On Jan 26, 2016, at 3:40 PM, Luke Bakken <lbakken at basho.com> wrote:
> >
> > Hi Alex -
> >
> > Are you monitoring any of Riak's statistics? Specifically object size
> > and sibling count, though all of the stats are useful.
> >
> > --
> > Luke Bakken
> > Engineer
> > lbakken at basho.com
> >
> > On Tue, Jan 26, 2016 at 11:40 AM, Alex Wolfe <alex at activeprospect.com>
> wrote:
> >> We have a 5 node Riak cluster running 2.1.1. This morning FSM Time
> (99th percentile) went way up. We couldn't find any clear signs of trouble
> with the cluster and ultimately chose to move the data files and restart
> the nodes. Once we started with an empty DB, the FSM Time normalized. But
> now it's headed back up again. We're stumped on how to trouble shoot this
> issue. Any suggestions?
> >> _______________________________________________
> >> riak-users mailing list
> >> riak-users at lists.basho.com
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



-- 
Richard Jonas
Erlang Solutions Hungary Kft

Address:
  Riverpark Office K.32
  Közraktár street 32. 3/1.
  1093 Budapest
  Hungary
Phone/fax:
  +36-1-7000-654
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160127/a1181729/attachment-0002.html>


More information about the riak-users mailing list