Riak 1.4.12 high I/O while almost idle

Engel Sanchez engel at basho.com
Fri May 22 11:34:09 EDT 2015


Hi Timo. Did you check if there is active anti-entropy activity? [1]  That
could generate a lot of I/O in the background while building the trees with
the data needed for automatic repair or running exchanges to verify
replicas are up to date.

[1] http://docs.basho.com/riak/latest/ops/advanced/aae/#Monitoring-AAE

On Fri, May 22, 2015 at 10:54 AM, Doug Rohrer <drohrer at basho.com> wrote:

> Timo:
>
> The general layout of the information is pretty similar to Top (and
> described on the etop docs http://www.erlang.org/doc/man/etop.html and
> http://www.erlang.org/doc/apps/observer/etop_ug.html) - the one downside
> is that it doesn't profile which function is using the most CPU (the "Reds"
> column is Reductions, which is Erlang's measure of work being done by a
> process). If you run it for a while longer, with a shorter -interval
> setting, you may be able to "catch" what function is being run most
> frequently other than the gen_server:loop and gen_event:loop functions
> you're seeing, as they are generic OTP functions and not very useful for
> diagnostics.
>
> Doug
>
> On May 22, 2015, at 9:58 AM, Timo Gatsonides <tpjg at me.com> wrote:
>
> Try running `riak-admin top` on the node to see what's going on:
>
> http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top
>
> This may give you some insight into what that node is doing.
>
>
> Thanks for the quick response. I had already done that, however I don’t
> understand the output …
>
> It is below.
>
> -Timo
>
>
>
> ===============================================================================================================================
>  ''                                                 13:56:31
>  Load:  cpu         0               Memory:  total       69336    binary
>     2717
>         procs     664                        processes    8439    code
>     10720
>         runq        0                        atom          501    ets
>     6227
>
> Pid                 Name or Initial Func         Time       Reds
> Memory       MsgQ Current Function
>
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'    5091705
> 2600          0 cpu_sup:measurement_server_loop/1
> <6206.181.0>        riak_core_vnode_manager       '-'    3064188
> 231536          0 gen_server:loop/6
> <6206.580.0>        proc_lib:init_p/5             '-'    2805797
> 34472          0 gen_fsm:loop/7
> <6206.715.0>        proc_lib:init_p/5             '-'    2485959
> 55144          0 gen_fsm:loop/7
> <6206.709.0>        proc_lib:init_p/5             '-'    2445894
> 13896          3 eleveldb:get/3
> <6206.94.0>         riak_sysmon_filter            '-'    2394053
> 13656          0 gen_server:loop/6
> <6206.714.0>        proc_lib:init_p/5             '-'    2177064
> 34472          0 gen_fsm:loop/7
> <6206.710.0>        proc_lib:init_p/5             '-'    2153568
> 34472          0 gen_fsm:loop/7
> <6206.3.0>          erl_prim_loader               '-'    2053533
> 142464          0 erl_prim_loader:loop/3
> <6206.707.0>        proc_lib:init_p/5             '-'    2032382
> 34472          0 gen_fsm:loop/7
>
>
> ===============================================================================================================================
>  ''                                                 13:56:36
>  Load:  cpu         7               Memory:  total       70450    binary
>     2994
>         procs     665                        processes    9133    code
>     10720
>         runq        3                        atom          501    ets
>     6247
>
> Pid                 Name or Initial Func         Time       Reds
> Memory       MsgQ Current Function
>
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'       8092
> 2600          0 cpu_sup:measurement_server_loop/1
> <6206.580.0>        proc_lib:init_p/5             '-'       6710
> 34472          0 eleveldb:write/3
> <6206.94.0>         riak_sysmon_filter            '-'       6447
> 13656          0 gen_server:loop/6
> <6206.709.0>        proc_lib:init_p/5             '-'       6174
> 34472          0 gen_fsm:loop/7
> <6206.28437.0>      erlang:apply/2                '-'       3623
> 21440          0 erlang:receive_emd/3
> <6206.707.0>        proc_lib:init_p/5             '-'       3089
> 21696          0 gen_fsm:loop/7
> <6206.711.0>        proc_lib:init_p/5             '-'       3075
> 34472          0 gen_fsm:loop/7
> <6206.374.0>        riak_kv_stat_sj_2             '-'       2679
> 3856          0 gen_server:loop/6
> <6206.714.0>        proc_lib:init_p/5             '-'       1955
> 21696          0 gen_fsm:loop/7
> <6206.7.0>          application_controller        '-'       1779
> 55880          0 gen_server:loop/6
>
>
> ===============================================================================================================================
>  ''                                                 13:56:40
>  Load:  cpu         8               Memory:  total       69471    binary
>     2827
>         procs     664                        processes    8469    code
>     10720
>         runq        0                        atom          501    ets
>     6251
>
> Pid                 Name or Initial Func         Time       Reds
> Memory       MsgQ Current Function
>
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'      10113
> 2600          0 cpu_sup:measurement_server_loop/1
> <6206.181.0>        riak_core_vnode_manager       '-'       9724
> 231536          0 gen_server:loop/6
> <6206.182.0>        riak_core_capability          '-'       5827
> 54960          0 gen_server:loop/6
> <6206.715.0>        proc_lib:init_p/5             '-'       5814
> 34472          0 gen_fsm:loop/7
> <6206.712.0>        proc_lib:init_p/5             '-'       5153
> 21696          0 gen_fsm:loop/7
> <6206.94.0>         riak_sysmon_filter            '-'       4465
> 13656          0 gen_server:loop/6
> <6206.580.0>        proc_lib:init_p/5             '-'       3619
> 34472          0 gen_fsm:loop/7
> <6206.376.0>        riak_kv_stat_sj_4             '-'       2542
> 3856          0 gen_server:loop/6
> <6206.707.0>        proc_lib:init_p/5             '-'       2298
> 34472          0 gen_fsm:loop/7
> <6206.7.0>          application_controller        '-'       2148
> 55880          0 gen_server:loop/6
>
>
> ===============================================================================================================================
>  ''                                                 13:56:45
>  Load:  cpu         9               Memory:  total       69332    binary
>     2786
>         procs     665                        processes    8324    code
>     10720
>         runq        0                        atom          501    ets
>     6269
>
> Pid                 Name or Initial Func         Time       Reds
> Memory       MsgQ Current Function
>
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'       8092
> 2600          0 cpu_sup:measurement_server_loop/1
> <6206.712.0>        proc_lib:init_p/5             '-'       5472
> 34472          0 gen_fsm:loop/7
> <6206.94.0>         riak_sysmon_filter            '-'       4674
> 13656          0 gen_server:loop/6
> <6206.580.0>        proc_lib:init_p/5             '-'       3939
> 55144          0 gen_fsm:loop/7
> <6206.161.0>        riak_core_ring_manager        '-'       3591
> 230096          0 gen_server:loop/6
> <6206.709.0>        proc_lib:init_p/5             '-'       2890
> 34472          0 gen_fsm:loop/7
> <6206.376.0>        riak_kv_stat_sj_4             '-'       2880
> 5720          0 gen_server:loop/6
> <6206.710.0>        proc_lib:init_p/5             '-'       2476
> 34472          0 gen_fsm:loop/7
> <6206.708.0>        proc_lib:init_p/5             '-'       2033
> 21696          0 gen_fsm:loop/7
> <6206.639.0>        proc_lib:init_p/5             '-'       1997
> 34472          0 gen_fsm:loop/7
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150522/21b16831/attachment-0002.html>


More information about the riak-users mailing list