Riak 1.4.12 high I/O while almost idle

Doug Rohrer drohrer at basho.com
Fri May 22 10:54:14 EDT 2015


Timo:

The general layout of the information is pretty similar to Top (and described on the etop docs http://www.erlang.org/doc/man/etop.html <http://www.erlang.org/doc/man/etop.html> and http://www.erlang.org/doc/apps/observer/etop_ug.html <http://www.erlang.org/doc/apps/observer/etop_ug.html>) - the one downside is that it doesn't profile which function is using the most CPU (the "Reds" column is Reductions, which is Erlang's measure of work being done by a process). If you run it for a while longer, with a shorter -interval setting, you may be able to "catch" what function is being run most frequently other than the gen_server:loop and gen_event:loop functions you're seeing, as they are generic OTP functions and not very useful for diagnostics.

Doug

> On May 22, 2015, at 9:58 AM, Timo Gatsonides <tpjg at me.com> wrote:
> 
>> Try running `riak-admin top` on the node to see what's going on:
>> 
>> http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top <http://docs.basho.com/riak/1.4.12/ops/running/tools/riak-admin/#top>
>> 
>> This may give you some insight into what that node is doing.
> 
> Thanks for the quick response. I had already done that, however I don’t understand the output … 
> 
> It is below.
> 
> -Timo
> 
> 
> ===============================================================================================================================
>  ''                                                 13:56:31
>  Load:  cpu         0               Memory:  total       69336    binary       2717
>         procs     664                        processes    8439    code        10720
>         runq        0                        atom          501    ets          6227
> 
> Pid                 Name or Initial Func         Time       Reds     Memory       MsgQ Current Function
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'    5091705       2600          0 cpu_sup:measurement_server_loop/1       
> <6206.181.0>        riak_core_vnode_manager       '-'    3064188     231536          0 gen_server:loop/6                       
> <6206.580.0>        proc_lib:init_p/5             '-'    2805797      34472          0 gen_fsm:loop/7                          
> <6206.715.0>        proc_lib:init_p/5             '-'    2485959      55144          0 gen_fsm:loop/7                          
> <6206.709.0>        proc_lib:init_p/5             '-'    2445894      13896          3 eleveldb:get/3                          
> <6206.94.0>         riak_sysmon_filter            '-'    2394053      13656          0 gen_server:loop/6                       
> <6206.714.0>        proc_lib:init_p/5             '-'    2177064      34472          0 gen_fsm:loop/7                          
> <6206.710.0>        proc_lib:init_p/5             '-'    2153568      34472          0 gen_fsm:loop/7                          
> <6206.3.0>          erl_prim_loader               '-'    2053533     142464          0 erl_prim_loader:loop/3                  
> <6206.707.0>        proc_lib:init_p/5             '-'    2032382      34472          0 gen_fsm:loop/7                          
> 
> ===============================================================================================================================
>  ''                                                 13:56:36
>  Load:  cpu         7               Memory:  total       70450    binary       2994
>         procs     665                        processes    9133    code        10720
>         runq        3                        atom          501    ets          6247
> 
> Pid                 Name or Initial Func         Time       Reds     Memory       MsgQ Current Function
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'       8092       2600          0 cpu_sup:measurement_server_loop/1       
> <6206.580.0>        proc_lib:init_p/5             '-'       6710      34472          0 eleveldb:write/3                        
> <6206.94.0>         riak_sysmon_filter            '-'       6447      13656          0 gen_server:loop/6                       
> <6206.709.0>        proc_lib:init_p/5             '-'       6174      34472          0 gen_fsm:loop/7                          
> <6206.28437.0>      erlang:apply/2                '-'       3623      21440          0 erlang:receive_emd/3                    
> <6206.707.0>        proc_lib:init_p/5             '-'       3089      21696          0 gen_fsm:loop/7                          
> <6206.711.0>        proc_lib:init_p/5             '-'       3075      34472          0 gen_fsm:loop/7                          
> <6206.374.0>        riak_kv_stat_sj_2             '-'       2679       3856          0 gen_server:loop/6                       
> <6206.714.0>        proc_lib:init_p/5             '-'       1955      21696          0 gen_fsm:loop/7                          
> <6206.7.0>          application_controller        '-'       1779      55880          0 gen_server:loop/6                       
> 
> ===============================================================================================================================
>  ''                                                 13:56:40
>  Load:  cpu         8               Memory:  total       69471    binary       2827
>         procs     664                        processes    8469    code        10720
>         runq        0                        atom          501    ets          6251
> 
> Pid                 Name or Initial Func         Time       Reds     Memory       MsgQ Current Function
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'      10113       2600          0 cpu_sup:measurement_server_loop/1       
> <6206.181.0>        riak_core_vnode_manager       '-'       9724     231536          0 gen_server:loop/6                       
> <6206.182.0>        riak_core_capability          '-'       5827      54960          0 gen_server:loop/6                       
> <6206.715.0>        proc_lib:init_p/5             '-'       5814      34472          0 gen_fsm:loop/7                          
> <6206.712.0>        proc_lib:init_p/5             '-'       5153      21696          0 gen_fsm:loop/7                          
> <6206.94.0>         riak_sysmon_filter            '-'       4465      13656          0 gen_server:loop/6                       
> <6206.580.0>        proc_lib:init_p/5             '-'       3619      34472          0 gen_fsm:loop/7                          
> <6206.376.0>        riak_kv_stat_sj_4             '-'       2542       3856          0 gen_server:loop/6                       
> <6206.707.0>        proc_lib:init_p/5             '-'       2298      34472          0 gen_fsm:loop/7                          
> <6206.7.0>          application_controller        '-'       2148      55880          0 gen_server:loop/6                       
> 
> ===============================================================================================================================
>  ''                                                 13:56:45
>  Load:  cpu         9               Memory:  total       69332    binary       2786
>         procs     665                        processes    8324    code        10720
>         runq        0                        atom          501    ets          6269
> 
> Pid                 Name or Initial Func         Time       Reds     Memory       MsgQ Current Function
> -------------------------------------------------------------------------------------------------------------------------------
> <6206.105.0>        erlang:apply/2                '-'       8092       2600          0 cpu_sup:measurement_server_loop/1       
> <6206.712.0>        proc_lib:init_p/5             '-'       5472      34472          0 gen_fsm:loop/7                          
> <6206.94.0>         riak_sysmon_filter            '-'       4674      13656          0 gen_server:loop/6                       
> <6206.580.0>        proc_lib:init_p/5             '-'       3939      55144          0 gen_fsm:loop/7                          
> <6206.161.0>        riak_core_ring_manager        '-'       3591     230096          0 gen_server:loop/6                       
> <6206.709.0>        proc_lib:init_p/5             '-'       2890      34472          0 gen_fsm:loop/7                          
> <6206.376.0>        riak_kv_stat_sj_4             '-'       2880       5720          0 gen_server:loop/6                       
> <6206.710.0>        proc_lib:init_p/5             '-'       2476      34472          0 gen_fsm:loop/7                          
> <6206.708.0>        proc_lib:init_p/5             '-'       2033      21696          0 gen_fsm:loop/7                          
> <6206.639.0>        proc_lib:init_p/5             '-'       1997      34472          0 gen_fsm:loop/7   
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150522/f2b7abae/attachment-0002.html>


More information about the riak-users mailing list