Issues with high node load and very slow response

bryan hunt bhunt at basho.com
Sun Jul 27 07:55:36 EDT 2014


Are those machines all on the same LAN ?

On 27 Jul 2014, at 11:12, Chaim Solomon <chaim at itcentralstation.com> wrote:

> The objects are small - a few kbyte at most. I disables allow_mult.
> It's JSON data (well, I stick a hash into the Python library).
> 
> I attached the status for all 7 nodes.
> 
> Thanks
> 
> Chaim Solomon
> 
> riak-admin member-status
> ================================= Membership ==================================
> Status     Ring    Pending    Node
> -------------------------------------------------------------------------------
> valid      15.6%     15.6%    'riak at 10.128.133.59'
> valid      15.6%     14.1%    'riak at 10.128.137.185'
> valid      14.1%     14.1%    'riak at 10.128.138.25'
> valid      17.2%     14.1%    'riak at 10.128.179.86'
> valid      14.1%     14.1%    'riak at 10.128.220.135'
> valid       9.4%     14.1%    'riak at 10.128.220.242'
> valid      14.1%     14.1%    'riak at 10.128.224.15'
> -------------------------------------------------------------------------------
> Valid:7 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
> 
> (I added the last node on Thursday - so it really should have finished earlier)
> 
> riak-admin ring-status
> ================================== Claimant ===================================
> Claimant:  'riak at 10.128.137.185'
> Status:     up
> Ring Ready: true
> 
> ============================== Ownership Handoff ==============================
> Owner:      riak at 10.128.137.185
> Next Owner: riak at 10.128.220.242
> 
> Index: 570899077082383952423314387779798054553098649600
>   Waiting on: [riak_kv_vnode]
>   Complete:   [riak_pipe_vnode]
> 
> -------------------------------------------------------------------------------
> Owner:      riak at 10.128.179.86
> Next Owner: riak at 10.128.220.242
> 
> Index: 388211372416021087647853783690262677096107081728
>   Waiting on: [riak_kv_vnode]
>   Complete:   [riak_pipe_vnode]
> 
> Index: 844930634081928249586505293914101120738586001408
>   Waiting on: [riak_kv_vnode]
>   Complete:   [riak_pipe_vnode]
> 
> -------------------------------------------------------------------------------
> 
> ============================== Unreachable Nodes ==============================
> All nodes are up and reachable
> 
> 
> riak-admin status
> 1-minute stats for 'riak at 10.128.224.15'
> -------------------------------------------
> riak_kv_stat_ts : 1406455700
> vnode_gets : 12
> vnode_gets_total : 1478707
> vnode_puts : 0
> vnode_puts_total : 297943
> vnode_index_refreshes : 0
> vnode_index_refreshes_total : 0
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 0
> vnode_index_writes_total : 297102
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 0
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> object_counter_merge : 0
> object_counter_merge_total : 0
> object_counter_merge_time_mean : 0
> object_counter_merge_time_median : 0
> object_counter_merge_time_95 : 0
> object_counter_merge_time_99 : 0
> object_counter_merge_time_100 : 0
> vnode_counter_update : 0
> vnode_counter_update_total : 0
> vnode_counter_update_time_mean : 0
> vnode_counter_update_time_median : 0
> vnode_counter_update_time_95 : 0
> vnode_counter_update_time_99 : 0
> vnode_counter_update_time_100 : 0
> object_set_merge : 0
> object_set_merge_total : 0
> object_set_merge_time_mean : 0
> object_set_merge_time_median : 0
> object_set_merge_time_95 : 0
> object_set_merge_time_99 : 0
> object_set_merge_time_100 : 0
> vnode_set_update : 0
> vnode_set_update_total : 0
> vnode_set_update_time_mean : 0
> vnode_set_update_time_median : 0
> vnode_set_update_time_95 : 0
> vnode_set_update_time_99 : 0
> vnode_set_update_time_100 : 0
> object_map_merge : 0
> object_map_merge_total : 0
> object_map_merge_time_mean : 0
> object_map_merge_time_median : 0
> object_map_merge_time_95 : 0
> object_map_merge_time_99 : 0
> object_map_merge_time_100 : 0
> object_merge : 0
> object_merge_total : 689
> object_merge_time_mean : 0
> object_merge_time_median : 0
> object_merge_time_95 : 0
> object_merge_time_99 : 0
> object_merge_time_100 : 0
> vnode_map_update : 0
> vnode_map_update_total : 0
> vnode_map_update_time_mean : 0
> vnode_map_update_time_median : 0
> vnode_map_update_time_95 : 0
> vnode_map_update_time_99 : 0
> vnode_map_update_time_100 : 0
> node_gets : 0
> node_gets_total : 34978
> node_get_fsm_siblings_mean : 0
> node_get_fsm_siblings_median : 0
> node_get_fsm_siblings_95 : 0
> node_get_fsm_siblings_99 : 0
> node_get_fsm_siblings_100 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_time_mean : 0
> node_get_fsm_time_median : 0
> node_get_fsm_time_95 : 0
> node_get_fsm_time_99 : 0
> node_get_fsm_time_100 : 0
> node_gets_counter : 0
> node_gets_counter_total : 0
> node_get_fsm_counter_siblings_mean : 0
> node_get_fsm_counter_siblings_median : 0
> node_get_fsm_counter_siblings_95 : 0
> node_get_fsm_counter_siblings_99 : 0
> node_get_fsm_counter_siblings_100 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_time_mean : 0
> node_get_fsm_counter_time_median : 0
> node_get_fsm_counter_time_95 : 0
> node_get_fsm_counter_time_99 : 0
> node_get_fsm_counter_time_100 : 0
> node_gets_set : 0
> node_gets_set_total : 0
> node_get_fsm_set_siblings_mean : 0
> node_get_fsm_set_siblings_median : 0
> node_get_fsm_set_siblings_95 : 0
> node_get_fsm_set_siblings_99 : 0
> node_get_fsm_set_siblings_100 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_time_mean : 0
> node_get_fsm_set_time_median : 0
> node_get_fsm_set_time_95 : 0
> node_get_fsm_set_time_99 : 0
> node_get_fsm_set_time_100 : 0
> node_gets_map : 0
> node_gets_map_total : 0
> node_get_fsm_map_siblings_mean : 0
> node_get_fsm_map_siblings_median : 0
> node_get_fsm_map_siblings_95 : 0
> node_get_fsm_map_siblings_99 : 0
> node_get_fsm_map_siblings_100 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_time_mean : 0
> node_get_fsm_map_time_median : 0
> node_get_fsm_map_time_95 : 0
> node_get_fsm_map_time_99 : 0
> node_get_fsm_map_time_100 : 0
> node_puts : 0
> node_puts_total : 72584
> node_put_fsm_time_mean : 0
> node_put_fsm_time_median : 0
> node_put_fsm_time_95 : 0
> node_put_fsm_time_99 : 0
> node_put_fsm_time_100 : 0
> node_puts_counter : 0
> node_puts_counter_total : 0
> node_put_fsm_counter_time_mean : 0
> node_put_fsm_counter_time_median : 0
> node_put_fsm_counter_time_95 : 0
> node_put_fsm_counter_time_99 : 0
> node_put_fsm_counter_time_100 : 0
> node_puts_set : 0
> node_puts_set_total : 0
> node_put_fsm_set_time_mean : 0
> node_put_fsm_set_time_median : 0
> node_put_fsm_set_time_95 : 0
> node_put_fsm_set_time_99 : 0
> node_put_fsm_set_time_100 : 0
> node_puts_map : 0
> node_puts_map_total : 0
> node_put_fsm_map_time_mean : 0
> node_put_fsm_map_time_median : 0
> node_put_fsm_map_time_95 : 0
> node_put_fsm_map_time_99 : 0
> node_put_fsm_map_time_100 : 0
> read_repairs : 0
> read_repairs_total : 19461
> read_repairs_counter : 0
> read_repairs_counter_total : 0
> read_repairs_set : 0
> read_repairs_set_total : 0
> read_repairs_map : 0
> read_repairs_map_total : 0
> coord_redirs_total : 5258
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 0
> pbc_connects : 0
> pbc_connects_total : 1
> counter_actor_counts_mean : 0
> counter_actor_counts_median : 0
> counter_actor_counts_95 : 0
> counter_actor_counts_99 : 0
> counter_actor_counts_100 : 0
> set_actor_counts_mean : 0
> set_actor_counts_median : 0
> set_actor_counts_95 : 0
> set_actor_counts_99 : 0
> set_actor_counts_100 : 0
> map_actor_counts_mean : 0
> map_actor_counts_median : 0
> map_actor_counts_95 : 0
> map_actor_counts_99 : 0
> map_actor_counts_100 : 0
> late_put_fsm_coordinator_ack : 0
> consistent_gets : 0
> consistent_gets_total : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_100 : 0
> consistent_get_time_mean : 0
> consistent_get_time_median : 0
> consistent_get_time_95 : 0
> consistent_get_time_99 : 0
> consistent_get_time_100 : 0
> consistent_puts : 0
> consistent_puts_total : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_100 : 0
> consistent_put_time_mean : 0
> consistent_put_time_median : 0
> consistent_put_time_95 : 0
> consistent_put_time_99 : 0
> consistent_put_time_100 : 0
> node_get_fsm_active : 0
> node_get_fsm_active_60s : 0
> node_get_fsm_in_rate : 0
> node_get_fsm_out_rate : 0
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 0
> node_put_fsm_active_60s : 0
> node_put_fsm_in_rate : 0
> node_put_fsm_out_rate : 0
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 0
> read_repairs_primary_outofdate_count : 359
> read_repairs_primary_notfound_one : 0
> read_repairs_primary_notfound_count : 19143
> read_repairs_fallback_notfound_one : 0
> read_repairs_fallback_notfound_count : 7
> leveldb_read_block_error : 0
> riak_pipe_stat_ts : 1406455699
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> pipeline_active : 0
> cpu_nprocs : 307
> cpu_avg1 : 49
> cpu_avg5 : 95
> cpu_avg15 : 67
> mem_total : 2099634176
> mem_allocated : 2009845760
> disk : [{"/",41153856,67},
>         {"/sys/fs/cgroup",4,0},
>         {"/dev",1015140,1},
>         {"/run",205044,1},
>         {"/run/lock",5120,0},
>         {"/run/shm",1025212,0},
>         {"/run/user",102400,0}]
> nodename : 'riak at 10.128.224.15'
> connected_nodes : ['riak at 10.128.138.25','riak at 10.128.220.135',
>                    'riak at 10.128.179.86','riak at 10.128.133.59',
>                    'riak at 10.128.220.242','riak at 10.128.137.185']
> sys_driver_version : <<"2.2">>
> sys_global_heaps_size : deprecated
> sys_heap_type : private
> sys_logical_processors : 2
> sys_monitor_count : 112
> sys_otp_release : <<"R16B02-basho5">>
> sys_port_count : 52
> sys_process_count : 781
> sys_smp_support : true
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 64
> sys_wordsize : 8
> ring_members : ['riak at 10.128.133.59','riak at 10.128.137.185',
>                 'riak at 10.128.138.25','riak at 10.128.179.86',
>                 'riak at 10.128.220.135','riak at 10.128.220.242',
>                 'riak at 10.128.224.15']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak at 10.128.220.242',6},\n {'riak at 10.128.224.15',9},\n {'riak at 10.128.137.185',10},\n {'riak at 10.128.220.135',9},\n {'riak at 10.128.138.25',9},\n {'riak at 10.128.179.86',11},\n {'riak at 10.128.133.59',10}]">>
> ring_creation_size : 64
> storage_backend : riak_kv_eleveldb_backend
> riak_auth_mods_version : <<"2.0.0beta1-0-g6ae36b2">>
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"2.0.0beta1-6-g3fe302e">>
> cluster_info_version : <<"2.0.0beta1-0-g1fae829">>
> yokozuna_version : <<"2.0.0beta1-81-g47214a6">>
> ibrowse_version : <<"4.0.1">>
> riak_search_version : <<"2.0.0beta1-14-g535a88f">>
> merge_index_version : <<"2.0.0beta1-0-gb701dde">>
> riak_kv_version : <<"2.0.0beta1-175-gcdde007">>
> riak_api_version : <<"2.0.0beta1-9-g8aad9cd">>
> riak_pb_version : <<"2.0.0.16-0-gfc18a9b">>
> protobuffs_version : <<"0.8.1p4-0-g5257dfe">>
> riak_dt_version : <<"2.0.0beta2-4-gc06820d">>
> sidejob_version : <<"2.0.0beta1-0-gc5aabba">>
> riak_pipe_version : <<"2.0.0beta1-7-gbae5fc6">>
> riak_core_version : <<"2.0.0beta1-66-gddab30a">>
> poolboy_version : <<"0.8.1p2-0-g84d836a">>
> pbkdf2_version : <<"2.0.0-0-g7076584">>
> eleveldb_version : <<"2.0.0beta1-31-g8f1d1f6">>
> bitcask_version : <<"1.7.0">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.5-0-ge5f8233">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9.6">>
> erlang_js_version : <<"1.3.0-0-g07467d8">>
> runtime_tools_version : <<"1.8.12">>
> os_mon_version : <<"2.2.13">>
> riak_sysmon_version : <<"2.0.0">>
> ssl_version : <<"5.3.1">>
> public_key_version : <<"0.20">>
> crypto_version : <<"3.1">>
> asn1_version : <<"2.0.3">>
> sasl_version : <<"2.3.3">>
> lager_version : <<"2.0.3">>
> goldrush_version : <<"0.1.6">>
> compiler_version : <<"4.9.3">>
> syntax_tools_version : <<"1.6.11">>
> stdlib_version : <<"1.19.3">>
> kernel_version : <<"2.16.3">>
> memory_total : 106623536
> memory_processes : 61074600
> memory_processes_used : 61071728
> memory_system : 45548936
> memory_atom : 654217
> memory_atom_used : 644356
> memory_binary : 12450392
> memory_code : 15600373
> memory_ets : 7574504
> search_query_throughput_count : 0
> search_query_throughput_one : 0
> search_query_fail_count : 0
> search_query_fail_one : 0
> search_query_latency_min : 0
> search_query_latency_max : 0
> search_query_latency_median : 0
> search_query_latency_95 : 0
> search_query_latency_99 : 0
> search_query_latency_999 : 0
> search_index_throughput_count : 294514
> search_index_throughtput_one : 0
> search_index_fail_count : 4
> search_index_fail_one : 0
> search_index_latency_min : 0
> search_index_latency_max : 0
> search_index_latency_median : 0
> search_index_latency_95 : 0
> search_index_latency_99 : 0
> search_index_latency_999 : 0
> riak_core_stat_ts : 1406455701
> ignored_gossip_total : 0
> rings_reconciled_total : 529
> rings_reconciled : 0
> gossip_received : 3
> rejected_handoffs : 0
> handoff_timeouts : 2
> dropped_vnode_requests_total : 0
> converge_delay_min : 0
> converge_delay_max : 0
> converge_delay_mean : 0
> converge_delay_last : 0
> rebalance_delay_min : 0
> rebalance_delay_max : 0
> rebalance_delay_mean : 0
> rebalance_delay_last : 0
> riak_kv_vnodes_running : 9
> riak_kv_vnodeq_min : 0
> riak_kv_vnodeq_median : 0
> riak_kv_vnodeq_mean : 0
> riak_kv_vnodeq_max : 0
> riak_kv_vnodeq_total : 0
> riak_pipe_vnodes_running : 9
> riak_pipe_vnodeq_min : 0
> riak_pipe_vnodeq_median : 0
> riak_pipe_vnodeq_mean : 0
> riak_pipe_vnodeq_max : 0
> riak_pipe_vnodeq_total : 0
> 
> riak-admin status
> 1-minute stats for 'riak at 10.128.220.242'
> -------------------------------------------
> riak_kv_stat_ts : 1406455622
> vnode_gets : 12
> vnode_gets_total : 1465475
> vnode_puts : 12
> vnode_puts_total : 1651452
> vnode_index_refreshes : 0
> vnode_index_refreshes_total : 0
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 12
> vnode_index_writes_total : 1648507
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 942428
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> object_counter_merge : 0
> object_counter_merge_total : 0
> object_counter_merge_time_mean : 0
> object_counter_merge_time_median : 0
> object_counter_merge_time_95 : 0
> object_counter_merge_time_99 : 0
> object_counter_merge_time_100 : 0
> vnode_counter_update : 0
> vnode_counter_update_total : 0
> vnode_counter_update_time_mean : 0
> vnode_counter_update_time_median : 0
> vnode_counter_update_time_95 : 0
> vnode_counter_update_time_99 : 0
> vnode_counter_update_time_100 : 0
> object_set_merge : 0
> object_set_merge_total : 0
> object_set_merge_time_mean : 0
> object_set_merge_time_median : 0
> object_set_merge_time_95 : 0
> object_set_merge_time_99 : 0
> object_set_merge_time_100 : 0
> vnode_set_update : 0
> vnode_set_update_total : 0
> vnode_set_update_time_mean : 0
> vnode_set_update_time_median : 0
> vnode_set_update_time_95 : 0
> vnode_set_update_time_99 : 0
> vnode_set_update_time_100 : 0
> object_map_merge : 0
> object_map_merge_total : 0
> object_map_merge_time_mean : 0
> object_map_merge_time_median : 0
> object_map_merge_time_95 : 0
> object_map_merge_time_99 : 0
> object_map_merge_time_100 : 0
> object_merge : 0
> object_merge_total : 660670
> object_merge_time_mean : 0
> object_merge_time_median : 0
> object_merge_time_95 : 0
> object_merge_time_99 : 0
> object_merge_time_100 : 0
> vnode_map_update : 0
> vnode_map_update_total : 0
> vnode_map_update_time_mean : 0
> vnode_map_update_time_median : 0
> vnode_map_update_time_95 : 0
> vnode_map_update_time_99 : 0
> vnode_map_update_time_100 : 0
> node_gets : 0
> node_gets_total : 149
> node_get_fsm_siblings_mean : 0
> node_get_fsm_siblings_median : 0
> node_get_fsm_siblings_95 : 0
> node_get_fsm_siblings_99 : 0
> node_get_fsm_siblings_100 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_time_mean : 0
> node_get_fsm_time_median : 0
> node_get_fsm_time_95 : 0
> node_get_fsm_time_99 : 0
> node_get_fsm_time_100 : 0
> node_gets_counter : 0
> node_gets_counter_total : 0
> node_get_fsm_counter_siblings_mean : 0
> node_get_fsm_counter_siblings_median : 0
> node_get_fsm_counter_siblings_95 : 0
> node_get_fsm_counter_siblings_99 : 0
> node_get_fsm_counter_siblings_100 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_time_mean : 0
> node_get_fsm_counter_time_median : 0
> node_get_fsm_counter_time_95 : 0
> node_get_fsm_counter_time_99 : 0
> node_get_fsm_counter_time_100 : 0
> node_gets_set : 0
> node_gets_set_total : 0
> node_get_fsm_set_siblings_mean : 0
> node_get_fsm_set_siblings_median : 0
> node_get_fsm_set_siblings_95 : 0
> node_get_fsm_set_siblings_99 : 0
> node_get_fsm_set_siblings_100 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_time_mean : 0
> node_get_fsm_set_time_median : 0
> node_get_fsm_set_time_95 : 0
> node_get_fsm_set_time_99 : 0
> node_get_fsm_set_time_100 : 0
> node_gets_map : 0
> node_gets_map_total : 0
> node_get_fsm_map_siblings_mean : 0
> node_get_fsm_map_siblings_median : 0
> node_get_fsm_map_siblings_95 : 0
> node_get_fsm_map_siblings_99 : 0
> node_get_fsm_map_siblings_100 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_time_mean : 0
> node_get_fsm_map_time_median : 0
> node_get_fsm_map_time_95 : 0
> node_get_fsm_map_time_99 : 0
> node_get_fsm_map_time_100 : 0
> node_puts : 0
> node_puts_total : 7873
> node_put_fsm_time_mean : 0
> node_put_fsm_time_median : 0
> node_put_fsm_time_95 : 0
> node_put_fsm_time_99 : 0
> node_put_fsm_time_100 : 0
> node_puts_counter : 0
> node_puts_counter_total : 0
> node_put_fsm_counter_time_mean : 0
> node_put_fsm_counter_time_median : 0
> node_put_fsm_counter_time_95 : 0
> node_put_fsm_counter_time_99 : 0
> node_put_fsm_counter_time_100 : 0
> node_puts_set : 0
> node_puts_set_total : 0
> node_put_fsm_set_time_mean : 0
> node_put_fsm_set_time_median : 0
> node_put_fsm_set_time_95 : 0
> node_put_fsm_set_time_99 : 0
> node_put_fsm_set_time_100 : 0
> node_puts_map : 0
> node_puts_map_total : 0
> node_put_fsm_map_time_mean : 0
> node_put_fsm_map_time_median : 0
> node_put_fsm_map_time_95 : 0
> node_put_fsm_map_time_99 : 0
> node_put_fsm_map_time_100 : 0
> read_repairs : 0
> read_repairs_total : 149
> read_repairs_counter : 0
> read_repairs_counter_total : 0
> read_repairs_set : 0
> read_repairs_set_total : 0
> read_repairs_map : 0
> read_repairs_map_total : 0
> coord_redirs_total : 0
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 0
> pbc_connects : 0
> pbc_connects_total : 0
> counter_actor_counts_mean : 0
> counter_actor_counts_median : 0
> counter_actor_counts_95 : 0
> counter_actor_counts_99 : 0
> counter_actor_counts_100 : 0
> set_actor_counts_mean : 0
> set_actor_counts_median : 0
> set_actor_counts_95 : 0
> set_actor_counts_99 : 0
> set_actor_counts_100 : 0
> map_actor_counts_mean : 0
> map_actor_counts_median : 0
> map_actor_counts_95 : 0
> map_actor_counts_99 : 0
> map_actor_counts_100 : 0
> late_put_fsm_coordinator_ack : 0
> consistent_gets : 0
> consistent_gets_total : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_100 : 0
> consistent_get_time_mean : 0
> consistent_get_time_median : 0
> consistent_get_time_95 : 0
> consistent_get_time_99 : 0
> consistent_get_time_100 : 0
> consistent_puts : 0
> consistent_puts_total : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_100 : 0
> consistent_put_time_mean : 0
> consistent_put_time_median : 0
> consistent_put_time_95 : 0
> consistent_put_time_99 : 0
> consistent_put_time_100 : 0
> node_get_fsm_active : 0
> node_get_fsm_active_60s : 0
> node_get_fsm_in_rate : 0
> node_get_fsm_out_rate : 0
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 0
> node_put_fsm_active_60s : 0
> node_put_fsm_in_rate : 0
> node_put_fsm_out_rate : 0
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 0
> read_repairs_primary_outofdate_count : 212
> leveldb_read_block_error : 0
> riak_pipe_stat_ts : 1406455621
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> pipeline_active : 0
> cpu_nprocs : 314
> cpu_avg1 : 8
> cpu_avg5 : 31
> cpu_avg15 : 51
> mem_total : 2099634176
> mem_allocated : 2028318720
> disk : [{"/",41153856,54},
>         {"/sys/fs/cgroup",4,0},
>         {"/dev",1015140,1},
>         {"/run",205044,1},
>         {"/run/lock",5120,0},
>         {"/run/shm",1025212,0},
>         {"/run/user",102400,0}]
> nodename : 'riak at 10.128.220.242'
> connected_nodes : ['riak at 10.128.133.59','riak at 10.128.224.15',
>                    'riak at 10.128.220.135','riak at 10.128.179.86',
>                    'riak at 10.128.138.25','riak at 10.128.137.185']
> sys_driver_version : <<"2.2">>
> sys_global_heaps_size : deprecated
> sys_heap_type : private
> sys_logical_processors : 2
> sys_monitor_count : 185
> sys_otp_release : <<"R16B02-basho5">>
> sys_port_count : 52
> sys_process_count : 894
> sys_smp_support : true
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 64
> sys_wordsize : 8
> ring_members : ['riak at 10.128.133.59','riak at 10.128.137.185',
>                 'riak at 10.128.138.25','riak at 10.128.179.86',
>                 'riak at 10.128.220.135','riak at 10.128.220.242',
>                 'riak at 10.128.224.15']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak at 10.128.220.242',6},\n {'riak at 10.128.224.15',9},\n {'riak at 10.128.137.185',10},\n {'riak at 10.128.220.135',9},\n {'riak at 10.128.138.25',9},\n {'riak at 10.128.179.86',11},\n {'riak at 10.128.133.59',10}]">>
> ring_creation_size : 64
> storage_backend : riak_kv_eleveldb_backend
> riak_auth_mods_version : <<"2.0.0beta1-0-g6ae36b2">>
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"2.0.0beta1-6-g3fe302e">>
> cluster_info_version : <<"2.0.0beta1-0-g1fae829">>
> yokozuna_version : <<"2.0.0beta1-81-g47214a6">>
> ibrowse_version : <<"4.0.1">>
> riak_search_version : <<"2.0.0beta1-14-g535a88f">>
> merge_index_version : <<"2.0.0beta1-0-gb701dde">>
> riak_kv_version : <<"2.0.0beta1-175-gcdde007">>
> riak_api_version : <<"2.0.0beta1-9-g8aad9cd">>
> riak_pb_version : <<"2.0.0.16-0-gfc18a9b">>
> protobuffs_version : <<"0.8.1p4-0-g5257dfe">>
> riak_dt_version : <<"2.0.0beta2-4-gc06820d">>
> sidejob_version : <<"2.0.0beta1-0-gc5aabba">>
> riak_pipe_version : <<"2.0.0beta1-7-gbae5fc6">>
> riak_core_version : <<"2.0.0beta1-66-gddab30a">>
> poolboy_version : <<"0.8.1p2-0-g84d836a">>
> pbkdf2_version : <<"2.0.0-0-g7076584">>
> eleveldb_version : <<"2.0.0beta1-31-g8f1d1f6">>
> bitcask_version : <<"1.7.0">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.5-0-ge5f8233">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9.6">>
> erlang_js_version : <<"1.3.0-0-g07467d8">>
> runtime_tools_version : <<"1.8.12">>
> os_mon_version : <<"2.2.13">>
> riak_sysmon_version : <<"2.0.0">>
> ssl_version : <<"5.3.1">>
> public_key_version : <<"0.20">>
> crypto_version : <<"3.1">>
> asn1_version : <<"2.0.3">>
> sasl_version : <<"2.3.3">>
> lager_version : <<"2.0.3">>
> goldrush_version : <<"0.1.6">>
> compiler_version : <<"4.9.3">>
> syntax_tools_version : <<"1.6.11">>
> stdlib_version : <<"1.19.3">>
> kernel_version : <<"2.16.3">>
> memory_total : 67814288
> memory_processes : 24467072
> memory_processes_used : 24461400
> memory_system : 43347216
> memory_atom : 654217
> memory_atom_used : 630410
> memory_binary : 10874832
> memory_code : 14929594
> memory_ets : 7659008
> search_query_throughput_count : 0
> search_query_throughput_one : 0
> search_query_fail_count : 0
> search_query_fail_one : 0
> search_query_latency_min : 0
> search_query_latency_max : 0
> search_query_latency_median : 0
> search_query_latency_95 : 0
> search_query_latency_99 : 0
> search_query_latency_999 : 0
> search_index_throughput_count : 1648309
> search_index_throughtput_one : 12
> search_index_fail_count : 0
> search_index_fail_one : 0
> search_index_latency_min : 2231
> search_index_latency_max : 4940
> search_index_latency_median : 3460
> search_index_latency_95 : 4327
> search_index_latency_99 : 4940
> search_index_latency_999 : 4940
> riak_core_stat_ts : 1406455622
> ignored_gossip_total : 0
> rings_reconciled_total : 93
> rings_reconciled : 0
> gossip_received : 11
> rejected_handoffs : 23
> handoff_timeouts : 0
> dropped_vnode_requests_total : 0
> converge_delay_min : 0
> converge_delay_max : 0
> converge_delay_mean : 0
> converge_delay_last : 0
> rebalance_delay_min : 0
> rebalance_delay_max : 0
> rebalance_delay_mean : 0
> rebalance_delay_last : 0
> riak_kv_vnodes_running : 20
> riak_kv_vnodeq_min : 0
> riak_kv_vnodeq_median : 0
> riak_kv_vnodeq_mean : 0
> riak_kv_vnodeq_max : 0
> riak_kv_vnodeq_total : 0
> riak_pipe_vnodes_running : 9
> riak_pipe_vnodeq_min : 0
> riak_pipe_vnodeq_median : 0
> riak_pipe_vnodeq_mean : 0
> riak_pipe_vnodeq_max : 0
> riak_pipe_vnodeq_total : 0
> 
> 
> 1-minute stats for 'riak at 10.128.220.135'
> -------------------------------------------
> riak_kv_stat_ts : 1406455622
> vnode_gets : 13
> vnode_gets_total : 1770097
> vnode_puts : 1
> vnode_puts_total : 356166
> vnode_index_refreshes : 0
> vnode_index_refreshes_total : 0
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 1
> vnode_index_writes_total : 355104
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 0
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> object_counter_merge : 0
> object_counter_merge_total : 0
> object_counter_merge_time_mean : 0
> object_counter_merge_time_median : 0
> object_counter_merge_time_95 : 0
> object_counter_merge_time_99 : 0
> object_counter_merge_time_100 : 0
> vnode_counter_update : 0
> vnode_counter_update_total : 0
> vnode_counter_update_time_mean : 0
> vnode_counter_update_time_median : 0
> vnode_counter_update_time_95 : 0
> vnode_counter_update_time_99 : 0
> vnode_counter_update_time_100 : 0
> object_set_merge : 0
> object_set_merge_total : 0
> object_set_merge_time_mean : 0
> object_set_merge_time_median : 0
> object_set_merge_time_95 : 0
> object_set_merge_time_99 : 0
> object_set_merge_time_100 : 0
> vnode_set_update : 0
> vnode_set_update_total : 0
> vnode_set_update_time_mean : 0
> vnode_set_update_time_median : 0
> vnode_set_update_time_95 : 0
> vnode_set_update_time_99 : 0
> vnode_set_update_time_100 : 0
> object_map_merge : 0
> object_map_merge_total : 0
> object_map_merge_time_mean : 0
> object_map_merge_time_median : 0
> object_map_merge_time_95 : 0
> object_map_merge_time_99 : 0
> object_map_merge_time_100 : 0
> object_merge : 0
> object_merge_total : 2341
> object_merge_time_mean : 0
> object_merge_time_median : 0
> object_merge_time_95 : 0
> object_merge_time_99 : 0
> object_merge_time_100 : 0
> vnode_map_update : 0
> vnode_map_update_total : 0
> vnode_map_update_time_mean : 0
> vnode_map_update_time_median : 0
> vnode_map_update_time_95 : 0
> vnode_map_update_time_99 : 0
> vnode_map_update_time_100 : 0
> node_gets : 0
> node_gets_total : 336770
> node_get_fsm_siblings_mean : 0
> node_get_fsm_siblings_median : 0
> node_get_fsm_siblings_95 : 0
> node_get_fsm_siblings_99 : 0
> node_get_fsm_siblings_100 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_time_mean : 0
> node_get_fsm_time_median : 0
> node_get_fsm_time_95 : 0
> node_get_fsm_time_99 : 0
> node_get_fsm_time_100 : 0
> node_gets_counter : 0
> node_gets_counter_total : 0
> node_get_fsm_counter_siblings_mean : 0
> node_get_fsm_counter_siblings_median : 0
> node_get_fsm_counter_siblings_95 : 0
> node_get_fsm_counter_siblings_99 : 0
> node_get_fsm_counter_siblings_100 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_time_mean : 0
> node_get_fsm_counter_time_median : 0
> node_get_fsm_counter_time_95 : 0
> node_get_fsm_counter_time_99 : 0
> node_get_fsm_counter_time_100 : 0
> node_gets_set : 0
> node_gets_set_total : 0
> node_get_fsm_set_siblings_mean : 0
> node_get_fsm_set_siblings_median : 0
> node_get_fsm_set_siblings_95 : 0
> node_get_fsm_set_siblings_99 : 0
> node_get_fsm_set_siblings_100 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_time_mean : 0
> node_get_fsm_set_time_median : 0
> node_get_fsm_set_time_95 : 0
> node_get_fsm_set_time_99 : 0
> node_get_fsm_set_time_100 : 0
> node_gets_map : 0
> node_gets_map_total : 0
> node_get_fsm_map_siblings_mean : 0
> node_get_fsm_map_siblings_median : 0
> node_get_fsm_map_siblings_95 : 0
> node_get_fsm_map_siblings_99 : 0
> node_get_fsm_map_siblings_100 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_time_mean : 0
> node_get_fsm_map_time_median : 0
> node_get_fsm_map_time_95 : 0
> node_get_fsm_map_time_99 : 0
> node_get_fsm_map_time_100 : 0
> node_puts : 0
> node_puts_total : 209799
> node_put_fsm_time_mean : 0
> node_put_fsm_time_median : 0
> node_put_fsm_time_95 : 0
> node_put_fsm_time_99 : 0
> node_put_fsm_time_100 : 0
> node_puts_counter : 0
> node_puts_counter_total : 0
> node_put_fsm_counter_time_mean : 0
> node_put_fsm_counter_time_median : 0
> node_put_fsm_counter_time_95 : 0
> node_put_fsm_counter_time_99 : 0
> node_put_fsm_counter_time_100 : 0
> node_puts_set : 0
> node_puts_set_total : 0
> node_put_fsm_set_time_mean : 0
> node_put_fsm_set_time_median : 0
> node_put_fsm_set_time_95 : 0
> node_put_fsm_set_time_99 : 0
> node_put_fsm_set_time_100 : 0
> node_puts_map : 0
> node_puts_map_total : 0
> node_put_fsm_map_time_mean : 0
> node_put_fsm_map_time_median : 0
> node_put_fsm_map_time_95 : 0
> node_put_fsm_map_time_99 : 0
> node_put_fsm_map_time_100 : 0
> read_repairs : 0
> read_repairs_total : 136612
> read_repairs_counter : 0
> read_repairs_counter_total : 0
> read_repairs_set : 0
> read_repairs_set_total : 0
> read_repairs_map : 0
> read_repairs_map_total : 0
> coord_redirs_total : 92875
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 0
> pbc_connects : 0
> pbc_connects_total : 8
> counter_actor_counts_mean : 0
> counter_actor_counts_median : 0
> counter_actor_counts_95 : 0
> counter_actor_counts_99 : 0
> counter_actor_counts_100 : 0
> set_actor_counts_mean : 0
> set_actor_counts_median : 0
> set_actor_counts_95 : 0
> set_actor_counts_99 : 0
> set_actor_counts_100 : 0
> map_actor_counts_mean : 0
> map_actor_counts_median : 0
> map_actor_counts_95 : 0
> map_actor_counts_99 : 0
> map_actor_counts_100 : 0
> late_put_fsm_coordinator_ack : 0
> consistent_gets : 0
> consistent_gets_total : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_100 : 0
> consistent_get_time_mean : 0
> consistent_get_time_median : 0
> consistent_get_time_95 : 0
> consistent_get_time_99 : 0
> consistent_get_time_100 : 0
> consistent_puts : 0
> consistent_puts_total : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_100 : 0
> consistent_put_time_mean : 0
> consistent_put_time_median : 0
> consistent_put_time_95 : 0
> consistent_put_time_99 : 0
> consistent_put_time_100 : 0
> node_get_fsm_active : 0
> node_get_fsm_active_60s : 0
> node_get_fsm_in_rate : 0
> node_get_fsm_out_rate : 0
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 0
> node_put_fsm_active_60s : 0
> node_put_fsm_in_rate : 0
> node_put_fsm_out_rate : 0
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 0
> read_repairs_primary_outofdate_count : 4195
> read_repairs_primary_notfound_one : 0
> read_repairs_primary_notfound_count : 129405
> read_repairs_fallback_notfound_one : 0
> read_repairs_fallback_notfound_count : 4137
> leveldb_read_block_error : 0
> riak_pipe_stat_ts : 1406455622
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> pipeline_active : 0
> cpu_nprocs : 324
> cpu_avg1 : 261
> cpu_avg5 : 282
> cpu_avg15 : 289
> mem_total : 2099634176
> mem_allocated : 1952714752
> disk : [{"/",41153856,81},
>         {"/sys/fs/cgroup",4,0},
>         {"/dev",1015140,1},
>         {"/run",205044,1},
>         {"/run/lock",5120,0},
>         {"/run/shm",1025212,0},
>         {"/run/user",102400,0}]
> nodename : 'riak at 10.128.220.135'
> connected_nodes : ['riak at 10.128.179.86','riak at 10.128.138.25',
>                    'riak at 10.128.224.15','riak at 10.128.133.59',
>                    'riak at 10.128.220.242','riak at 10.128.137.185']
> sys_driver_version : <<"2.2">>
> sys_global_heaps_size : deprecated
> sys_heap_type : private
> sys_logical_processors : 2
> sys_monitor_count : 124
> sys_otp_release : <<"R16B02-basho5">>
> sys_port_count : 44
> sys_process_count : 817
> sys_smp_support : true
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 64
> sys_wordsize : 8
> ring_members : ['riak at 10.128.133.59','riak at 10.128.137.185',
>                 'riak at 10.128.138.25','riak at 10.128.179.86',
>                 'riak at 10.128.220.135','riak at 10.128.220.242',
>                 'riak at 10.128.224.15']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak at 10.128.220.242',6},\n {'riak at 10.128.224.15',9},\n {'riak at 10.128.137.185',10},\n {'riak at 10.128.220.135',9},\n {'riak at 10.128.138.25',9},\n {'riak at 10.128.179.86',11},\n {'riak at 10.128.133.59',10}]">>
> ring_creation_size : 64
> storage_backend : riak_kv_eleveldb_backend
> riak_auth_mods_version : <<"2.0.0beta1-0-g6ae36b2">>
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"2.0.0beta1-6-g3fe302e">>
> cluster_info_version : <<"2.0.0beta1-0-g1fae829">>
> yokozuna_version : <<"2.0.0beta1-81-g47214a6">>
> ibrowse_version : <<"4.0.1">>
> riak_search_version : <<"2.0.0beta1-14-g535a88f">>
> merge_index_version : <<"2.0.0beta1-0-gb701dde">>
> riak_kv_version : <<"2.0.0beta1-175-gcdde007">>
> riak_api_version : <<"2.0.0beta1-9-g8aad9cd">>
> riak_pb_version : <<"2.0.0.16-0-gfc18a9b">>
> protobuffs_version : <<"0.8.1p4-0-g5257dfe">>
> riak_dt_version : <<"2.0.0beta2-4-gc06820d">>
> sidejob_version : <<"2.0.0beta1-0-gc5aabba">>
> riak_pipe_version : <<"2.0.0beta1-7-gbae5fc6">>
> riak_core_version : <<"2.0.0beta1-66-gddab30a">>
> poolboy_version : <<"0.8.1p2-0-g84d836a">>
> pbkdf2_version : <<"2.0.0-0-g7076584">>
> eleveldb_version : <<"2.0.0beta1-31-g8f1d1f6">>
> bitcask_version : <<"1.7.0">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.5-0-ge5f8233">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9.6">>
> erlang_js_version : <<"1.3.0-0-g07467d8">>
> runtime_tools_version : <<"1.8.12">>
> os_mon_version : <<"2.2.13">>
> riak_sysmon_version : <<"2.0.0">>
> ssl_version : <<"5.3.1">>
> public_key_version : <<"0.20">>
> crypto_version : <<"3.1">>
> asn1_version : <<"2.0.3">>
> sasl_version : <<"2.3.3">>
> lager_version : <<"2.0.3">>
> goldrush_version : <<"0.1.6">>
> compiler_version : <<"4.9.3">>
> syntax_tools_version : <<"1.6.11">>
> stdlib_version : <<"1.19.3">>
> kernel_version : <<"2.16.3">>
> memory_total : 109437920
> memory_processes : 67156600
> memory_processes_used : 67155832
> memory_system : 42281320
> memory_atom : 654217
> memory_atom_used : 644986
> memory_binary : 8593280
> memory_code : 15609973
> memory_ets : 7849720
> search_query_throughput_count : 0
> search_query_throughput_one : 0
> search_query_fail_count : 0
> search_query_fail_one : 0
> search_query_latency_min : 0
> search_query_latency_max : 0
> search_query_latency_median : 0
> search_query_latency_95 : 0
> search_query_latency_99 : 0
> search_query_latency_999 : 0
> search_index_throughput_count : 326156
> search_index_throughtput_one : 0
> search_index_fail_count : 25849
> search_index_fail_one : 1
> search_index_latency_min : 0
> search_index_latency_max : 0
> search_index_latency_median : 0
> search_index_latency_95 : 0
> search_index_latency_99 : 0
> search_index_latency_999 : 0
> riak_core_stat_ts : 1406455621
> ignored_gossip_total : 0
> rings_reconciled_total : 714
> rings_reconciled : 0
> gossip_received : 5
> rejected_handoffs : 0
> handoff_timeouts : 4
> dropped_vnode_requests_total : 19199
> converge_delay_min : 0
> converge_delay_max : 0
> converge_delay_mean : 0
> converge_delay_last : 0
> rebalance_delay_min : 0
> rebalance_delay_max : 0
> rebalance_delay_mean : 0
> rebalance_delay_last : 0
> riak_kv_vnodes_running : 10
> riak_kv_vnodeq_min : 0
> riak_kv_vnodeq_median : 0
> riak_kv_vnodeq_mean : 993
> riak_kv_vnodeq_max : 9930
> riak_kv_vnodeq_total : 9930
> riak_pipe_vnodes_running : 9
> riak_pipe_vnodeq_min : 0
> riak_pipe_vnodeq_median : 0
> riak_pipe_vnodeq_mean : 0
> riak_pipe_vnodeq_max : 0
> riak_pipe_vnodeq_total : 0
> 
> riak-admin status
> 1-minute stats for 'riak at 10.128.179.86'
> -------------------------------------------
> riak_kv_stat_ts : 1406455612
> vnode_gets : 14
> vnode_gets_total : 182305
> vnode_puts : 2
> vnode_puts_total : 95860
> vnode_index_refreshes : 0
> vnode_index_refreshes_total : 0
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 2
> vnode_index_writes_total : 94729
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 0
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> object_counter_merge : 0
> object_counter_merge_total : 0
> object_counter_merge_time_mean : 0
> object_counter_merge_time_median : 0
> object_counter_merge_time_95 : 0
> object_counter_merge_time_99 : 0
> object_counter_merge_time_100 : 0
> vnode_counter_update : 0
> vnode_counter_update_total : 0
> vnode_counter_update_time_mean : 0
> vnode_counter_update_time_median : 0
> vnode_counter_update_time_95 : 0
> vnode_counter_update_time_99 : 0
> vnode_counter_update_time_100 : 0
> object_set_merge : 0
> object_set_merge_total : 0
> object_set_merge_time_mean : 0
> object_set_merge_time_median : 0
> object_set_merge_time_95 : 0
> object_set_merge_time_99 : 0
> object_set_merge_time_100 : 0
> vnode_set_update : 0
> vnode_set_update_total : 0
> vnode_set_update_time_mean : 0
> vnode_set_update_time_median : 0
> vnode_set_update_time_95 : 0
> vnode_set_update_time_99 : 0
> vnode_set_update_time_100 : 0
> object_map_merge : 0
> object_map_merge_total : 0
> object_map_merge_time_mean : 0
> object_map_merge_time_median : 0
> object_map_merge_time_95 : 0
> object_map_merge_time_99 : 0
> object_map_merge_time_100 : 0
> object_merge : 0
> object_merge_total : 696
> object_merge_time_mean : 0
> object_merge_time_median : 0
> object_merge_time_95 : 0
> object_merge_time_99 : 0
> object_merge_time_100 : 0
> vnode_map_update : 0
> vnode_map_update_total : 0
> vnode_map_update_time_mean : 0
> vnode_map_update_time_median : 0
> vnode_map_update_time_95 : 0
> vnode_map_update_time_99 : 0
> vnode_map_update_time_100 : 0
> node_gets : 0
> node_gets_total : 18739
> node_get_fsm_siblings_mean : 0
> node_get_fsm_siblings_median : 0
> node_get_fsm_siblings_95 : 0
> node_get_fsm_siblings_99 : 0
> node_get_fsm_siblings_100 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_time_mean : 0
> node_get_fsm_time_median : 0
> node_get_fsm_time_95 : 0
> node_get_fsm_time_99 : 0
> node_get_fsm_time_100 : 0
> node_gets_counter : 0
> node_gets_counter_total : 0
> node_get_fsm_counter_siblings_mean : 0
> node_get_fsm_counter_siblings_median : 0
> node_get_fsm_counter_siblings_95 : 0
> node_get_fsm_counter_siblings_99 : 0
> node_get_fsm_counter_siblings_100 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_time_mean : 0
> node_get_fsm_counter_time_median : 0
> node_get_fsm_counter_time_95 : 0
> node_get_fsm_counter_time_99 : 0
> node_get_fsm_counter_time_100 : 0
> node_gets_set : 0
> node_gets_set_total : 0
> node_get_fsm_set_siblings_mean : 0
> node_get_fsm_set_siblings_median : 0
> node_get_fsm_set_siblings_95 : 0
> node_get_fsm_set_siblings_99 : 0
> node_get_fsm_set_siblings_100 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_time_mean : 0
> node_get_fsm_set_time_median : 0
> node_get_fsm_set_time_95 : 0
> node_get_fsm_set_time_99 : 0
> node_get_fsm_set_time_100 : 0
> node_gets_map : 0
> node_gets_map_total : 0
> node_get_fsm_map_siblings_mean : 0
> node_get_fsm_map_siblings_median : 0
> node_get_fsm_map_siblings_95 : 0
> node_get_fsm_map_siblings_99 : 0
> node_get_fsm_map_siblings_100 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_time_mean : 0
> node_get_fsm_map_time_median : 0
> node_get_fsm_map_time_95 : 0
> node_get_fsm_map_time_99 : 0
> node_get_fsm_map_time_100 : 0
> node_puts : 0
> node_puts_total : 3819
> node_put_fsm_time_mean : 0
> node_put_fsm_time_median : 0
> node_put_fsm_time_95 : 0
> node_put_fsm_time_99 : 0
> node_put_fsm_time_100 : 0
> node_puts_counter : 0
> node_puts_counter_total : 0
> node_put_fsm_counter_time_mean : 0
> node_put_fsm_counter_time_median : 0
> node_put_fsm_counter_time_95 : 0
> node_put_fsm_counter_time_99 : 0
> node_put_fsm_counter_time_100 : 0
> node_puts_set : 0
> node_puts_set_total : 0
> node_put_fsm_set_time_mean : 0
> node_put_fsm_set_time_median : 0
> node_put_fsm_set_time_95 : 0
> node_put_fsm_set_time_99 : 0
> node_put_fsm_set_time_100 : 0
> node_puts_map : 0
> node_puts_map_total : 0
> node_put_fsm_map_time_mean : 0
> node_put_fsm_map_time_median : 0
> node_put_fsm_map_time_95 : 0
> node_put_fsm_map_time_99 : 0
> node_put_fsm_map_time_100 : 0
> read_repairs : 0
> read_repairs_total : 5401
> read_repairs_counter : 0
> read_repairs_counter_total : 0
> read_repairs_set : 0
> read_repairs_set_total : 0
> read_repairs_map : 0
> read_repairs_map_total : 0
> coord_redirs_total : 10699
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 0
> pbc_connects : 0
> pbc_connects_total : 7
> counter_actor_counts_mean : 0
> counter_actor_counts_median : 0
> counter_actor_counts_95 : 0
> counter_actor_counts_99 : 0
> counter_actor_counts_100 : 0
> set_actor_counts_mean : 0
> set_actor_counts_median : 0
> set_actor_counts_95 : 0
> set_actor_counts_99 : 0
> set_actor_counts_100 : 0
> map_actor_counts_mean : 0
> map_actor_counts_median : 0
> map_actor_counts_95 : 0
> map_actor_counts_99 : 0
> map_actor_counts_100 : 0
> late_put_fsm_coordinator_ack : 0
> consistent_gets : 0
> consistent_gets_total : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_100 : 0
> consistent_get_time_mean : 0
> consistent_get_time_median : 0
> consistent_get_time_95 : 0
> consistent_get_time_99 : 0
> consistent_get_time_100 : 0
> consistent_puts : 0
> consistent_puts_total : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_100 : 0
> consistent_put_time_mean : 0
> consistent_put_time_median : 0
> consistent_put_time_95 : 0
> consistent_put_time_99 : 0
> consistent_put_time_100 : 0
> node_get_fsm_active : 0
> node_get_fsm_active_60s : 0
> node_get_fsm_in_rate : 0
> node_get_fsm_out_rate : 0
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 0
> node_put_fsm_active_60s : 0
> node_put_fsm_in_rate : 0
> node_put_fsm_out_rate : 0
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 0
> read_repairs_primary_outofdate_count : 420
> read_repairs_primary_notfound_one : 0
> read_repairs_primary_notfound_count : 3897
> read_repairs_fallback_notfound_one : 0
> read_repairs_fallback_notfound_count : 1237
> leveldb_read_block_error : <<"stale: 0">>
> riak_pipe_stat_ts : 1406455611
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> pipeline_active : 0
> cpu_nprocs : 301
> cpu_avg1 : 20
> cpu_avg5 : 26
> cpu_avg15 : 33
> mem_total : 2099634176
> mem_allocated : 1993543680
> disk : [{"/",41153856,80},
>         {"/sys/fs/cgroup",4,0},
>         {"/dev",1015140,1},
>         {"/run",205044,1},
>         {"/run/lock",5120,0},
>         {"/run/shm",1025212,0},
>         {"/run/user",102400,0}]
> nodename : 'riak at 10.128.179.86'
> connected_nodes : ['riak at 10.128.138.25','riak at 10.128.220.135',
>                    'riak at 10.128.224.15','riak at 10.128.133.59',
>                    'riak at 10.128.220.242','riak at 10.128.137.185']
> sys_driver_version : <<"2.2">>
> sys_global_heaps_size : deprecated
> sys_heap_type : private
> sys_logical_processors : 2
> sys_monitor_count : 183
> sys_otp_release : <<"R16B02-basho5">>
> sys_port_count : 52
> sys_process_count : 1046
> sys_smp_support : true
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 64
> sys_wordsize : 8
> ring_members : ['riak at 10.128.133.59','riak at 10.128.137.185',
>                 'riak at 10.128.138.25','riak at 10.128.179.86',
>                 'riak at 10.128.220.135','riak at 10.128.220.242',
>                 'riak at 10.128.224.15']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak at 10.128.220.242',6},\n {'riak at 10.128.224.15',9},\n {'riak at 10.128.137.185',10},\n {'riak at 10.128.220.135',9},\n {'riak at 10.128.138.25',9},\n {'riak at 10.128.179.86',11},\n {'riak at 10.128.133.59',10}]">>
> ring_creation_size : 64
> storage_backend : riak_kv_eleveldb_backend
> riak_auth_mods_version : <<"2.0.0beta1-0-g6ae36b2">>
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"2.0.0beta1-6-g3fe302e">>
> cluster_info_version : <<"2.0.0beta1-0-g1fae829">>
> yokozuna_version : <<"2.0.0beta1-81-g47214a6">>
> ibrowse_version : <<"4.0.1">>
> riak_search_version : <<"2.0.0beta1-14-g535a88f">>
> merge_index_version : <<"2.0.0beta1-0-gb701dde">>
> riak_kv_version : <<"2.0.0beta1-175-gcdde007">>
> riak_api_version : <<"2.0.0beta1-9-g8aad9cd">>
> riak_pb_version : <<"2.0.0.16-0-gfc18a9b">>
> protobuffs_version : <<"0.8.1p4-0-g5257dfe">>
> riak_dt_version : <<"2.0.0beta2-4-gc06820d">>
> sidejob_version : <<"2.0.0beta1-0-gc5aabba">>
> riak_pipe_version : <<"2.0.0beta1-7-gbae5fc6">>
> riak_core_version : <<"2.0.0beta1-66-gddab30a">>
> poolboy_version : <<"0.8.1p2-0-g84d836a">>
> pbkdf2_version : <<"2.0.0-0-g7076584">>
> eleveldb_version : <<"2.0.0beta1-31-g8f1d1f6">>
> bitcask_version : <<"1.7.0">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.5-0-ge5f8233">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9.6">>
> erlang_js_version : <<"1.3.0-0-g07467d8">>
> runtime_tools_version : <<"1.8.12">>
> os_mon_version : <<"2.2.13">>
> riak_sysmon_version : <<"2.0.0">>
> ssl_version : <<"5.3.1">>
> public_key_version : <<"0.20">>
> crypto_version : <<"3.1">>
> asn1_version : <<"2.0.3">>
> sasl_version : <<"2.3.3">>
> lager_version : <<"2.0.3">>
> goldrush_version : <<"0.1.6">>
> compiler_version : <<"4.9.3">>
> syntax_tools_version : <<"1.6.11">>
> stdlib_version : <<"1.19.3">>
> kernel_version : <<"2.16.3">>
> memory_total : 157392936
> memory_processes : 96613096
> memory_processes_used : 96612984
> memory_system : 60779840
> memory_atom : 654217
> memory_atom_used : 645382
> memory_binary : 25483288
> memory_code : 15600373
> memory_ets : 8203000
> search_query_throughput_count : 0
> search_query_throughput_one : 0
> search_query_fail_count : 0
> search_query_fail_one : 0
> search_query_latency_min : 0
> search_query_latency_max : 0
> search_query_latency_median : 0
> search_query_latency_95 : 0
> search_query_latency_99 : 0
> search_query_latency_999 : 0
> search_index_throughput_count : 4767
> search_index_throughtput_one : 0
> search_index_fail_count : 88617
> search_index_fail_one : 2
> search_index_latency_min : 0
> search_index_latency_max : 0
> search_index_latency_median : 0
> search_index_latency_95 : 0
> search_index_latency_99 : 0
> search_index_latency_999 : 0
> riak_core_stat_ts : 1406455621
> ignored_gossip_total : 0
> rings_reconciled_total : 869
> rings_reconciled : 0
> gossip_received : 4
> rejected_handoffs : 3
> handoff_timeouts : 0
> dropped_vnode_requests_total : 305046
> converge_delay_min : 0
> converge_delay_max : 0
> converge_delay_mean : 0
> converge_delay_last : 0
> rebalance_delay_min : 0
> rebalance_delay_max : 0
> rebalance_delay_mean : 0
> rebalance_delay_last : 0
> riak_kv_vnodes_running : 21
> riak_kv_vnodeq_min : 0
> riak_kv_vnodeq_median : 0
> riak_kv_vnodeq_mean : 2343
> riak_kv_vnodeq_max : 29334
> riak_kv_vnodeq_total : 49222
> riak_pipe_vnodes_running : 9
> riak_pipe_vnodeq_min : 0
> riak_pipe_vnodeq_median : 0
> riak_pipe_vnodeq_mean : 0
> riak_pipe_vnodeq_max : 0
> riak_pipe_vnodeq_total : 0
> 
> 
> 1-minute stats for 'riak at 10.128.137.185'
> -------------------------------------------
> riak_kv_stat_ts : 1406455619
> vnode_gets : 2
> vnode_gets_total : 58645
> vnode_puts : 2
> vnode_puts_total : 44071
> vnode_index_refreshes : 0
> vnode_index_refreshes_total : 0
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 2
> vnode_index_writes_total : 43836
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 0
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> object_counter_merge : 0
> object_counter_merge_total : 0
> object_counter_merge_time_mean : 0
> object_counter_merge_time_median : 0
> object_counter_merge_time_95 : 0
> object_counter_merge_time_99 : 0
> object_counter_merge_time_100 : 0
> vnode_counter_update : 0
> vnode_counter_update_total : 0
> vnode_counter_update_time_mean : 0
> vnode_counter_update_time_median : 0
> vnode_counter_update_time_95 : 0
> vnode_counter_update_time_99 : 0
> vnode_counter_update_time_100 : 0
> object_set_merge : 0
> object_set_merge_total : 0
> object_set_merge_time_mean : 0
> object_set_merge_time_median : 0
> object_set_merge_time_95 : 0
> object_set_merge_time_99 : 0
> object_set_merge_time_100 : 0
> vnode_set_update : 0
> vnode_set_update_total : 0
> vnode_set_update_time_mean : 0
> vnode_set_update_time_median : 0
> vnode_set_update_time_95 : 0
> vnode_set_update_time_99 : 0
> vnode_set_update_time_100 : 0
> object_map_merge : 0
> object_map_merge_total : 0
> object_map_merge_time_mean : 0
> object_map_merge_time_median : 0
> object_map_merge_time_95 : 0
> object_map_merge_time_99 : 0
> object_map_merge_time_100 : 0
> object_merge : 1
> object_merge_total : 424
> object_merge_time_mean : 0
> object_merge_time_median : 0
> object_merge_time_95 : 0
> object_merge_time_99 : 0
> object_merge_time_100 : 0
> vnode_map_update : 0
> vnode_map_update_total : 0
> vnode_map_update_time_mean : 0
> vnode_map_update_time_median : 0
> vnode_map_update_time_95 : 0
> vnode_map_update_time_99 : 0
> vnode_map_update_time_100 : 0
> node_gets : 0
> node_gets_total : 5507
> node_get_fsm_siblings_mean : 0
> node_get_fsm_siblings_median : 0
> node_get_fsm_siblings_95 : 0
> node_get_fsm_siblings_99 : 0
> node_get_fsm_siblings_100 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_time_mean : 0
> node_get_fsm_time_median : 0
> node_get_fsm_time_95 : 0
> node_get_fsm_time_99 : 0
> node_get_fsm_time_100 : 0
> node_gets_counter : 0
> node_gets_counter_total : 0
> node_get_fsm_counter_siblings_mean : 0
> node_get_fsm_counter_siblings_median : 0
> node_get_fsm_counter_siblings_95 : 0
> node_get_fsm_counter_siblings_99 : 0
> node_get_fsm_counter_siblings_100 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_time_mean : 0
> node_get_fsm_counter_time_median : 0
> node_get_fsm_counter_time_95 : 0
> node_get_fsm_counter_time_99 : 0
> node_get_fsm_counter_time_100 : 0
> node_gets_set : 0
> node_gets_set_total : 0
> node_get_fsm_set_siblings_mean : 0
> node_get_fsm_set_siblings_median : 0
> node_get_fsm_set_siblings_95 : 0
> node_get_fsm_set_siblings_99 : 0
> node_get_fsm_set_siblings_100 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_time_mean : 0
> node_get_fsm_set_time_median : 0
> node_get_fsm_set_time_95 : 0
> node_get_fsm_set_time_99 : 0
> node_get_fsm_set_time_100 : 0
> node_gets_map : 0
> node_gets_map_total : 0
> node_get_fsm_map_siblings_mean : 0
> node_get_fsm_map_siblings_median : 0
> node_get_fsm_map_siblings_95 : 0
> node_get_fsm_map_siblings_99 : 0
> node_get_fsm_map_siblings_100 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_time_mean : 0
> node_get_fsm_map_time_median : 0
> node_get_fsm_map_time_95 : 0
> node_get_fsm_map_time_99 : 0
> node_get_fsm_map_time_100 : 0
> node_puts : 0
> node_puts_total : 2019
> node_put_fsm_time_mean : 0
> node_put_fsm_time_median : 0
> node_put_fsm_time_95 : 0
> node_put_fsm_time_99 : 0
> node_put_fsm_time_100 : 0
> node_puts_counter : 0
> node_puts_counter_total : 0
> node_put_fsm_counter_time_mean : 0
> node_put_fsm_counter_time_median : 0
> node_put_fsm_counter_time_95 : 0
> node_put_fsm_counter_time_99 : 0
> node_put_fsm_counter_time_100 : 0
> node_puts_set : 0
> node_puts_set_total : 0
> node_put_fsm_set_time_mean : 0
> node_put_fsm_set_time_median : 0
> node_put_fsm_set_time_95 : 0
> node_put_fsm_set_time_99 : 0
> node_put_fsm_set_time_100 : 0
> node_puts_map : 0
> node_puts_map_total : 0
> node_put_fsm_map_time_mean : 0
> node_put_fsm_map_time_median : 0
> node_put_fsm_map_time_95 : 0
> node_put_fsm_map_time_99 : 0
> node_put_fsm_map_time_100 : 0
> read_repairs : 0
> read_repairs_total : 1349
> read_repairs_counter : 0
> read_repairs_counter_total : 0
> read_repairs_set : 0
> read_repairs_set_total : 0
> read_repairs_map : 0
> read_repairs_map_total : 0
> coord_redirs_total : 0
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 0
> pbc_connects : 0
> pbc_connects_total : 0
> counter_actor_counts_mean : 0
> counter_actor_counts_median : 0
> counter_actor_counts_95 : 0
> counter_actor_counts_99 : 0
> counter_actor_counts_100 : 0
> set_actor_counts_mean : 0
> set_actor_counts_median : 0
> set_actor_counts_95 : 0
> set_actor_counts_99 : 0
> set_actor_counts_100 : 0
> map_actor_counts_mean : 0
> map_actor_counts_median : 0
> map_actor_counts_95 : 0
> map_actor_counts_99 : 0
> map_actor_counts_100 : 0
> late_put_fsm_coordinator_ack : 0
> consistent_gets : 0
> consistent_gets_total : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_100 : 0
> consistent_get_time_mean : 0
> consistent_get_time_median : 0
> consistent_get_time_95 : 0
> consistent_get_time_99 : 0
> consistent_get_time_100 : 0
> consistent_puts : 0
> consistent_puts_total : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_100 : 0
> consistent_put_time_mean : 0
> consistent_put_time_median : 0
> consistent_put_time_95 : 0
> consistent_put_time_99 : 0
> consistent_put_time_100 : 0
> node_get_fsm_active : 0
> node_get_fsm_active_60s : 0
> node_get_fsm_in_rate : 0
> node_get_fsm_out_rate : 0
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 0
> node_put_fsm_active_60s : 0
> node_put_fsm_in_rate : 0
> node_put_fsm_out_rate : 0
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 0
> read_repairs_primary_outofdate_count : 1376
> leveldb_read_block_error : <<"stale: 0">>
> riak_pipe_stat_ts : 1406455619
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> pipeline_active : 0
> cpu_nprocs : 315
> cpu_avg1 : 4925
> cpu_avg5 : 5002
> cpu_avg15 : 5079
> mem_total : 2099634176
> mem_allocated : 2009513984
> disk : [{"/",41153856,78},
>         {"/sys/fs/cgroup",4,0},
>         {"/dev",1015140,1},
>         {"/run",205044,1},
>         {"/run/lock",5120,0},
>         {"/run/shm",1025212,0},
>         {"/run/user",102400,0}]
> nodename : 'riak at 10.128.137.185'
> connected_nodes : ['riak at 10.128.179.86','riak at 10.128.224.15',
>                    'riak at 10.128.138.25','riak at 10.128.133.59',
>                    'riak at 10.128.220.242','riak at 10.128.220.135']
> sys_driver_version : <<"2.2">>
> sys_global_heaps_size : deprecated
> sys_heap_type : private
> sys_logical_processors : 2
> sys_monitor_count : 157
> sys_otp_release : <<"R16B02-basho5">>
> sys_port_count : 52
> sys_process_count : 834
> sys_smp_support : true
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 64
> sys_wordsize : 8
> ring_members : ['riak at 10.128.133.59','riak at 10.128.137.185',
>                 'riak at 10.128.138.25','riak at 10.128.179.86',
>                 'riak at 10.128.220.135','riak at 10.128.220.242',
>                 'riak at 10.128.224.15']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak at 10.128.220.242',6},\n {'riak at 10.128.224.15',9},\n {'riak at 10.128.137.185',10},\n {'riak at 10.128.220.135',9},\n {'riak at 10.128.138.25',9},\n {'riak at 10.128.179.86',11},\n {'riak at 10.128.133.59',10}]">>
> ring_creation_size : 64
> storage_backend : riak_kv_eleveldb_backend
> riak_auth_mods_version : <<"2.0.0beta1-0-g6ae36b2">>
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"2.0.0beta1-6-g3fe302e">>
> cluster_info_version : <<"2.0.0beta1-0-g1fae829">>
> yokozuna_version : <<"2.0.0beta1-81-g47214a6">>
> ibrowse_version : <<"4.0.1">>
> riak_search_version : <<"2.0.0beta1-14-g535a88f">>
> merge_index_version : <<"2.0.0beta1-0-gb701dde">>
> riak_kv_version : <<"2.0.0beta1-175-gcdde007">>
> riak_api_version : <<"2.0.0beta1-9-g8aad9cd">>
> riak_pb_version : <<"2.0.0.16-0-gfc18a9b">>
> protobuffs_version : <<"0.8.1p4-0-g5257dfe">>
> riak_dt_version : <<"2.0.0beta2-4-gc06820d">>
> sidejob_version : <<"2.0.0beta1-0-gc5aabba">>
> riak_pipe_version : <<"2.0.0beta1-7-gbae5fc6">>
> riak_core_version : <<"2.0.0beta1-66-gddab30a">>
> poolboy_version : <<"0.8.1p2-0-g84d836a">>
> pbkdf2_version : <<"2.0.0-0-g7076584">>
> eleveldb_version : <<"2.0.0beta1-31-g8f1d1f6">>
> bitcask_version : <<"1.7.0">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.5-0-ge5f8233">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9.6">>
> erlang_js_version : <<"1.3.0-0-g07467d8">>
> runtime_tools_version : <<"1.8.12">>
> os_mon_version : <<"2.2.13">>
> riak_sysmon_version : <<"2.0.0">>
> ssl_version : <<"5.3.1">>
> public_key_version : <<"0.20">>
> crypto_version : <<"3.1">>
> asn1_version : <<"2.0.3">>
> sasl_version : <<"2.3.3">>
> lager_version : <<"2.0.3">>
> goldrush_version : <<"0.1.6">>
> compiler_version : <<"4.9.3">>
> syntax_tools_version : <<"1.6.11">>
> stdlib_version : <<"1.19.3">>
> kernel_version : <<"2.16.3">>
> memory_total : 124850656
> memory_processes : 74114648
> memory_processes_used : 74112376
> memory_system : 50736008
> memory_atom : 594537
> memory_atom_used : 562883
> memory_binary : 18911520
> memory_code : 14655749
> memory_ets : 7067440
> search_query_throughput_count : 0
> search_query_throughput_one : 0
> search_query_fail_count : 0
> search_query_fail_one : 0
> search_query_latency_min : 0
> search_query_latency_max : 0
> search_query_latency_median : 0
> search_query_latency_95 : 0
> search_query_latency_99 : 0
> search_query_latency_999 : 0
> search_index_throughput_count : 6665
> search_index_throughtput_one : 0
> search_index_fail_count : 37171
> search_index_fail_one : 2
> search_index_latency_min : 0
> search_index_latency_max : 0
> search_index_latency_median : 0
> search_index_latency_95 : 0
> search_index_latency_99 : 0
> search_index_latency_999 : 0
> riak_core_stat_ts : 1406455622
> ignored_gossip_total : 0
> rings_reconciled_total : 37
> rings_reconciled : 0
> gossip_received : 4
> rejected_handoffs : 131
> handoff_timeouts : 1
> dropped_vnode_requests_total : 6389
> converge_delay_min : 0
> converge_delay_max : 0
> converge_delay_mean : 0
> converge_delay_last : 62568
> rebalance_delay_min : 0
> rebalance_delay_max : 0
> rebalance_delay_mean : 0
> rebalance_delay_last : 0
> riak_kv_vnodes_running : 15
> riak_kv_vnodeq_min : 0
> riak_kv_vnodeq_median : 0
> riak_kv_vnodeq_mean : 744
> riak_kv_vnodeq_max : 5111
> riak_kv_vnodeq_total : 11172
> riak_pipe_vnodes_running : 11
> riak_pipe_vnodeq_min : 0
> riak_pipe_vnodeq_median : 0
> riak_pipe_vnodeq_mean : 0
> riak_pipe_vnodeq_max : 0
> riak_pipe_vnodeq_total : 0
> 
> 
> 1-minute stats for 'riak at 10.128.138.25'
> -------------------------------------------
> riak_kv_stat_ts : 1406455621
> vnode_gets : 0
> vnode_gets_total : 1844794
> vnode_puts : 0
> vnode_puts_total : 368521
> vnode_index_refreshes : 0
> vnode_index_refreshes_total : 0
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 0
> vnode_index_writes_total : 367834
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 0
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> object_counter_merge : 0
> object_counter_merge_total : 0
> object_counter_merge_time_mean : 0
> object_counter_merge_time_median : 0
> object_counter_merge_time_95 : 0
> object_counter_merge_time_99 : 0
> object_counter_merge_time_100 : 0
> vnode_counter_update : 0
> vnode_counter_update_total : 0
> vnode_counter_update_time_mean : 0
> vnode_counter_update_time_median : 0
> vnode_counter_update_time_95 : 0
> vnode_counter_update_time_99 : 0
> vnode_counter_update_time_100 : 0
> object_set_merge : 0
> object_set_merge_total : 0
> object_set_merge_time_mean : 0
> object_set_merge_time_median : 0
> object_set_merge_time_95 : 0
> object_set_merge_time_99 : 0
> object_set_merge_time_100 : 0
> vnode_set_update : 0
> vnode_set_update_total : 0
> vnode_set_update_time_mean : 0
> vnode_set_update_time_median : 0
> vnode_set_update_time_95 : 0
> vnode_set_update_time_99 : 0
> vnode_set_update_time_100 : 0
> object_map_merge : 0
> object_map_merge_total : 0
> object_map_merge_time_mean : 0
> object_map_merge_time_median : 0
> object_map_merge_time_95 : 0
> object_map_merge_time_99 : 0
> object_map_merge_time_100 : 0
> object_merge : 0
> object_merge_total : 1844
> object_merge_time_mean : 0
> object_merge_time_median : 0
> object_merge_time_95 : 0
> object_merge_time_99 : 0
> object_merge_time_100 : 0
> vnode_map_update : 0
> vnode_map_update_total : 0
> vnode_map_update_time_mean : 0
> vnode_map_update_time_median : 0
> vnode_map_update_time_95 : 0
> vnode_map_update_time_99 : 0
> vnode_map_update_time_100 : 0
> node_gets : 0
> node_gets_total : 290832
> node_get_fsm_siblings_mean : 0
> node_get_fsm_siblings_median : 0
> node_get_fsm_siblings_95 : 0
> node_get_fsm_siblings_99 : 0
> node_get_fsm_siblings_100 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_time_mean : 0
> node_get_fsm_time_median : 0
> node_get_fsm_time_95 : 0
> node_get_fsm_time_99 : 0
> node_get_fsm_time_100 : 0
> node_gets_counter : 0
> node_gets_counter_total : 0
> node_get_fsm_counter_siblings_mean : 0
> node_get_fsm_counter_siblings_median : 0
> node_get_fsm_counter_siblings_95 : 0
> node_get_fsm_counter_siblings_99 : 0
> node_get_fsm_counter_siblings_100 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_time_mean : 0
> node_get_fsm_counter_time_median : 0
> node_get_fsm_counter_time_95 : 0
> node_get_fsm_counter_time_99 : 0
> node_get_fsm_counter_time_100 : 0
> node_gets_set : 0
> node_gets_set_total : 0
> node_get_fsm_set_siblings_mean : 0
> node_get_fsm_set_siblings_median : 0
> node_get_fsm_set_siblings_95 : 0
> node_get_fsm_set_siblings_99 : 0
> node_get_fsm_set_siblings_100 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_time_mean : 0
> node_get_fsm_set_time_median : 0
> node_get_fsm_set_time_95 : 0
> node_get_fsm_set_time_99 : 0
> node_get_fsm_set_time_100 : 0
> node_gets_map : 0
> node_gets_map_total : 0
> node_get_fsm_map_siblings_mean : 0
> node_get_fsm_map_siblings_median : 0
> node_get_fsm_map_siblings_95 : 0
> node_get_fsm_map_siblings_99 : 0
> node_get_fsm_map_siblings_100 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_time_mean : 0
> node_get_fsm_map_time_median : 0
> node_get_fsm_map_time_95 : 0
> node_get_fsm_map_time_99 : 0
> node_get_fsm_map_time_100 : 0
> node_puts : 0
> node_puts_total : 210705
> node_put_fsm_time_mean : 0
> node_put_fsm_time_median : 0
> node_put_fsm_time_95 : 0
> node_put_fsm_time_99 : 0
> node_put_fsm_time_100 : 0
> node_puts_counter : 0
> node_puts_counter_total : 0
> node_put_fsm_counter_time_mean : 0
> node_put_fsm_counter_time_median : 0
> node_put_fsm_counter_time_95 : 0
> node_put_fsm_counter_time_99 : 0
> node_put_fsm_counter_time_100 : 0
> node_puts_set : 0
> node_puts_set_total : 0
> node_put_fsm_set_time_mean : 0
> node_put_fsm_set_time_median : 0
> node_put_fsm_set_time_95 : 0
> node_put_fsm_set_time_99 : 0
> node_put_fsm_set_time_100 : 0
> node_puts_map : 0
> node_puts_map_total : 0
> node_put_fsm_map_time_mean : 0
> node_put_fsm_map_time_median : 0
> node_put_fsm_map_time_95 : 0
> node_put_fsm_map_time_99 : 0
> node_put_fsm_map_time_100 : 0
> read_repairs : 0
> read_repairs_total : 103498
> read_repairs_counter : 0
> read_repairs_counter_total : 0
> read_repairs_set : 0
> read_repairs_set_total : 0
> read_repairs_map : 0
> read_repairs_map_total : 0
> coord_redirs_total : 89689
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 0
> pbc_connects : 0
> pbc_connects_total : 13
> counter_actor_counts_mean : 0
> counter_actor_counts_median : 0
> counter_actor_counts_95 : 0
> counter_actor_counts_99 : 0
> counter_actor_counts_100 : 0
> set_actor_counts_mean : 0
> set_actor_counts_median : 0
> set_actor_counts_95 : 0
> set_actor_counts_99 : 0
> set_actor_counts_100 : 0
> map_actor_counts_mean : 0
> map_actor_counts_median : 0
> map_actor_counts_95 : 0
> map_actor_counts_99 : 0
> map_actor_counts_100 : 0
> late_put_fsm_coordinator_ack : 0
> consistent_gets : 0
> consistent_gets_total : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_100 : 0
> consistent_get_time_mean : 0
> consistent_get_time_median : 0
> consistent_get_time_95 : 0
> consistent_get_time_99 : 0
> consistent_get_time_100 : 0
> consistent_puts : 0
> consistent_puts_total : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_100 : 0
> consistent_put_time_mean : 0
> consistent_put_time_median : 0
> consistent_put_time_95 : 0
> consistent_put_time_99 : 0
> consistent_put_time_100 : 0
> node_get_fsm_active : 0
> node_get_fsm_active_60s : 0
> node_get_fsm_in_rate : 0
> node_get_fsm_out_rate : 0
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 0
> node_put_fsm_active_60s : 0
> node_put_fsm_in_rate : 0
> node_put_fsm_out_rate : 0
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 0
> read_repairs_primary_outofdate_count : 1762
> read_repairs_primary_notfound_one : 0
> read_repairs_primary_notfound_count : 99695
> read_repairs_fallback_notfound_one : 0
> read_repairs_fallback_notfound_count : 2511
> leveldb_read_block_error : 0
> riak_pipe_stat_ts : 1406455621
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> pipeline_active : 0
> cpu_nprocs : 308
> cpu_avg1 : 292
> cpu_avg5 : 323
> cpu_avg15 : 333
> mem_total : 2099634176
> mem_allocated : 2029105152
> disk : [{"/",41153856,70},
>         {"/sys/fs/cgroup",4,0},
>         {"/dev",1015140,1},
>         {"/run",205044,1},
>         {"/run/lock",5120,0},
>         {"/run/shm",1025212,0},
>         {"/run/user",102400,0}]
> nodename : 'riak at 10.128.138.25'
> connected_nodes : ['riak at 10.128.179.86','riak at 10.128.220.135',
>                    'riak at 10.128.224.15','riak at 10.128.133.59',
>                    'riak at 10.128.220.242','riak at 10.128.137.185']
> sys_driver_version : <<"2.2">>
> sys_global_heaps_size : deprecated
> sys_heap_type : private
> sys_logical_processors : 2
> sys_monitor_count : 128
> sys_otp_release : <<"R16B02-basho5">>
> sys_port_count : 52
> sys_process_count : 937
> sys_smp_support : true
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 64
> sys_wordsize : 8
> ring_members : ['riak at 10.128.133.59','riak at 10.128.137.185',
>                 'riak at 10.128.138.25','riak at 10.128.179.86',
>                 'riak at 10.128.220.135','riak at 10.128.220.242',
>                 'riak at 10.128.224.15']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak at 10.128.220.242',6},\n {'riak at 10.128.224.15',9},\n {'riak at 10.128.137.185',10},\n {'riak at 10.128.220.135',9},\n {'riak at 10.128.138.25',9},\n {'riak at 10.128.179.86',11},\n {'riak at 10.128.133.59',10}]">>
> ring_creation_size : 64
> storage_backend : riak_kv_eleveldb_backend
> riak_auth_mods_version : <<"2.0.0beta1-0-g6ae36b2">>
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"2.0.0beta1-6-g3fe302e">>
> cluster_info_version : <<"2.0.0beta1-0-g1fae829">>
> yokozuna_version : <<"2.0.0beta1-81-g47214a6">>
> ibrowse_version : <<"4.0.1">>
> riak_search_version : <<"2.0.0beta1-14-g535a88f">>
> merge_index_version : <<"2.0.0beta1-0-gb701dde">>
> riak_kv_version : <<"2.0.0beta1-175-gcdde007">>
> riak_api_version : <<"2.0.0beta1-9-g8aad9cd">>
> riak_pb_version : <<"2.0.0.16-0-gfc18a9b">>
> protobuffs_version : <<"0.8.1p4-0-g5257dfe">>
> riak_dt_version : <<"2.0.0beta2-4-gc06820d">>
> sidejob_version : <<"2.0.0beta1-0-gc5aabba">>
> riak_pipe_version : <<"2.0.0beta1-7-gbae5fc6">>
> riak_core_version : <<"2.0.0beta1-66-gddab30a">>
> poolboy_version : <<"0.8.1p2-0-g84d836a">>
> pbkdf2_version : <<"2.0.0-0-g7076584">>
> eleveldb_version : <<"2.0.0beta1-31-g8f1d1f6">>
> bitcask_version : <<"1.7.0">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.5-0-ge5f8233">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9.6">>
> erlang_js_version : <<"1.3.0-0-g07467d8">>
> runtime_tools_version : <<"1.8.12">>
> os_mon_version : <<"2.2.13">>
> riak_sysmon_version : <<"2.0.0">>
> ssl_version : <<"5.3.1">>
> public_key_version : <<"0.20">>
> crypto_version : <<"3.1">>
> asn1_version : <<"2.0.3">>
> sasl_version : <<"2.3.3">>
> lager_version : <<"2.0.3">>
> goldrush_version : <<"0.1.6">>
> compiler_version : <<"4.9.3">>
> syntax_tools_version : <<"1.6.11">>
> stdlib_version : <<"1.19.3">>
> kernel_version : <<"2.16.3">>
> memory_total : 70247552
> memory_processes : 24818872
> memory_processes_used : 24804224
> memory_system : 45428680
> memory_atom : 654217
> memory_atom_used : 637895
> memory_binary : 11874352
> memory_code : 15099631
> memory_ets : 8502744
> search_query_throughput_count : 0
> search_query_throughput_one : 0
> search_query_fail_count : 0
> search_query_fail_one : 0
> search_query_latency_min : 0
> search_query_latency_max : 0
> search_query_latency_median : 0
> search_query_latency_95 : 0
> search_query_latency_99 : 0
> search_query_latency_999 : 0
> search_index_throughput_count : 363586
> search_index_throughtput_one : 0
> search_index_fail_count : 0
> search_index_fail_one : 0
> search_index_latency_min : 0
> search_index_latency_max : 0
> search_index_latency_median : 0
> search_index_latency_95 : 0
> search_index_latency_99 : 0
> search_index_latency_999 : 0
> riak_core_stat_ts : 1406455621
> ignored_gossip_total : 0
> rings_reconciled_total : 765
> rings_reconciled : 0
> gossip_received : 5
> rejected_handoffs : 0
> handoff_timeouts : 0
> dropped_vnode_requests_total : 0
> converge_delay_min : 0
> converge_delay_max : 0
> converge_delay_mean : 0
> converge_delay_last : 0
> rebalance_delay_min : 0
> rebalance_delay_max : 0
> rebalance_delay_mean : 0
> rebalance_delay_last : 0
> riak_kv_vnodes_running : 13
> riak_kv_vnodeq_min : 0
> riak_kv_vnodeq_median : 0
> riak_kv_vnodeq_mean : 0
> riak_kv_vnodeq_max : 0
> riak_kv_vnodeq_total : 0
> riak_pipe_vnodes_running : 9
> riak_pipe_vnodeq_min : 0
> riak_pipe_vnodeq_median : 0
> riak_pipe_vnodeq_mean : 0
> riak_pipe_vnodeq_max : 0
> riak_pipe_vnodeq_total : 0
> 
> 1-minute stats for 'riak at 10.128.133.59'
> -------------------------------------------
> riak_kv_stat_ts : 1406455621
> vnode_gets : 12
> vnode_gets_total : 292487
> vnode_puts : 11
> vnode_puts_total : 815717
> vnode_index_refreshes : 0
> vnode_index_refreshes_total : 0
> vnode_index_reads : 0
> vnode_index_reads_total : 0
> vnode_index_writes : 11
> vnode_index_writes_total : 550450
> vnode_index_writes_postings : 0
> vnode_index_writes_postings_total : 339363
> vnode_index_deletes : 0
> vnode_index_deletes_total : 0
> vnode_index_deletes_postings : 0
> vnode_index_deletes_postings_total : 0
> object_counter_merge : 0
> object_counter_merge_total : 0
> object_counter_merge_time_mean : 0
> object_counter_merge_time_median : 0
> object_counter_merge_time_95 : 0
> object_counter_merge_time_99 : 0
> object_counter_merge_time_100 : 0
> vnode_counter_update : 0
> vnode_counter_update_total : 0
> vnode_counter_update_time_mean : 0
> vnode_counter_update_time_median : 0
> vnode_counter_update_time_95 : 0
> vnode_counter_update_time_99 : 0
> vnode_counter_update_time_100 : 0
> object_set_merge : 0
> object_set_merge_total : 0
> object_set_merge_time_mean : 0
> object_set_merge_time_median : 0
> object_set_merge_time_95 : 0
> object_set_merge_time_99 : 0
> object_set_merge_time_100 : 0
> vnode_set_update : 0
> vnode_set_update_total : 0
> vnode_set_update_time_mean : 0
> vnode_set_update_time_median : 0
> vnode_set_update_time_95 : 0
> vnode_set_update_time_99 : 0
> vnode_set_update_time_100 : 0
> object_map_merge : 0
> object_map_merge_total : 0
> object_map_merge_time_mean : 0
> object_map_merge_time_median : 0
> object_map_merge_time_95 : 0
> object_map_merge_time_99 : 0
> object_map_merge_time_100 : 0
> object_merge : 0
> object_merge_total : 112749
> object_merge_time_mean : 0
> object_merge_time_median : 0
> object_merge_time_95 : 0
> object_merge_time_99 : 0
> object_merge_time_100 : 0
> vnode_map_update : 0
> vnode_map_update_total : 0
> vnode_map_update_time_mean : 0
> vnode_map_update_time_median : 0
> vnode_map_update_time_95 : 0
> vnode_map_update_time_99 : 0
> vnode_map_update_time_100 : 0
> node_gets : 24
> node_gets_total : 101312
> node_get_fsm_siblings_mean : 1
> node_get_fsm_siblings_median : 1
> node_get_fsm_siblings_95 : 1
> node_get_fsm_siblings_99 : 1
> node_get_fsm_siblings_100 : 1
> node_get_fsm_objsize_mean : 945
> node_get_fsm_objsize_median : 917
> node_get_fsm_objsize_95 : 1066
> node_get_fsm_objsize_99 : 1072
> node_get_fsm_objsize_100 : 1072
> node_get_fsm_time_mean : 5223
> node_get_fsm_time_median : 1616
> node_get_fsm_time_95 : 35883
> node_get_fsm_time_99 : 41992
> node_get_fsm_time_100 : 41992
> node_gets_counter : 0
> node_gets_counter_total : 0
> node_get_fsm_counter_siblings_mean : 0
> node_get_fsm_counter_siblings_median : 0
> node_get_fsm_counter_siblings_95 : 0
> node_get_fsm_counter_siblings_99 : 0
> node_get_fsm_counter_siblings_100 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_time_mean : 0
> node_get_fsm_counter_time_median : 0
> node_get_fsm_counter_time_95 : 0
> node_get_fsm_counter_time_99 : 0
> node_get_fsm_counter_time_100 : 0
> node_gets_set : 0
> node_gets_set_total : 0
> node_get_fsm_set_siblings_mean : 0
> node_get_fsm_set_siblings_median : 0
> node_get_fsm_set_siblings_95 : 0
> node_get_fsm_set_siblings_99 : 0
> node_get_fsm_set_siblings_100 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_time_mean : 0
> node_get_fsm_set_time_median : 0
> node_get_fsm_set_time_95 : 0
> node_get_fsm_set_time_99 : 0
> node_get_fsm_set_time_100 : 0
> node_gets_map : 0
> node_gets_map_total : 0
> node_get_fsm_map_siblings_mean : 0
> node_get_fsm_map_siblings_median : 0
> node_get_fsm_map_siblings_95 : 0
> node_get_fsm_map_siblings_99 : 0
> node_get_fsm_map_siblings_100 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_time_mean : 0
> node_get_fsm_map_time_median : 0
> node_get_fsm_map_time_95 : 0
> node_get_fsm_map_time_99 : 0
> node_get_fsm_map_time_100 : 0
> node_puts : 0
> node_puts_total : 39507
> node_put_fsm_time_mean : 0
> node_put_fsm_time_median : 0
> node_put_fsm_time_95 : 0
> node_put_fsm_time_99 : 0
> node_put_fsm_time_100 : 0
> node_puts_counter : 0
> node_puts_counter_total : 0
> node_put_fsm_counter_time_mean : 0
> node_put_fsm_counter_time_median : 0
> node_put_fsm_counter_time_95 : 0
> node_put_fsm_counter_time_99 : 0
> node_put_fsm_counter_time_100 : 0
> node_puts_set : 0
> node_puts_set_total : 0
> node_put_fsm_set_time_mean : 0
> node_put_fsm_set_time_median : 0
> node_put_fsm_set_time_95 : 0
> node_put_fsm_set_time_99 : 0
> node_put_fsm_set_time_100 : 0
> node_puts_map : 0
> node_puts_map_total : 0
> node_put_fsm_map_time_mean : 0
> node_put_fsm_map_time_median : 0
> node_put_fsm_map_time_95 : 0
> node_put_fsm_map_time_99 : 0
> node_put_fsm_map_time_100 : 0
> read_repairs : 23
> read_repairs_total : 57542
> read_repairs_counter : 0
> read_repairs_counter_total : 0
> read_repairs_set : 0
> read_repairs_set_total : 0
> read_repairs_map : 0
> read_repairs_map_total : 0
> coord_redirs_total : 0
> executing_mappers : 0
> precommit_fail : 0
> postcommit_fail : 0
> index_fsm_create : 0
> index_fsm_create_error : 0
> index_fsm_active : 0
> list_fsm_create : 0
> list_fsm_create_error : 0
> list_fsm_active : 0
> pbc_active : 0
> pbc_connects : 0
> pbc_connects_total : 0
> counter_actor_counts_mean : 0
> counter_actor_counts_median : 0
> counter_actor_counts_95 : 0
> counter_actor_counts_99 : 0
> counter_actor_counts_100 : 0
> set_actor_counts_mean : 0
> set_actor_counts_median : 0
> set_actor_counts_95 : 0
> set_actor_counts_99 : 0
> set_actor_counts_100 : 0
> map_actor_counts_mean : 0
> map_actor_counts_median : 0
> map_actor_counts_95 : 0
> map_actor_counts_99 : 0
> map_actor_counts_100 : 0
> late_put_fsm_coordinator_ack : 0
> consistent_gets : 0
> consistent_gets_total : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_100 : 0
> consistent_get_time_mean : 0
> consistent_get_time_median : 0
> consistent_get_time_95 : 0
> consistent_get_time_99 : 0
> consistent_get_time_100 : 0
> consistent_puts : 0
> consistent_puts_total : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_100 : 0
> consistent_put_time_mean : 0
> consistent_put_time_median : 0
> consistent_put_time_95 : 0
> consistent_put_time_99 : 0
> consistent_put_time_100 : 0
> node_get_fsm_active : 0
> node_get_fsm_active_60s : 24
> node_get_fsm_in_rate : 0
> node_get_fsm_out_rate : 0
> node_get_fsm_rejected : 0
> node_get_fsm_rejected_60s : 0
> node_get_fsm_rejected_total : 0
> node_put_fsm_active : 0
> node_put_fsm_active_60s : 0
> node_put_fsm_in_rate : 0
> node_put_fsm_out_rate : 0
> node_put_fsm_rejected : 0
> node_put_fsm_rejected_60s : 0
> node_put_fsm_rejected_total : 0
> read_repairs_primary_outofdate_one : 2
> read_repairs_primary_outofdate_count : 27
> read_repairs_primary_notfound_one : 22
> read_repairs_primary_notfound_count : 57526
> leveldb_read_block_error : 0
> riak_pipe_stat_ts : 1406455620
> pipeline_create_count : 0
> pipeline_create_one : 0
> pipeline_create_error_count : 0
> pipeline_create_error_one : 0
> pipeline_active : 0
> cpu_nprocs : 292
> cpu_avg1 : 69
> cpu_avg5 : 41
> cpu_avg15 : 38
> mem_total : 2099634176
> mem_allocated : 2008449024
> disk : [{"/",41153856,37},
>         {"/sys/fs/cgroup",4,0},
>         {"/dev",1015140,1},
>         {"/run",205044,1},
>         {"/run/lock",5120,0},
>         {"/run/shm",1025212,0},
>         {"/run/user",102400,0}]
> nodename : 'riak at 10.128.133.59'
> connected_nodes : ['riak at 10.128.220.135','riak at 10.128.138.25',
>                    'riak at 10.128.224.15','riak at 10.128.179.86',
>                    'riak at 10.128.220.242','riak at 10.128.137.185']
> sys_driver_version : <<"2.2">>
> sys_global_heaps_size : deprecated
> sys_heap_type : private
> sys_logical_processors : 2
> sys_monitor_count : 158
> sys_otp_release : <<"R16B02-basho5">>
> sys_port_count : 54
> sys_process_count : 1057
> sys_smp_support : true
> sys_system_version : <<"Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]">>
> sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
> sys_threads_enabled : true
> sys_thread_pool_size : 64
> sys_wordsize : 8
> ring_members : ['riak at 10.128.133.59','riak at 10.128.137.185',
>                 'riak at 10.128.138.25','riak at 10.128.179.86',
>                 'riak at 10.128.220.135','riak at 10.128.220.242',
>                 'riak at 10.128.224.15']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak at 10.128.220.242',6},\n {'riak at 10.128.224.15',9},\n {'riak at 10.128.137.185',10},\n {'riak at 10.128.220.135',9},\n {'riak at 10.128.138.25',9},\n {'riak at 10.128.179.86',11},\n {'riak at 10.128.133.59',10}]">>
> ring_creation_size : 64
> storage_backend : riak_kv_eleveldb_backend
> riak_auth_mods_version : <<"2.0.0beta1-0-g6ae36b2">>
> erlydtl_version : <<"0.7.0">>
> riak_control_version : <<"2.0.0beta1-6-g3fe302e">>
> cluster_info_version : <<"2.0.0beta1-0-g1fae829">>
> yokozuna_version : <<"2.0.0beta1-81-g47214a6">>
> ibrowse_version : <<"4.0.1">>
> riak_search_version : <<"2.0.0beta1-14-g535a88f">>
> merge_index_version : <<"2.0.0beta1-0-gb701dde">>
> riak_kv_version : <<"2.0.0beta1-175-gcdde007">>
> riak_api_version : <<"2.0.0beta1-9-g8aad9cd">>
> riak_pb_version : <<"2.0.0.16-0-gfc18a9b">>
> protobuffs_version : <<"0.8.1p4-0-g5257dfe">>
> riak_dt_version : <<"2.0.0beta2-4-gc06820d">>
> sidejob_version : <<"2.0.0beta1-0-gc5aabba">>
> riak_pipe_version : <<"2.0.0beta1-7-gbae5fc6">>
> riak_core_version : <<"2.0.0beta1-66-gddab30a">>
> poolboy_version : <<"0.8.1p2-0-g84d836a">>
> pbkdf2_version : <<"2.0.0-0-g7076584">>
> eleveldb_version : <<"2.0.0beta1-31-g8f1d1f6">>
> bitcask_version : <<"1.7.0">>
> basho_stats_version : <<"1.0.3">>
> webmachine_version : <<"1.10.5-0-ge5f8233">>
> mochiweb_version : <<"1.5.1p6">>
> inets_version : <<"5.9.6">>
> erlang_js_version : <<"1.3.0-0-g07467d8">>
> runtime_tools_version : <<"1.8.12">>
> os_mon_version : <<"2.2.13">>
> riak_sysmon_version : <<"2.0.0">>
> ssl_version : <<"5.3.1">>
> public_key_version : <<"0.20">>
> crypto_version : <<"3.1">>
> asn1_version : <<"2.0.3">>
> sasl_version : <<"2.3.3">>
> lager_version : <<"2.0.3">>
> goldrush_version : <<"0.1.6">>
> compiler_version : <<"4.9.3">>
> syntax_tools_version : <<"1.6.11">>
> stdlib_version : <<"1.19.3">>
> kernel_version : <<"2.16.3">>
> memory_total : 119434496
> memory_processes : 73769160
> memory_processes_used : 73764864
> memory_system : 45665336
> memory_atom : 602729
> memory_atom_used : 575574
> memory_binary : 13429560
> memory_code : 14651483
> memory_ets : 7791568
> search_query_throughput_count : 0
> search_query_throughput_one : 0
> search_query_fail_count : 0
> search_query_fail_one : 0
> search_query_latency_min : 0
> search_query_latency_max : 0
> search_query_latency_median : 0
> search_query_latency_95 : 0
> search_query_latency_99 : 0
> search_query_latency_999 : 0
> search_index_throughput_count : 550269
> search_index_throughtput_one : 11
> search_index_fail_count : 7
> search_index_fail_one : 0
> search_index_latency_min : 2342
> search_index_latency_max : 4575
> search_index_latency_median : 3264
> search_index_latency_95 : 4460
> search_index_latency_99 : 4575
> search_index_latency_999 : 4575
> riak_core_stat_ts : 1406455622
> ignored_gossip_total : 0
> rings_reconciled_total : 160
> rings_reconciled : 0
> gossip_received : 6
> rejected_handoffs : 0
> handoff_timeouts : 2
> dropped_vnode_requests_total : 0
> converge_delay_min : 0
> converge_delay_max : 0
> converge_delay_mean : 0
> converge_delay_last : 0
> rebalance_delay_min : 0
> rebalance_delay_max : 0
> rebalance_delay_mean : 0
> rebalance_delay_last : 0
> riak_kv_vnodes_running : 29
> riak_kv_vnodeq_min : 0
> riak_kv_vnodeq_median : 0
> riak_kv_vnodeq_mean : 0
> riak_kv_vnodeq_max : 0
> riak_kv_vnodeq_total : 0
> riak_pipe_vnodes_running : 10
> riak_pipe_vnodeq_min : 0
> riak_pipe_vnodeq_median : 0
> riak_pipe_vnodeq_mean : 0
> riak_pipe_vnodeq_max : 0
> riak_pipe_vnodeq_total : 0
> 
> 
> On Sun, Jul 27, 2014 at 12:12 PM, Bryan Hunt <bhunt at basho.com> wrote:
> Hello Chaim,
> 
> How big is the object you are trying to index? Are siblings (allow_mult) enabled? What type of object is it? CRDT, binary blob, text? Can you paste the output from riak-admin member-status and ring-status. Also riak status. 
> Thanks,
> 
> Bryan
> 
> On 27 Jul 2014 04:55, "Chaim Solomon" <chaim at itcentralstation.com> wrote:
> Hi,
> 
> I've been having issues for some days with the cluster I am running (or now - trying to run). The last major operation was going through all items of a bucket and doing a read/update operation on them from Python code - nothing fancy, just modifying some data.
> 
> I got issues with one node being VERY busy - 2 CPUs fully loaded - and the requests were timing out or at least VERY slow.
> I took the node out of the cluster and added another node - then after clearing the node brought it back. After some days of waiting for AAE and the migration of data to finish, I am still having one node at 2 CPUs at 100% and one node with one CPU at 100% and the requests are still VERY slow. I am getting a lot of these in the console.log:
> 
> 2014-07-26 23:37:33.459 [error] <0.7427.266>@yz_kv:index:206 failed to index object {<<"linkmeta">>,<<"...">>} with error {"Failed to index docs",{error,req_timedout}} because [{yz_solr,index,3,[{file,"src/yz_solr.erl"},{line,192}]},{yz_kv,index,7,[{file,"src/yz_kv.erl"},{line,258}]},{yz_kv,index,3,[{file,"src/yz_kv.erl"},{line,193}]},{riak_kv_vnode,actual_put,6,[{file,"src/riak_kv_vnode.erl"},{line,1416}]},{riak_kv_vnode,perform_put,3,[{file,"src/riak_kv_vnode.erl"},{line,1404}]},{riak_kv_vnode,do_put,7,[{file,"src/riak_kv_vnode.erl"},{line,1199}]},{riak_kv_vnode,handle_command,3,[{file,"src/riak_kv_vnode.erl"},{line,485}]},{riak_core_vnode,vnode_command,3,[{file,"src/riak_core_vnode.erl"},{line,345}]}]
> 
> And on almost all other nodex I am getting a lot of these:
> 2014-07-26 23:50:58.796 [info] <0.12268.221>@yz_kv:should_handoff:157 waiting for bucket types prefix and indexes to agree between 'riak at 10.128.138.25' and 'riak at 10.128.137.185'
> 
> How can I get the cluster to work again?
> 
> It is a 2.0.0rc1 cluster.
> 
> Chaim Solomon
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140727/fc41e943/attachment.html>


More information about the riak-users mailing list