Waiting for riak to handoff ?

Marcel Koopman marcel.koopman at gmail.com
Fri Aug 22 05:28:41 EDT 2014


Hello, we upgraded to 1.4.10 on all nodes. Please see below:

 riak-admin transfers

'riak at newnode5 waiting to handoff 11 partitions

'riak at newnode4' waiting to handoff 11 partitions

'riak at newnode3' waiting to handoff 11 partitions

'riak at newnode2' waiting to handoff 11 partitions

'riak at newnode1' waiting to handoff 11 partitions

'riak at oldnode2' waiting to handoff 11 partitions

'riak at oldnode3' waiting to handoff 11 partitions


Active Transfers:



$ riak-admin transfer-limit

=============================== Transfer Limit
================================

Limit        Node

-------------------------------------------------------------------------------

    2        'riak at oldnode3'

    2        'riak at oldnode2'

    2        'riak at oldnode1'

    2        'riak at newnode1'

    2        'riak at newnode2'

    2        'riak at newnode3'

    2        'riak at newnode4'

    2        'riak at newnode5'

-------------------------------------------------------------------------------

Note: You can change transfer limits with 'riak-admin transfer_limit
<limit>'

      and 'riak-admin transfer_limit <node> <limit>'

$ riak-admin status

1-minute stats for 'riak at newnode1'

-------------------------------------------

riak_kv_stat_ts : 1408692262

vnode_gets : 0

vnode_gets_total : 374554

vnode_puts : 0

vnode_puts_total : 11795

vnode_index_refreshes : 0

vnode_index_refreshes_total : 0

vnode_index_reads : 0

vnode_index_reads_total : 683394

vnode_index_writes : 0

vnode_index_writes_total : 11715

vnode_index_writes_postings : 0

vnode_index_writes_postings_total : 10449

vnode_index_deletes : 0

vnode_index_deletes_total : 0

vnode_index_deletes_postings : 0

vnode_index_deletes_postings_total : 675

node_gets : 0

node_gets_total : 117806

node_get_fsm_siblings_mean : 0

node_get_fsm_siblings_median : 0

node_get_fsm_siblings_95 : 0

node_get_fsm_siblings_99 : 0

node_get_fsm_siblings_100 : 0

node_get_fsm_objsize_mean : 0

node_get_fsm_objsize_median : 0

node_get_fsm_objsize_95 : 0

node_get_fsm_objsize_99 : 0

node_get_fsm_objsize_100 : 0

node_get_fsm_time_mean : 0

node_get_fsm_time_median : 0

node_get_fsm_time_95 : 0

node_get_fsm_time_99 : 0

node_get_fsm_time_100 : 0

node_puts : 0

node_puts_total : 4575

node_put_fsm_time_mean : 0

node_put_fsm_time_median : 0

node_put_fsm_time_95 : 0

node_put_fsm_time_99 : 0

node_put_fsm_time_100 : 0

read_repairs : 0

read_repairs_total : 129

coord_redirs_total : 1427

executing_mappers : 0

precommit_fail : 0

postcommit_fail : 0

index_fsm_create : 0

index_fsm_create_error : 0

index_fsm_active : 0

list_fsm_create : 0

list_fsm_create_error : 0

list_fsm_active : 0

pbc_active : 56

pbc_connects : 60

pbc_connects_total : 163594

node_get_fsm_active : 0

node_get_fsm_active_60s : 10

node_get_fsm_in_rate : 0

node_get_fsm_out_rate : 0

node_get_fsm_rejected : 0

node_get_fsm_rejected_60s : 0

node_get_fsm_rejected_total : 0

node_put_fsm_active : 0

node_put_fsm_active_60s : 0

node_put_fsm_in_rate : 0

node_put_fsm_out_rate : 0

node_put_fsm_rejected : 0

node_put_fsm_rejected_60s : 0

node_put_fsm_rejected_total : 0

read_repairs_primary_outofdate_one : 0

read_repairs_primary_outofdate_count : 104

read_repairs_primary_notfound_one : 0

read_repairs_primary_notfound_count : 61

read_repairs_fallback_notfound_one : 0

read_repairs_fallback_notfound_count : 40

leveldb_read_block_error : 0

riak_pipe_stat_ts : 1408692262

pipeline_active : 0

pipeline_create_count : 0

pipeline_create_one : 0

pipeline_create_error_count : 0

pipeline_create_error_one : 0

cpu_nprocs : 263

cpu_avg1 : 20

cpu_avg5 : 23

cpu_avg15 : 33

mem_total : 8374890496

mem_allocated : 7878340608

disk : [{"/",5766260,42},

        {"/dev",4080488,1},

        {"/run",1635724,1},

        {"/run/lock",5120,0},

        {"/run/shm",4089300,0},

        {"/var/log",1922060,4},

        {"/data",258030980,69}]

nodename : 'riak@ <riak at 91.238.155.121>*oldnode2*'

connected_nodes :
['newnode1','newnode2','newnode3','newnode4',newnode5','oldnode1','oldnode3',

                   'riak at oldnode2']

sys_driver_version : <<"2.0">>

sys_global_heaps_size : 0

sys_heap_type : private

sys_logical_processors : 2

sys_otp_release : <<"R15B01">>

sys_process_count : 1175

sys_smp_support : true

sys_system_version : <<"Erlang R15B01 (erts-5.9.1) [source] [64-bit]
[smp:2:2] [async-threads:64] [kernel-poll:true]">>

sys_system_architecture : <<"x86_64-unknown-linux-gnu">>

sys_threads_enabled : true

sys_thread_pool_size : 64

sys_wordsize : 8

ring_members :
 ['newnode1','newnode2','newnode3','newnode4',newnode5','oldnode1','oldnode3',

                   'riak at oldnode2']

ring_num_partitions : 64

ring_ownership : <<"[{'riak at oldnode1',16},\n {'riak at oldnode2',32},\n
{'riak at oldnode3',16}]">>

ring_creation_size : 64

storage_backend : riak_kv_eleveldb_backend

erlydtl_version : <<"0.7.0">>

riak_control_version : <<"1.4.10-0-g73c43c3">>

cluster_info_version : <<"1.2.4">>

riak_search_version : <<"1.4.10-0-g6e548e7">>

merge_index_version : <<"1.3.2-0-gcb38ee7">>

riak_kv_version : <<"1.4.10-0-g64b6ad8">>

sidejob_version : <<"0.2.0">>

riak_api_version : <<"1.4.10-0-gc407ac0">>

riak_pipe_version : <<"1.4.10-0-g9353526">>

riak_core_version : <<"1.4.10">>

bitcask_version : <<"1.6.6-0-g230b6d6">>

basho_stats_version : <<"1.0.3">>

webmachine_version : <<"1.10.4-0-gfcff795">>

mochiweb_version : <<"1.5.1p6">>

inets_version : <<"5.9">>

erlang_js_version : <<"1.2.2">>

runtime_tools_version : <<"1.8.8">>

os_mon_version : <<"2.2.9">>

riak_sysmon_version : <<"1.1.3">>

ssl_version : <<"5.0.1">>

public_key_version : <<"0.15">>

crypto_version : <<"2.1">>

sasl_version : <<"2.2.1">>

lager_version : <<"2.0.1">>

goldrush_version : <<"0.1.5">>

compiler_version : <<"4.8.1">>

syntax_tools_version : <<"1.6.8">>

stdlib_version : <<"1.18.1">>

kernel_version : <<"2.15.1">>

memory_total : 222011368

memory_processes : 29166010

memory_processes_used : 29165996

memory_system : 192845358

memory_atom : 545377

memory_atom_used : 519982

memory_binary : 133403072

memory_code : 11477147

memory_ets : 7076680


2014-08-21 17:53 GMT+02:00 Luke Bakken <lbakken at basho.com>:

> Hi Marcel -
>
> What version of Riak are you using?
>
> What is the output of "riak-admin transfer-limit" ?
> --
> Luke Bakken
> CSE
> lbakken at basho.com
>
>
> On Wed, Aug 20, 2014 at 8:02 AM, Marcel Koopman
> <marcel.koopman at gmail.com> wrote:
> > Hello,
> >
> > It seems that adding new nodes does not have any effect.
> >
> > Why it Riak waiting for transfers? When does it start handoff the new
> nodes?
> >
> >
> > riak-admin member-status
> >
> > ================================= Membership
> > ==================================
> >
> > Status     Ring    Pending    Node
> >
> >
> -------------------------------------------------------------------------------
> >
> > valid      25.0%     34.4%    'riak at oldnode1'
> >
> > valid      25.0%     32.8%    'riak at oldnode2'
> >
> > valid      50.0%     32.8%    'riak at oldnode3
> >
> > valid       0.0%      0.0%    'riak at newnode1'
> >
> > valid       0.0%      0.0%    'riak at newnode2'
> >
> > valid       0.0%      0.0%    'riak at newnode3'
> >
> > valid       0.0%      0.0%    'riak at newnode4'
> >
> > valid       0.0%      0.0%    'riak at newnode5'
> >
> >
> -------------------------------------------------------------------------------
> >
> > Valid:8 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
> >
> >
> > riak-admin transfers
> >
> > 'riak at newnode5' waiting to handoff 11 partitions
> >
> > 'riak at newnode4' waiting to handoff 11 partitions
> >
> > 'riak at newnode3' waiting to handoff 11 partitions
> >
> > 'riak at newnode2' waiting to handoff 11 partitions
> >
> > 'riak at newnode1' waiting to handoff 11 partitions
> >
> > 'riak at oldnode1' waiting to handoff 11 partitions
> >
> > 'riak at oldnode2' waiting to handoff 11 partitions
> >
> > Active Transfers:
> >
> > (nothing here)
> >
> >
> >
> >
> >
> > --
> > Met vriendelijke groet,
> >
> > Marcel Koopman
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>



-- 
Met vriendelijke groet,

Marcel Koopman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140822/5e585745/attachment.html>


More information about the riak-users mailing list