Fetch method returns null point exception error

성동찬_Chan chan at kakao.com
Tue Oct 15 23:31:50 EDT 2013


Hi~!

I'm testing riak as a cache server with 6 server cluster, 2 hours expiration time.
I gave traffic to cluster about 40,000 quries per second from 12hours ago.
From start point, there was no problem, but after 8hours fetch method returns fail with "null point excpetion".
I change nothing while test operation.

Below, you can see "riak-admin status" and riak-admin top" result.
Could show me what cause this and how to solve this problem?

Thank you.
Chan.

riak-admin status
-------------------------------------------------------------------------------------------------------------------------------
riak_kv_stat_ts : 1381893196
vnode_gets : 1301374
vnode_gets_total : 1488219135
vnode_puts : 650670
vnode_puts_total : 273164708
vnode_index_reads : 0
vnode_index_reads_total : 0
vnode_index_writes : 650670
vnode_index_writes_total : 273164708
vnode_index_writes_postings : 0
vnode_index_writes_postings_total : 0
vnode_index_deletes : 0
vnode_index_deletes_total : 0
vnode_index_deletes_postings : 0
vnode_index_deletes_postings_total : 0
node_gets : 532128
node_gets_total : 605453222
node_get_fsm_siblings_mean : 0
node_get_fsm_siblings_median : 0
node_get_fsm_siblings_95 : 0
node_get_fsm_siblings_99 : 0
node_get_fsm_siblings_100 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_time_mean : 476
node_get_fsm_time_median : 467
node_get_fsm_time_95 : 865
node_get_fsm_time_99 : 1119
node_get_fsm_time_100 : 3213
node_puts : 226235
node_puts_total : 95080556
node_put_fsm_time_mean : 1179
node_put_fsm_time_median : 1116
node_put_fsm_time_95 : 1806
node_put_fsm_time_99 : 2337
node_put_fsm_time_100 : 10206
read_repairs : 0
read_repairs_total : 2697556
coord_redirs_total : 58178410
executing_mappers : 0
precommit_fail : 0
postcommit_fail : 0
index_fsm_create : 0
index_fsm_create_error : 0
index_fsm_active : 0
list_fsm_create : 0
list_fsm_create_error : 0
list_fsm_active : 0
pbc_active : 29
pbc_connects : 1
pbc_connects_total : 2284
node_get_fsm_active : 22
node_get_fsm_active_60s : 527810
node_get_fsm_in_rate : 8562
node_get_fsm_out_rate : 8560
node_get_fsm_rejected : 0
node_get_fsm_rejected_60s : 0
node_get_fsm_rejected_total : 0
node_put_fsm_active : 5
node_put_fsm_active_60s : 368997
node_put_fsm_in_rate : 6130
node_put_fsm_out_rate : 6128
node_put_fsm_rejected : 0
node_put_fsm_rejected_60s : 0
node_put_fsm_rejected_total : 0
read_repairs_primary_outofdate_one : 0
read_repairs_primary_outofdate_count : 126588
read_repairs_primary_notfound_one : 0
read_repairs_primary_notfound_count : 4169558
leveldb_read_block_error : undefined
riak_pipe_stat_ts : 1381893196
pipeline_active : 0
pipeline_create_count : 0
pipeline_create_one : 0
pipeline_create_error_count : 0
pipeline_create_error_one : 0
cpu_nprocs : 361
cpu_avg1 : 2068
cpu_avg5 : 2045
cpu_avg15 : 2028
mem_total : 16799096832
mem_allocated : 4083048448
disk : [{"/",279713876,2},{"/dev/shm",8202684,0}]
nodename : 'chan at fox656.kr2.iwilab.com<mailto:chan at fox656.kr2.iwilab.com>'
connected_nodes : ['chan at fox136.kr2.iwilab.com<mailto:chan at fox136.kr2.iwilab.com>','chan at fox167.kr2.iwilab.com<mailto:chan at fox167.kr2.iwilab.com>',
                   'chan at fox655.kr2.iwilab.com<mailto:chan at fox655.kr2.iwilab.com>','chan at fox654.kr2.iwilab.com<mailto:chan at fox654.kr2.iwilab.com>',
                   'chan at fox168.kr2.iwilab.com<mailto:chan at fox168.kr2.iwilab.com>']
sys_driver_version : <<"2.0">>
sys_global_heaps_size : 0
sys_heap_type : private
sys_logical_processors : 8
sys_otp_release : <<"R15B01">>
sys_process_count : 566
sys_smp_support : true
sys_system_version : <<"Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:8:8] [async-threads:128] [kernel-poll:true]">>
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_threads_enabled : true
sys_thread_pool_size : 128
sys_wordsize : 8
ring_members : ['chan at fox136.kr2.iwilab.com<mailto:chan at fox136.kr2.iwilab.com>','chan at fox167.kr2.iwilab.com<mailto:chan at fox167.kr2.iwilab.com>',
                'chan at fox168.kr2.iwilab.com<mailto:chan at fox168.kr2.iwilab.com>','chan at fox654.kr2.iwilab.com<mailto:chan at fox654.kr2.iwilab.com>',
                'chan at fox655.kr2.iwilab.com<mailto:chan at fox655.kr2.iwilab.com>','chan at fox656.kr2.iwilab.com<mailto:chan at fox656.kr2.iwilab.com>']
ring_num_partitions : 32
ring_ownership : <<"[{'chan at fox656.kr2.iwilab.com<mailto:chan at fox656.kr2.iwilab.com>',5},\n {'chan at fox655.kr2.iwilab.com<mailto:chan at fox655.kr2.iwilab.com>',5},\n {'chan at fox654.kr2.iwilab.com<mailto:chan at fox654.kr2.iwilab.com>',5},\n {'chan at fox168.kr2.iwilab.com<mailto:chan at fox168.kr2.iwilab.com>',5},\n {'chan at fox167.kr2.iwilab.com<mailto:chan at fox167.kr2.iwilab.com>',6},\n {'chan at fox136.kr2.iwilab.com<mailto:chan at fox136.kr2.iwilab.com>',6}]">>
ring_creation_size : 32
storage_backend : riak_kv_multi_backend
erlydtl_version : <<"0.7.0">>
riak_control_version : <<"1.4.2-0-g2f08324">>
cluster_info_version : <<"1.2.4">>
riak_search_version : <<"1.4.2-0-ge31368d">>
merge_index_version : <<"1.3.1">>
riak_kv_version : <<"1.4.2-0-g61ac9d8">>
sidejob_version : <<"0.2.0">>
riak_api_version : <<"1.4.2-0-ga7e00e2">>
riak_pipe_version : <<"1.4.2-0-g59d1bae">>
riak_core_version : <<"1.4.2">>
bitcask_version : <<"1.6.3">>
basho_stats_version : <<"1.0.3">>
webmachine_version : <<"1.10.4-0-gfcff795">>
mochiweb_version : <<"1.5.1p6">>
inets_version : <<"5.9">>
erlang_js_version : <<"1.2.2">>
runtime_tools_version : <<"1.8.8">>
os_mon_version : <<"2.2.9">>
riak_sysmon_version : <<"1.1.3">>
ssl_version : <<"5.0.1">>
public_key_version : <<"0.15">>
crypto_version : <<"2.1">>
sasl_version : <<"2.2.1">>
lager_version : <<"2.0.0">>
compiler_version : <<"4.8.1">>
syntax_tools_version : <<"1.6.8">>
stdlib_version : <<"1.18.1">>
kernel_version : <<"2.15.1">>
memory_total : 333983032
memory_processes : 10197917
memory_processes_used : 9925151
memory_system : 323785115
memory_atom : 504409
memory_atom_used : 485838
memory_binary : 479552
memory_code : 10516587
memory_ets : 227826744

riak-admin top
-------------------------------------------------------------------------------------------------------------------------------
Pid                 Name or Initial Func         Time       Reds     Memory       MsgQ Current Function
-------------------------------------------------------------------------------------------------------------------------------
<6210.632.0>        proc_lib:init_p/5             '-'    8916284      21744          1 riak_core_vnode:continue/2
<6210.631.0>        proc_lib:init_p/5             '-'    8890232      21696          0 gen_fsm:loop/7
<6210.565.0>        proc_lib:init_p/5             '-'    8881011       8920          0 gen_fsm:loop/7
<6210.511.0>        proc_lib:init_p/5             '-'    8795742      21728          1 gen_fsm:loop/7
<6210.633.0>        proc_lib:init_p/5             '-'    8791178      21696          0 gen_fsm:loop/7
<6210.337.0>        riak_kv_stat_sj_8             '-'    4292821       3856          0 gen_server:loop/6
<6210.331.0>        riak_kv_stat_sj_2             '-'    3583242       3856          0 gen_server:loop/6
<6210.332.0>        riak_kv_stat_sj_3             '-'    3374842       5720          0 gen_server:loop/6
<6210.336.0>        riak_kv_stat_sj_7             '-'    3362782       5720          0 gen_server:loop/6
<6210.334.0>        riak_kv_stat_sj_5             '-'    3328089       3856          0 gen_server:loop/6
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131016/2daf42bc/attachment.html>


More information about the riak-users mailing list