Urgent help with a down node.

Bryan Hughes bryan at go-factory.net
Sun Jul 7 18:01:21 EDT 2013


Hi Mark,

DOH - sorry for the lack of detail.  Didnt have enough coffee this morning.

OS:     CentOS release 6.3 (Final)
Riak:   Riak 1.2.1

Hadnt had a chance to upgrade to 1.3 yet.

Got the node back up - but not entirely sure why which is a little 
concerning.  Been verifying the data, and everything looks intact. When 
I try to run riak-admin status, I get the following (note I am not 
entirely sure this was the case when we first set the node up):

$ riak-admin status
Status failed, see log for details

The logs shows:

2013-07-07 14:55:03.858 [error] <0.12982.0>@riak_kv_console:status:173 
Status failed error:function_clause
2013-07-07 14:55:03.858 [error] emulator Error in process <0.12983.0> on 
node 'riak at 127.0.0.1' with exit value: 
{badarg,[{erlang,system_info,[global_heaps_size],[]},{riak_kv_stat,system_stats,0,[{file,"src/riak_kv_stat.erl"},{line,421}]},{riak_kv_stat,produce_stats,0,[{file,"src/riak_kv_stat.erl"},{line,320}]},{timer,tc,3,[{file,"timer...


This is on a dev cluster with an out-of-the box configuration using bitcask.

Thanks!

Bryan

On 7/7/13 2:51 PM, Mark Phillips wrote:
> Hi Bryan,
>
> I remember seeing something similar on the list a while ago. I'll dig 
> through the archives (Riak.markmail.org <http://Riak.markmail.org>) if 
> I have a few minutes later tonight.
>
> In the mean time, what version of Riak is this? And what OS?
>
> Mark
>
> On Sunday, July 7, 2013, Bryan Hughes wrote:
>
>     Anyone familiar with this error message?
>
>     2013-07-07 12:51:42 =ERROR REPORT====
>     Hintfile
>     './data/bitcask/22835963083295358096932575511191922182123945984/3.bitcask.hint'
>     contains pointer 16555635 566 that is greater than total data size
>     16556032
>     2013-07-07 12:51:45 =ERROR REPORT====
>     Hintfile
>     './data/bitcask/114179815416476790484662877555959610910619729920/3.bitcask.hint'
>     contains pointer 17817310 567 that is greater than total data size
>     17817600
>     2013-07-07 12:51:46 =ERROR REPORT====
>     Hintfile
>     './data/bitcask/159851741583067506678528028578343455274867621888/3.bitcask.hint'
>     contains pointer 7573448 567 that is greater than total data size
>     7573504
>     2013-07-07 12:51:46 =ERROR REPORT====
>     Bad datafile entry 1:
>     {ok,<<131,104,2,109,0,0,0,9,65,80,73,67,79,85,78,84,83,109,0,0,0,33,55,56,54,57,52,49,56,49,94,103,111,115,101,114,118,105,99,101,95,99>>}
>     2013-07-07 12:51:56 =ERROR REPORT====
>     Hintfile
>     './data/bitcask/730750818665451459101842416358141509827966271488/3.bitcask.hint'
>     contains pointer 13229833 581 that is greater than total data size
>     13230080
>     2013-07-07 12:52:05 =ERROR REPORT====
>     Hintfile
>     './data/bitcask/1187470080331358621040493926581979953470445191168/3.bitcask.hint'
>     contains pointer 23465420 578 that is greater than total data size
>     23465984
>     2013-07-07 12:52:06 =ERROR REPORT====
>     Hintfile
>     './data/bitcask/1210306043414653979137426502093171875652569137152/3.bitcask.hint'
>     contains pointer 27733824 578 that is greater than total data size
>     27734016
>     2013-07-07 12:52:07 =ERROR REPORT====
>     Hintfile
>     './data/bitcask/1233142006497949337234359077604363797834693083136/3.bitcask.hint'
>     contains pointer 15014008 578 that is greater than total data size
>     15014586
>     2013-07-07 12:54:43 =ERROR REPORT====
>     Bad datafile entry, discarding(383/566 bytes)
>     2013-07-07 12:54:45 =ERROR REPORT====
>     Bad datafile entry, discarding(276/567 bytes)
>     2013-07-07 12:54:46 =ERROR REPORT====
>     Bad datafile entry, discarding(42/567 bytes)
>     2013-07-07 12:54:57 =ERROR REPORT====
>     Bad datafile entry, discarding(233/581 bytes)
>     2013-07-07 12:55:06 =ERROR REPORT====
>     Bad datafile entry, discarding(550/578 bytes)
>     2013-07-07 12:55:07 =ERROR REPORT====
>     Bad datafile entry, discarding(178/578 bytes)
>     2013-07-07 12:56:00 =ERROR REPORT====
>     Error in process <0.1536.0> on node 'riak at 127.0.0.1
>     <javascript:_e({}, 'cvml', 'riak at 127.0.0.1');>' with exit value:
>     {badarg,[{erlang,system_info,[global_heaps_size],[]},{riak_kv_stat,system_stats,0,[{file,"src/riak_kv_stat.erl"},{line,421}]},{riak_kv_stat,produce_stats,0,[{file,"src/riak_kv_stat.erl"},{line,320}]},{timer,tc,3,[{file,"timer...
>
>     -- 
>
>     Bryan Hughes
>     *Go Factory*
>     http://www.go-factory.net
>
>     /"Internet Class, Enterprise Grade"/
>
>

-- 

Bryan Hughes
CTO and Founder / *Go Factory*
(415) 515-7916
http://www.go-factory.net

/"Internet Class, Enterprise Grade"/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130707/0ec2a593/attachment.html>


More information about the riak-users mailing list