Truncated bit-cask files

Arun Rajagopalan arun.v.rajagopalan at gmail.com
Tue Feb 14 09:46:16 EST 2017


Hi Magnus

RIAK crashes on startup when I have trucated bitcask file

It also crashes when the AAE files are bad too I think. Example below

2017-02-13 21:18:30 =CRASH REPORT====

  crasher:

    initial call: riak_kv_index_hashtree:init/1

    pid: <0.6037.0>

    registered_name: []

    exception exit: {{{badmatch,{error,{db_open,"Corruption: truncated
record at end of file"}}},[{hashtree,new_segment_

store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_h

ashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,124

8}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,

init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]}

,{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line

,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

    ancestors: [<0.715.0>,riak_core_vnode_sup,riak_core_sup,<0.160.0>]

    messages: []

    links: []

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 1598

    stack_size: 27

    reductions: 889

  neighbours:



Regards
Arun


On Tue, Feb 14, 2017 at 4:48 AM, Magnus Kessler <mkessler at basho.com> wrote:

> On 13 February 2017 at 19:56, Arun Rajagopalan <
> arun.v.rajagopalan at gmail.com> wrote:
>
>> Hello Riak Users
>>
>> We have situations where we dont or cant gracefully stop riak. When that
>> happens we occasionally get a truncated last-record in bitcask files
>>
>> If I delete those bitcask dir and the anti_entropy directory, Riak
>> rebuilds those bitcask files correctly
>>
>> Is there a way to rectify those borken bitcask files ?
>>
>> Thanks
>> Arun
>>
>>
>>
> Hi Arun,
>
> There should be no need to remove truncated bitcask files. Any objects up
> to the point of truncation should still be available to Riak. However, it
> may take longer for the affected partition to start up, as the
> corresponding hint file will not match the data file, and Riak will scan
> the latter to populate the key set it keeps in memory.
>
> By removing anti_entropy files you are forcing the AAE trees to be
> rebuilt. Once this has completed, AAE will fill back any objects missing
> due to the truncated bitcask files. You can also force a full partition
> repair by following the instructions from the documentation [0].
>
> Can you let me know why you cannot shut down the node gracefully? Unclean
> shutdowns should be a last resort and not part of normal operating
> procedures.
>
> Kind Regards,
>
> Magnus
>
>
> [0]: http://docs.basho.com/riak/kv/2.2.0/using/repair-recovery/
> repairs/#repairing-partitions.
>
> --
> Magnus Kessler
> Client Services Engineer
> Basho Technologies Limited
>
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170214/0dcc09bd/attachment-0002.html>


More information about the riak-users mailing list