Bitcask fold key issue, unhandled data corruption ?

Arnaud Wetzel arnaud.wetzel at gmail.com
Wed Oct 1 07:36:45 EDT 2014


Hello all,
I am using riak 1.4.3
I have an error during map reduce (not every time) :

{
  "phase": "listkeys",
  "error": "function_clause",
  "input": "xxx",
  "type": "error",
  "stack": "[
    {riak_kv_pipe_listkeys,keysend,
      [ error,
        {worker_crash,{badarg,[{erlang,binary_to_term,[<<>>],[]},
        {riak_kv_bitcask_backend,'-fold_keys_fun/2-fun-1-',4,
                [{file,\"src/riak_kv_bitcask_backend.erl\"},{line,461}]},
         {bitcask_nifs,keydir_fold_cont,4,[{file,\"src/bitcask_nifs.erl\"},{line,419}]},
         {bitcask_nifs,keydir_frozen,4,[{file,\"src/bitcask_nifs.erl\"},{line,265}]},
         {riak_kv_bitcask_backend,'-fold_keys/4-fun-0-',5,
                [{file,\"src/riak_kv_bitcask_backend.erl\"},{line,254}]},
         {riak_kv_worker,handle_work,3,[{file,...},...]},
        ...]},...},...],...},...]"
}

So it seems to be a corruption on a bitcask file making a <<>> bitcask
key, which is not a term_to_binary({B,K}), so the fold function crash.

I will make a manual bitcask fold in order to find the corrupted file,
but I have some questions :

- I saw that in last riak2.0 the fold function does not handle
corrupted key format either, but is their an additional consistency
check in newer version making this problem impossible ?
- What is the best way for me to reconstruct the file : delete the
partition en repair it ?
- Do you think it's a good idea to make the fold function more
resilient to a corrupted key format ?
- Do you have an idea of the possible source of the problem ? If
anybody had this issue before ?

Thank you very much

Arnaud

-- 
Arnaud Wetzel
Co-founder
Shopping Adventure
13 rue saint Anastase, 75003 Paris




More information about the riak-users mailing list