Differences between riak_client and riak_kv_mrc_pipe MapReduce when one node is down.

gunin at mail.mipt.ru gunin at mail.mipt.ru
Thu Jan 31 13:46:55 EST 2013


I'm try fix this bug.
1. Add fucntion for  generating ConsHashCookie in riak_kv_mrc_pipe
const_hash_cookie()->
	Random = chash:key_of(now()),
	{ok, Ring} = riak_core_ring_manager:get_my_ring(),
	NodeCount = length(riak_core_ring:all_members(Ring)),
	case riak_core_apl:get_primary_apl(Random,NodeCount,riak_pipe) of
		[{{Partition, _Node},_}|_]->
			riak_pipe_vnode:hash_for_partition(Partition);
		_->
			Random
	end.
It's taking account of which nodes are up.
2. Remove Hash = chash:key_of(ConstHashCookie) in reduce2pipe function.

After that all work fine.

May be I must write const_hash_cookie more accurate and push this changed to github?

----- Исходное сообщение -----
От: gunin at mail.mipt.ru
Кому: "Bryan Fink" <bryan at basho.com>
Копия: "Riak-Users" <riak-users at lists.basho.com>
Отправленные: Четверг, 31 Январь 2013 г 21:51:21
Тема: Re: Differences between riak_client and riak_kv_mrc_pipe	MapReduce	when one node is down.

Hello Bryan.
I'm detect problem.

Problem is in reduce phase.

1. See riak_kv_mrc_pipe:mr2pipe_phases implementation. It convert MapReduce job spec to riak_pipe spec. 
In this fun created ConstHashCookie as  Now = now(), and use it as chashfun value for fitting  in reduce phase.
This generated value actually used in riak_kv_w_reduce:done function, you try make prereduce not reduced data and send to output.
But output vnode in that case is
preflist for ConstHashCookie,i.e. some random value and n_val for this phase is always 1, that why sometimes calculated perflist for this phase is empty.

Do you have any suggestion how we can fix it?

Thanks,
Alexander Gunin.





More information about the riak-users mailing list