Riak error: no_candidate_nodes, exhausted_preflist

Martin Woods mw2134 at gmail.com
Thu Oct 13 10:32:35 EDT 2011


Hi Kelly

Thanks for the reply. I've trawled through the error logs produced by Riak
but unfortunately I can't see any other useful info in there. It's possible
that the mapreduce functions were timing out, though I didn't see any
evidence of this in the logs. For sure it was under heavy load, and it was a
lonesome Riak node in a cluster of one.

We haven't seen this error since, even though we've subsequently loaded it
heavily.

Migrating to Riak 1.0 is in our sights, so hopefully this will reduce the
likelihood of this occurring even more. If we see it again, we'll let you
know.

Thanks and regards,
Martin.



On 10 October 2011 16:37, Kelly McLaughlin <kelly at basho.com> wrote:

> Martin,
>
> The error you are seeing is caused by the death of some of map phase worker
> processes. It's hard to tell exactly why they would be dying from the
> limited output, but I suspect since you mention that it happens only under
> heavy load that it's hitting the map reduce timeout or perhaps (but more
> unlikely) even exceeding the erlang process limit. Could you check for any
> other error output in the riak log directory that might be relevant? You
> could also just try increasing the timeout used in your mapreduce queries
> and see if that alleviates the problem.
>
> FWIW, I would expect you to have a better time with Riak 1.0 using the
> riak_pipe mapreduce. It was designed to be more robust for long running jobs
> and in the case when the Riak cluster is heavily loaded. Cheers.
>
> Kelly
>
>
> On Sep 15, 2011, at 9:49 AM, Martin Woods wrote:
>
> Hi all
>
> When running Riak recently on a development platform (Ubuntu 9.10 Server
> 64-bit with Riak 0.14.2) the Riak error logs reported the following
> termination:
>
>
> ** Reason for termination =
> ** {error,
>     {phase_error,
>      {error,
>       {no_candidate_nodes,exhausted_preflist,
>        [{riak_kv_mapred_planner,claim_keys,3},
>         {riak_kv_map_phase,handle_info,2},
>         {luke_phase,handle_info,3},
>         {gen_fsm,handle_msg,7},
>         {proc_lib,init_p_do_apply,3}],
>
>
> At the time we were heavily loading Riak using the riakc client with
> multiple Erlang mapreduce requests, each involving key filters. Could that
> load have caused this error? We can see in the code where this is raised but
> don't know enough about Riak under the hood to work out what the underlying
> cause is.
>
>
> Any help would be much appreciated!
>
>
> Kind regards,
>
> Martin.
>
>
>  _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111013/2ed6dfb9/attachment-0001.html>


More information about the riak-users mailing list