Nodes crashes on every map reduce query

Alexandre Ravey alexandre.ravey at gmail.com
Wed Nov 9 06:18:14 EST 2011


Hi,

Following the Riak fast track I'm stuck at the map reduce phase.

I've tried the following with devrel 4 nodes on the same box and 4
distinct virtual boxes with riak rel, same thing appends.

Cluster is working fine, can put and get data, everything explained in
the fast track work as expected until the map reduce test.

Every time I do a map reduce, 2 or 3 nodes crash, the rest is
unresponding. Not always the sames nodes, but always result in an
unusable cluster. No nodes are responding, either crashed or not
responding.

Tried the simple map reduce and the others examples.
Also tried to switch to legacy map reduce, same thing. (using
"application:set_env(riak_kv, mapred_system, legacy)").

System is Gentoo on a 3.0.6 kernel.
512 Mb of ram for each box (60 Mb used after boot, including riak).
Erlang 14B03
Riak 1.0.1 (From source, from the download page)

Only thing changed are the node's name and IP's.


Time line:
+00s
Cluster running (since hours), all nodes OK, responding to requests.

+10s
Starting map reduce query: "curl -X POST
http://192.168.0.35:8098/mapred -H "Content-Type: application/json" -d
@simple-map.json"
(simple-map.json is taken from the fast track page and the goog data
is inside the cluster, readable by a direct get)

+11s
All nodes start hogging cpu (>80%)

+12s
riak 1 and 2 crashes, see logs attached.
riak 3 is running but not responding (cant input console) and using 100% cpu.
riak 4 segfault. (see logs too)

+99s
Still in the same state, riak 3 still at 100% CPU, other obviously
still crashed :)

See logs on pastbin:
Query: http://pastebin.com/nxLjmvbK
Riak1: http://pastebin.com/cGVzc15d
Riak2: http://pastebin.com/WP0yKRgW
Riak3: http://pastebin.com/2jmyFvGk
Riak4: http://pastebin.com/a6sTqQuf

Am-I missing something?

Thanks in advance

Regards

Alexandre Ravey




More information about the riak-users mailing list