Nodes crashes on every map reduce query.

Alexandre Ravey alexandre.ravey at gmail.com
Tue Nov 8 04:19:03 EST 2011


Hi,

Following the Riak fast track I'm stuck at the map reduce phase.

I've tried the following with devrel 4 nodes on the same box and 4
distinct virtual boxes with riak rel, same thing appends.

Cluster is working fine, can put and get data, everything explained in
the fast track work as expected until the map reduce test.

Every time I do a map reduce, 2 or 3 nodes crash, the rest is
unresponding. Not always the sames nodes, but always result in an
unusable cluster. No nodes are responding, either crashed or not
responding.

Tried the simple map reduce and the others examples.
Also tried to switch to legacy map reduce, same thing. (using
"application:set_env(riak_kv, mapred_system, legacy)").

System is Gentoo on a 3.0.6 kernel.
512 Mb of ram for each box (60 Mb used after boot, including riak).
Erlang 14B03
Riak 1.0.1 (From source, from the download page)

Only thing changed are the node's name and IP's.


Time line:
+00s
Cluster running (since hours), all nodes OK, responding to requests.

+10s
Starting map reduce query: "curl -X POST
http://192.168.0.35:8098/mapred -H "Content-Type: application/json" -d
@simple-map.json"
(simple-map.json is taken from the fast track page and the goog data
is inside the cluster, readable by a direct get)

+11s
All nodes start hogging cpu (>80%)

+12s
riak 1 and 2 crashes, see logs attached.
riak 3 is running but not responding (cant input console) and using 100% cpu.
riak 4 segfault. (see logs too)

+99s
Still in the same state, riak 3 still at 100% CPU, other obviously
still crashed :)

Attached logs for each nodes, and log for the query.

Am-I missing something?

Thanks in advance

Regards

Alexandre Ravey
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak.log
Type: application/octet-stream
Size: 367 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111108/4891bca7/attachment.log>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak1.log
Type: application/octet-stream
Size: 12469 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111108/4891bca7/attachment-0001.log>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak2.log
Type: application/octet-stream
Size: 11255 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111108/4891bca7/attachment-0002.log>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak3.log
Type: application/octet-stream
Size: 4204 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111108/4891bca7/attachment-0003.log>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak4.log
Type: application/octet-stream
Size: 4448 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111108/4891bca7/attachment-0004.log>


More information about the riak-users mailing list