Riak 1.3.1 crashing with segfault

Ciprian Manea ciprian at basho.com
Tue Feb 17 07:24:39 EST 2015


Hi Daniel,

Have you investigated your server's dmesg output? Segfaults can be
triggered also by memory corruption. Please check that first.


Regards,
Ciprian

On Tue, Feb 17, 2015 at 1:00 PM, Daniel Iwan <iwan.daniel at gmail.com> wrote:

> We are experiencing crash of beam.smp on one of nodes in 3-node cluster
> (ring
> 128)
> Distro is Ubuntu 12.04 with 16GB of memory (almost exclusive for Riak)
>
> ===== Sun Feb 15 10:02:23 UTC 2015
> Erlang has closed/usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup:
> Erlang has closed.
>
> Hi I've got following error in syslog
>
> Feb 15 10:02:23 node2 kernel: [157782.787481] beam.smp[2023]: segfault at
> 800005239d0 ip 00000800005239d0 sp 00007f47e3fe6d68 error 14 in
> 000061.log[7f463e32f000+1400000]
>
> I'm not sure which causes the other.
>
> 000061.log was from leveldb folder but it ahs been deleted after restart of
> Riak I believe
>
> Last thing I could find in console log is AAE activity
>
>
> 2015-02-14 15:24:18.329 [info]
> <0.7258.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 3 keys during
> active anti-entropy exchange of
> {251195593916248939066258330623111144003363405824,3} between
> {251195593916248939066258330623111144003363405824,'riak at 10.173.240.2'} and
> {262613575457896618114724618378707105094425378816,'riak at 10.173.240.3'}
> 2015-02-14 15:31:03.594 [info]
> <0.10920.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 58 keys during
> active anti-entropy exchange of
> {376793390874373408599387495934666716005045108736,3} between
> {376793390874373408599387495934666716005045108736,'riak at 10.173.240.2'} and
> {388211372416021087647853783690262677096107081728,'riak at 10.173.240.2'}
> 2015-02-14 15:33:48.637 [info]
> <0.12367.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 37 keys during
> active anti-entropy exchange of
> {422465317040964124793252646957050560369293000704,3} between
> {422465317040964124793252646957050560369293000704,'riak at 10.173.240.2'} and
> {445301280124259482890185222468242482551416946688,'riak at 10.173.240.3'}
> 2015-02-14 15:34:03.454 [info]
> <0.12546.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 37 keys during
> active anti-entropy exchange of
> {422465317040964124793252646957050560369293000704,3} between
> {433883298582611803841718934712646521460354973696,'riak at 10.173.240.2'} and
> {445301280124259482890185222468242482551416946688,'riak at 10.173.240.3'}
> 2015-02-14 15:55:18.518 [info]
> <0.23498.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 1 keys during
> active anti-entropy exchange of
> {1061872283373234151507364761270424381468763488256,3} between
> {1061872283373234151507364761270424381468763488256,'riak at 10.173.240.2'}
> and
> {1073290264914881830555831049026020342559825461248,'riak at 10.173.240.3'}
> 2015-02-14 15:59:33.522 [info]
> <0.25935.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 1 keys during
> active anti-entropy exchange of
> {1187470080331358621040493926581979953470445191168,3} between
> {1198888061873006300088960214337575914561507164160,'riak at 10.173.240.2'}
> and
> {1210306043414653979137426502093171875652569137152,'riak at 10.173.240.3'}
> 2015-02-14 15:59:48.513 [info]
> <0.26044.26>@riak_kv_exchange_fsm:key_exchange:204 Repaired 1 keys during
> active anti-entropy exchange of
> {1198888061873006300088960214337575914561507164160,3} between
> {1198888061873006300088960214337575914561507164160,'riak at 10.173.240.2'}
> and
> {1210306043414653979137426502093171875652569137152,'riak at 10.173.240.3'}
> 2015-02-14 20:08:49.674 [info]
> <0.29386.30>@riak_kv_exchange_fsm:key_exchange:204 Repaired 5 keys during
> active anti-entropy exchange of
> {148433760041419827630061740822747494183805648896,3} between
> {148433760041419827630061740822747494183805648896,'riak at 10.173.240.2'} and
> {171269723124715185726994316333939416365929594880,'riak at 10.173.240.3'}
> 2015-02-14 20:09:04.516 [info]
> <0.29501.30>@riak_kv_exchange_fsm:key_exchange:204 Repaired 5 keys during
> active anti-entropy exchange of
> {148433760041419827630061740822747494183805648896,3} between
> {159851741583067506678528028578343455274867621888,'riak at 10.173.240.2'} and
> {171269723124715185726994316333939416365929594880,'riak at 10.173.240.3'}
>
> Is it possible that AAE is causing the problems here.
>
> Regards
> Daniel
>
>
>
> --
> View this message in context:
> http://riak-users.197444.n3.nabble.com/Riak-1-3-1-crashing-with-segfault-tp4032638.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150217/4e2c390b/attachment-0002.html>


More information about the riak-users mailing list