Riak 1.3.1 crashing with segfault

Magnus Kessler mkessler at basho.com
Wed Feb 25 11:24:38 EST 2015


On 25 February 2015 at 12:11, Daniel Iwan <iwan.daniel at gmail.com> wrote:

> Hi
>
> I've checked all logs and there is nothing regarding memory issues.
> Since then I've had several Riak crashes but looks like other processes are
> failing as well
>
> Feb  2 22:05:28 node2 kernel: [20052.901884] beam.smp[1830]: segfault at
> 80000523111 ip 0000080000523111 sp 00007f03ba821be8 error 14 in
> 000013.log[7f01b6890000+1400000]
> Feb  4 10:16:01 node2 kernel: [150082.149742] MegaCli[29635]: segfault at
> 80000549b14 ip 0000080000549b14 sp 00007fff3c5f7840 error 14 in
> libtinfo.so.5.9[7f3ee87ba000+22000]
> Feb  4 16:35:27 node2 kernel: [172812.410113] MegaCli[23019]: segfault at
> 800005473d6 ip 00000800005473d6 sp 00007fffc8f49f20 error 14 in
> libtinfo.so.5.9[7fa9e90c3000+22000]
> Feb  6 14:50:38 node2 kernel: [339062.483587] sh[24637]: segfault at
> 8000040460d ip 000008000040460d sp 00007fff35251160 error 14 in
> libc-2.15.so[7fd4cc395000+1b5000]
> Feb  7 11:59:32 node2 kernel: [415077.342034] df[6393]: segfault at
> 800004029a0 ip 00000800004029a0 sp 00007fff758406d0 error 14 in
> libc-2.15.so[7f5433243000+1b5000]
>
> Feb  8 10:04:31 node2 kernel: [494451.877635] df[22107]: segfault at
> 80000404b00 ip 0000080000404b00 sp 00007ffffeee1b08 error 14 in
> libc-2.15.so[7fbd82ce2000+1b5000]
> Feb  9 16:30:20 node2 kernel: [603829.476142] ls[2873]: segfault at
> 8000040d04e ip 000008000040d04e sp 00007fffaff38c60 error 14 in
> libnss_files-2.15.so[7f257c9c4000+c000]
> Feb  9 18:26:13 node2 kernel: [ 6503.710549] beam.smp[2140]: segfault at
> 80000523a00 ip 0000080000523a00 sp 00007f3955ff2d80 error 14 in
> 000006.log[7f377f27b000+1400000]
> Feb 10 17:34:46 node2 kernel: [36949.199740] beam.smp[1877]: segfault at
> 800005650b2 ip 00000800005650b2 sp 00007faba120fa70 error 14 in
> 000009.log[7fa99827c000+1400000]
> Feb 11 20:37:15 node2 kernel: [134145.969112] beam.smp[7276]: segfault at
> 8000052287e ip 000008000052287e sp 00007ff8625c9be0 error 14 in
> 000012.log[7ff66f703000+1400000]
> Feb 13 08:58:57 node2 kernel: [ 6414.659327] beam.smp[1877]: segfault at
> 80000569cfc ip 0000080000569cfc sp 00007f55aa48bab0 error 14 in
> 000012.log[7f537dc0e000+1400000]
> Feb 15 03:20:30 node2 kernel: [133707.360153] MegaCli[7442]: segfault at
> 800005473d6 ip 00000800005473d6 sp 00007fff39ba1570 error 14 in
> libtinfo.so.5.9[7f2728f79000+22000]
>
> Feb 15 10:02:23 node2 kernel: [157782.787481] beam.smp[2023]: segfault at
> 800005239d0 ip 00000800005239d0 sp 00007f47e3fe6d68 error 14 in
> 000061.log[7f463e32f000+1400000]
> Feb 16 17:30:18 node2 kernel: [270880.717532] console-kit-dae[1548]:
> segfault at800004123e8  ip 00000800004123e8 sp 00007fffc6d8c0c0 error 14
> Feb 16 19:31:45 node2 kernel: [278156.348900] beam.smp[16617]: segfault at
> 800005650b2 ip 00000800005650b2 sp 00007f3dbe74ba70 error 14 in
> 000019.log[7f3b89f65000+1400000]
> Feb 16 21:45:34 node2 kernel: [286172.695110] sh[12432]: segfault at
> 8000040460d ip 000008000040460d sp 00007fffe6e2b3b0 error 14 in
> libc-2.15.so[7f9b57c77000+1b5000]
> Feb 17 07:27:23 node2 kernel: [  457.418215] beam.smp[1824]: segfault at
> 80000523111 ip 0000080000523111 sp 00007f36e9574be8 error 14 in
> 000021.log[7f34c24f3000+1400000]
>
> Feb 25 10:46:04 node2 kernel: [702478.037041] beam.smp[8832]: segfault at
> 80000522980 ip 0000080000522980 sp 00007fe8bbffede8 error 14 in
> 000006.log[7fe713e2c000+1400000]
>
> Riak is always touching anti-entropy files, like in the last example:
>
>
> /var/lib/riak/anti_entropy/22835963083295358096932575511191922182123945984/000006.log
>
> Could it be an SSD failing?
>
> Daniel
>
>
Hi Daniel,

Random segfaults in all sorts of different programs and libraries are a
strong indicator of hardware failure, most likely memory failure.  You
might want to check the memory modules of your server using memtest86 (
http://www.memtest86.com).

For a an online tool explaining the different segfault error codes, please
have a look at http://rgeissert.blogspot.com/p/segmentation-fault-error.html

Regards,

Magnus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150225/baa630fc/attachment-0002.html>


More information about the riak-users mailing list