Basho Product Alert: LevelDB SEGV in Riak KV 2.1.3

Seema Jethani seema at basho.com
Wed Apr 13 15:06:57 EDT 2016


 Overview

Riak KV (aka Riak / Riak EE) 2.1.3 has a race condition with opening
LevelDB databases and may crash with a segmentation violation (SIGSEGV).
LevelDB databases are opened on primary or fallback vnode startup and
during AAE expiration The issue is possible but much less likely on the
Riak TS 2.0, Riak KV 2.0 and earlier Riak KV 2.1 series.

Description

Beginning with Riak KV 2.1.3, a list of opened databases is used by the
GroomingCompactions feature to look for databases with low write activity
that can benefit from being compacted.

The race condition occurs because newly opened databases are added to the
list before their initialization is complete. Scanning for
GroomedCompaction candidates takes place once every minute or whenever one
completes. If the initializing database is accessed during initialization,
it may reference uninitialized memory and crash with a segmentation
violation.

The open database list was originally added to support the FlexCache
feature in Riak 2.0. There is a much smaller chance this race condition
will occur in Riak 2.0, since it requires more than one database to open or
close at the same time.

This issue is corrected by ensuring that the list of open databases is
updated after the initialization of the database is complete. The
correction is shipped within Riak KV 2.0.7, Riak KV 2.1.4, and Riak TS
1.3.0

Affected Users

- Users running Riak KV 2.1.3 with the LevelDB backend, with or without AAE
enabled.

- Users running Riak KV 2.1.3 with other backends (multi, memory, bitcask)
and AAE enabled could also hit this issue.

- Users running Riak KV 2.0.0 and higher with AAE enabled are also
affected, but the likelihood of hitting the race condition is much lower.

- Users running Riak TS 1.0.0 and higher are also affected, but the
likelihood of hitting the race condition is much lower.

Impact

An affected node will reference uninitialized memory and will likely exit
with a segmentation violation. It may be logged by the operating system as
a crash by the beam.smp process.

Mitigation Strategy

Upgrade to a release of Riak KV at 2.0.7⁄2.1.4 or later to resolve the
issue, or patch the eleveldb.so shared library.

Riak KV 2.1.4 is the same as Riak KV 2.1.3 with a corrected version of the
eleveldb.so library. Riak KV 2.1.4 is available at our downloads site here:
http://docs.basho.com/riak/kv/2.1.4/downloads/

Riak KV 2.0.7 is a regular patch release that is in progress and will be
released in the next few weeks.

If you are unable to upgrade/patch, the occurrence can be reduced by
temporarily disabling AAE until the node is upgraded or patched.

Instructions

Installation and upgrade instructions can be found here
http://docs.basho.com/community/productadvisories/leveldbsegfault/

-- 
Seema Jethani
Director of Product Management, Basho <http://basho.com>
4083455739 | @seemaj <http://twitter.com/seemaj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160413/b68f27f9/attachment-0002.html>


More information about the riak-users mailing list