Object not found after successful PUT on S3 API

Matthew Von-Maszewski matthewv at basho.com
Fri Apr 28 10:15:16 EDT 2017


Something is wrong.  All instances of leveldb within a node share the total memory configuration.  The memory is equally divided between all active vnodes.  It is possible to create an OOM situation if total RAM is low and vnodes count per node is high relative to RAM size.

The best next step would be for you to execute the riak-debug program on one of the nodes known to experience OOM.  Send the resulting .tar.gz file directly to me (no need to share that with the mailing list).  I will review the memory situation and suggest options.


> On Apr 28, 2017, at 8:22 AM, Daniel Miller <dmiller at dimagi.com> wrote:
> Hi Luke,
> I'm reviving this thread from March where we discussed a new backend configuration for our riak cluster. We have had a chance to test out the new recommended configuration, and so far we have not been successful in limiting the RAM usage of leveldb with multi_backend. We have tried various configurations to limit memory usage without success.
> First try (default config).
> riak.conf: leveldb.maximum_memory.percent = 70
> Second try.
> riak.conf: leveldb.maximum_memory.percent = 40
> Third try
> riak.conf: #leveldb.maximum_memory.percent = 40 (commented out)
> advanced.config: [{eleveldb, [{total_leveldb_mem_percent, 30}]}, ...
> In all cases (under load) riak consumes all available RAM and eventually becomes unresponsive, presumably due to OOM conditions. Is there a way to limit the amount of RAM consumed by riak with the new multi_backend configuration? For example, do we need to consider ring size or other configuration parameters when calculating the value of total_leveldb_mem_percent?
> Notably, the old (storage_backend = leveldb in riak.conf, empty advanced.config) clusters have had very good RAM and disk usage characteristics. Is there any way we can make riak or riak cs avoid the rare occasions where it overwrites the manifest file while using this (non-multi) backend?
> Thank you,
> Daniel Miller
> On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken <lbakken at basho.com <mailto:lbakken at basho.com>> wrote:
> Hi Daniel,
> Thanks for providing all of that information.
> You are missing important configuration for riak_kv that can only be provided in an /etc/riak/advanced.config file. Please see the following document, especially the section to which I link here:
> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend <http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend>
> [
>     {riak_kv, [
>         % NOTE: double-check this path for your environment:
>         {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
>         {storage_backend, riak_cs_kv_multi_backend},
>         {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
>         {multi_backend_default, be_default},
>         {multi_backend, [
>             {be_default, riak_kv_eleveldb_backend, [
>                 {data_root, "/opt/data/ecryptfs/riak"}
>             ]},
>             {be_blocks, riak_kv_eleveldb_backend, [
>                 {data_root, "/opt/data/ecryptfs/riak_blocks"}
>             ]}
>         ]}
>     ]}
> ].
> Your configuration will look like the above. The contents of this file are merged with the contents of /etc/riak/riak.conf to produce the configuration that Riak uses.
> Notice that I chose riak_kv_eleveldb_backend twice because of the discussion you had previously about RAM usage and bitcask (http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html <http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html>)
> In your current configuration, you are not using the expected prefix for the block data. My guess is that on very rare occasions your data happens to overwrite the manifest for a file. You may also have corrupted files at this point without noticing it at all.
> IMPORTANT: you can't switch from your current configuration to this new one without re-saving all of your data.
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170428/e55ef9fc/attachment-0002.html>

More information about the riak-users mailing list