Object not found after successful PUT on S3 API

Magnus Kessler mkessler at basho.com
Fri Apr 28 12:28:48 EDT 2017


On 28 April 2017 at 15:15, Matthew Von-Maszewski <matthewv at basho.com> wrote:

> Daniel,
>
> Something is wrong.  All instances of leveldb within a node share the
> total memory configuration.  The memory is equally divided between all
> active vnodes.  It is possible to create an OOM situation if total RAM is
> low and vnodes count per node is high relative to RAM size.
>
> The best next step would be for you to execute the riak-debug program on
> one of the nodes known to experience OOM.  Send the resulting .tar.gz file
> directly to me (no need to share that with the mailing list).  I will
> review the memory situation and suggest options.
>
> Matthew
>
> On Apr 28, 2017, at 8:22 AM, Daniel Miller <dmiller at dimagi.com> wrote:
>
> Hi Luke,
>
> I'm reviving this thread from March where we discussed a new backend
> configuration for our riak cluster. We have had a chance to test out the
> new recommended configuration, and so far we have not been successful in
> limiting the RAM usage of leveldb with multi_backend. We have tried various
> configurations to limit memory usage without success.
>
> First try (default config).
> riak.conf: leveldb.maximum_memory.percent = 70
>
> Second try.
> riak.conf: leveldb.maximum_memory.percent = 40
>
> Third try
> riak.conf: #leveldb.maximum_memory.percent = 40 (commented out)
> advanced.config: [{eleveldb, [{total_leveldb_mem_percent, 30}]}, ...
>
> In all cases (under load) riak consumes all available RAM and eventually
> becomes unresponsive, presumably due to OOM conditions. Is there a way to
> limit the amount of RAM consumed by riak with the new multi_backend
> configuration? For example, do we need to consider ring size or other
> configuration parameters when calculating the value of
> total_leveldb_mem_percent?
>
> Notably, the old (storage_backend = leveldb in riak.conf, empty
> advanced.config) clusters have had very good RAM and disk usage
> characteristics. Is there any way we can make riak or riak cs avoid the
> rare occasions where it overwrites the manifest file while using this
> (non-multi) backend?
>
> Thank you,
> Daniel Miller
>
>
> On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken <lbakken at basho.com> wrote:
>
>> Hi Daniel,
>>
>> Thanks for providing all of that information.
>>
>> You are missing important configuration for riak_kv that can only be
>> provided in an /etc/riak/advanced.config file. Please see the following
>> document, especially the section to which I link here:
>>
>> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/
>> riak-for-cs/#setting-up-the-proper-riak-backend
>>
>> [
>>     {riak_kv, [
>>         *% NOTE: double-check this path for your environment:*
>>         {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
>>         {storage_backend, riak_cs_kv_multi_backend},
>>         {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
>>         {multi_backend_default, be_default},
>>         {multi_backend, [
>>             {be_default, riak_kv_eleveldb_backend, [
>>                 {data_root, "/opt/data/ecryptfs/riak"}
>>             ]},
>>             {be_blocks, riak_kv_eleveldb_backend, [
>>                 {data_root, "/opt/data/ecryptfs/riak_blocks"}
>>             ]}
>>         ]}
>>     ]}
>> ].
>>
>> Your configuration will look like the above. The contents of this file
>> are merged with the contents of /etc/riak/riak.conf to produce the
>> configuration that Riak uses.
>>
>> Notice that I chose riak_kv_eleveldb_backend twice because of the
>> discussion you had previously about RAM usage and bitcask (
>> http://lists.basho.com/pipermail/riak-users_lists.basho.com
>> /2016-November/018801.html)
>>
>> In your current configuration, you are not using the expected prefix for
>> the block data. My guess is that on very rare occasions your data happens
>> to overwrite the manifest for a file. You may also have corrupted files at
>> this point without noticing it at all.
>>
>> *IMPORTANT:* you can't switch from your current configuration to this
>> new one without re-saving all of your data.
>>
>
>
Hi Daniel,

In a typical CS setup, both the bitcask and leveldb backends are in use.
Bitcasks memory use depends directly on the amount of keys stored in this
backend. Can you please let me know how many files and how much total data
is stored in the CS cluster?

Kind Regards,

Magnus


-- 
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170428/f47fb435/attachment-0002.html>


More information about the riak-users mailing list