Object not found after successful PUT on S3 API

Luke Bakken lbakken at basho.com
Fri Mar 10 12:16:50 EST 2017


Just to clarify ...

What Alexander is suggesting is what Daniel is currently using, and
what I suspect may be causing Daniel's issues.

If you wish to run a leveldb-only Riak CS cluster, you still *must*
use the advanced.config file and the riak_cs_kv_multi_backend, and the
other settings that I mention in my response and in the docs. Notice
the multi_backend_prefix_list setting, for one thing.

Daniel -

The storage_backend setting in advanced.config will *override*
storage_backend in riak.conf. If you wish to ensure the riak.conf
setting is overridden, you may comment it out in that file.

--
Luke Bakken
Engineer
lbakken at basho.com

On Fri, Mar 10, 2017 at 9:08 AM, Alexander Sicular <siculars at basho.com> wrote:
>
> Hi Daniel,
>
> Riak CS uses multi by default. By default the manifests are stored in leveldb and the blobs/chunks are stored in bitcask. If you're looking to force everything to level you should remove multi and use level as the backend setting. As Luke noted elsewhere, this configuration hasn't been fully tested and is not supported.
>
> Off the top of my head, take a look at the email Martin (?) sent about his modified level backend a few weeks ago for reasons why using level for data chunks may not be the best idea at this time.
>
> Thanks,
> Alexander
>
> @siculars
> http://siculars.posthaven.com
>
> Sent from my iRotaryPhone
>
> On Mar 10, 2017, at 10:50, Daniel Miller <dmiller at dimagi.com> wrote:
>
> Hi Luke,
>
> Again, thanks for your help. We are currently preparing to move all objects into a new cluster using the S3 API. One question on configuration: currently I have "storage_backend = leveldb" in my riak.conf. I assume that on the new cluster, in addition to using the advanced.config you provided, I also need to set "storage_backend = multi" in riak.conf – is that correct?
>
> Referring back to the subject of this thread for a bit, I'm assuming your current theory for why the (most recent) object went missing is because we have a bad backend configuration. Note that that object went missing weeks after it was originally written into riak, and it was successfully retrieved many times before it went missing. Is there a way I can query riak to verify your theory that the manifest was overwritten? Russel Brown suggested: "I wonder if you can get the manifest and then see if any/all of the chunks are present?" Would that help to answer the question about why the object went missing? Can you provide any hints on how to do that?
>
> While bad configuration may be the cause of this most recent object going missing, it does not explain the original two objects that went missing immediately after they were PUT. Those original incidents happened when our cluster was still using bitcask/mutli backend, so should not have been affected by bad configuration.
>
> ~ Daniel
>
> On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken <lbakken at basho.com> wrote:
>>
>> Hi Daniel,
>>
>> Thanks for providing all of that information.
>>
>> You are missing important configuration for riak_kv that can only be provided in an /etc/riak/advanced.config file. Please see the following document, especially the section to which I link here:
>>
>> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend
>>
>> [
>>     {riak_kv, [
>>         % NOTE: double-check this path for your environment:
>>         {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
>>         {storage_backend, riak_cs_kv_multi_backend},
>>         {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
>>         {multi_backend_default, be_default},
>>         {multi_backend, [
>>             {be_default, riak_kv_eleveldb_backend, [
>>                 {data_root, "/opt/data/ecryptfs/riak"}
>>             ]},
>>             {be_blocks, riak_kv_eleveldb_backend, [
>>                 {data_root, "/opt/data/ecryptfs/riak_blocks"}
>>             ]}
>>         ]}
>>     ]}
>> ].
>>
>> Your configuration will look like the above. The contents of this file are merged with the contents of /etc/riak/riak.conf to produce the configuration that Riak uses.
>>
>> Notice that I chose riak_kv_eleveldb_backend twice because of the discussion you had previously about RAM usage and bitcask (http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html)
>>
>> In your current configuration, you are not using the expected prefix for the block data. My guess is that on very rare occasions your data happens to overwrite the manifest for a file. You may also have corrupted files at this point without noticing it at all.
>>
>> IMPORTANT: you can't switch from your current configuration to this new one without re-saving all of your data.
>>
>> --
>> Luke Bakken
>> Engineer
>> lbakken at basho.com
>>
>> --
>> Luke Bakken
>> Engineer
>> lbakken at basho.com
>>
>> On Tue, Mar 7, 2017 at 6:47 AM, Daniel Miller <dmiller at dimagi.com> wrote:
>>>
>>> Responses inline.
>>>
>>> On Mon, Mar 6, 2017 at 3:04 PM, Luke Bakken <lbakken at basho.com> wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> Two questions:
>>>>
>>>> * Do you happen to have an /etc/riak/app.config file present?
>>>
>>>
>>> No.
>>>
>>> Not sure if relevant, but I did notice that /etc/riak-cs/advanced.config does exist, which contradicts with what I said earlier. This is surprising to me because I did not create this file. Maybe it was created by the riak installer? Anyway, the content is:
>>>
>>> $ cat /etc/riak-cs/advanced.config
>>> [
>>>  {riak_cs,
>>>   [
>>>   ]}
>>> ].
>>>
>>>>
>>>>
>>>> * On one of your Riak nodes, could you please execute the following commands:
>>>>
>>>> riak attach
>>>> rp(application:get_all_env(riak_kv)).
>>>>
>>>> Copy the output of the previous command and attach as a separate file
>>>> to your response. Please note that the period is significant. Use
>>>> CTRL-C CTRL-C to exit the "riak attach" session.
>>>
>>>
>>> Attached.
>>>
>>>
>>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




More information about the riak-users mailing list