Object not found after successful PUT on S3 API

Alexander Sicular siculars at basho.com
Fri Mar 10 12:22:34 EST 2017


I apologize for the confusion and defer to Luke on all the technical Riak
things.


On Fri, Mar 10, 2017 at 11:16 Luke Bakken <lbakken at basho.com> wrote:

> Just to clarify ...
>
> What Alexander is suggesting is what Daniel is currently using, and
> what I suspect may be causing Daniel's issues.
>
> If you wish to run a leveldb-only Riak CS cluster, you still *must*
> use the advanced.config file and the riak_cs_kv_multi_backend, and the
> other settings that I mention in my response and in the docs. Notice
> the multi_backend_prefix_list setting, for one thing.
>
> Daniel -
>
> The storage_backend setting in advanced.config will *override*
> storage_backend in riak.conf. If you wish to ensure the riak.conf
> setting is overridden, you may comment it out in that file.
>
> --
> Luke Bakken
> Engineer
> lbakken at basho.com
>
> On Fri, Mar 10, 2017 at 9:08 AM, Alexander Sicular <siculars at basho.com>
> wrote:
> >
> > Hi Daniel,
> >
> > Riak CS uses multi by default. By default the manifests are stored in
> leveldb and the blobs/chunks are stored in bitcask. If you're looking to
> force everything to level you should remove multi and use level as the
> backend setting. As Luke noted elsewhere, this configuration hasn't been
> fully tested and is not supported.
> >
> > Off the top of my head, take a look at the email Martin (?) sent about
> his modified level backend a few weeks ago for reasons why using level for
> data chunks may not be the best idea at this time.
> >
> > Thanks,
> > Alexander
> >
> > @siculars
> > http://siculars.posthaven.com
> >
> > Sent from my iRotaryPhone
> >
> > On Mar 10, 2017, at 10:50, Daniel Miller <dmiller at dimagi.com> wrote:
> >
> > Hi Luke,
> >
> > Again, thanks for your help. We are currently preparing to move all
> objects into a new cluster using the S3 API. One question on configuration:
> currently I have "storage_backend = leveldb" in my riak.conf. I assume that
> on the new cluster, in addition to using the advanced.config you provided,
> I also need to set "storage_backend = multi" in riak.conf – is that correct?
> >
> > Referring back to the subject of this thread for a bit, I'm assuming
> your current theory for why the (most recent) object went missing is
> because we have a bad backend configuration. Note that that object went
> missing weeks after it was originally written into riak, and it was
> successfully retrieved many times before it went missing. Is there a way I
> can query riak to verify your theory that the manifest was overwritten?
> Russel Brown suggested: "I wonder if you can get the manifest and then see
> if any/all of the chunks are present?" Would that help to answer the
> question about why the object went missing? Can you provide any hints on
> how to do that?
> >
> > While bad configuration may be the cause of this most recent object
> going missing, it does not explain the original two objects that went
> missing immediately after they were PUT. Those original incidents happened
> when our cluster was still using bitcask/mutli backend, so should not have
> been affected by bad configuration.
> >
> > ~ Daniel
> >
> > On Tue, Mar 7, 2017 at 3:58 PM, Luke Bakken <lbakken at basho.com> wrote:
> >>
> >> Hi Daniel,
> >>
> >> Thanks for providing all of that information.
> >>
> >> You are missing important configuration for riak_kv that can only be
> provided in an /etc/riak/advanced.config file. Please see the following
> document, especially the section to which I link here:
> >>
> >>
> http://docs.basho.com/riak/cs/2.1.1/cookbooks/configuration/riak-for-cs/#setting-up-the-proper-riak-backend
> >>
> >> [
> >>     {riak_kv, [
> >>         % NOTE: double-check this path for your environment:
> >>         {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-2.1.1/ebin"]},
> >>         {storage_backend, riak_cs_kv_multi_backend},
> >>         {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
> >>         {multi_backend_default, be_default},
> >>         {multi_backend, [
> >>             {be_default, riak_kv_eleveldb_backend, [
> >>                 {data_root, "/opt/data/ecryptfs/riak"}
> >>             ]},
> >>             {be_blocks, riak_kv_eleveldb_backend, [
> >>                 {data_root, "/opt/data/ecryptfs/riak_blocks"}
> >>             ]}
> >>         ]}
> >>     ]}
> >> ].
> >>
> >> Your configuration will look like the above. The contents of this file
> are merged with the contents of /etc/riak/riak.conf to produce the
> configuration that Riak uses.
> >>
> >> Notice that I chose riak_kv_eleveldb_backend twice because of the
> discussion you had previously about RAM usage and bitcask (
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2016-November/018801.html
> )
> >>
> >> In your current configuration, you are not using the expected prefix
> for the block data. My guess is that on very rare occasions your data
> happens to overwrite the manifest for a file. You may also have corrupted
> files at this point without noticing it at all.
> >>
> >> IMPORTANT: you can't switch from your current configuration to this new
> one without re-saving all of your data.
> >>
> >> --
> >> Luke Bakken
> >> Engineer
> >> lbakken at basho.com
> >>
> >> --
> >> Luke Bakken
> >> Engineer
> >> lbakken at basho.com
> >>
> >> On Tue, Mar 7, 2017 at 6:47 AM, Daniel Miller <dmiller at dimagi.com>
> wrote:
> >>>
> >>> Responses inline.
> >>>
> >>> On Mon, Mar 6, 2017 at 3:04 PM, Luke Bakken <lbakken at basho.com> wrote:
> >>>>
> >>>> Hi Daniel,
> >>>>
> >>>> Two questions:
> >>>>
> >>>> * Do you happen to have an /etc/riak/app.config file present?
> >>>
> >>>
> >>> No.
> >>>
> >>> Not sure if relevant, but I did notice that
> /etc/riak-cs/advanced.config does exist, which contradicts with what I said
> earlier. This is surprising to me because I did not create this file. Maybe
> it was created by the riak installer? Anyway, the content is:
> >>>
> >>> $ cat /etc/riak-cs/advanced.config
> >>> [
> >>>  {riak_cs,
> >>>   [
> >>>   ]}
> >>> ].
> >>>
> >>>>
> >>>>
> >>>> * On one of your Riak nodes, could you please execute the following
> commands:
> >>>>
> >>>> riak attach
> >>>> rp(application:get_all_env(riak_kv)).
> >>>>
> >>>> Copy the output of the previous command and attach as a separate file
> >>>> to your response. Please note that the period is significant. Use
> >>>> CTRL-C CTRL-C to exit the "riak attach" session.
> >>>
> >>>
> >>> Attached.
> >>>
> >>>
> >>
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-- 


Alexander Sicular
Solutions Architect
Basho Technologies
9175130679
@siculars
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170310/f27ddbd3/attachment-0002.html>


More information about the riak-users mailing list