Object not found after successful PUT on S3 API

Daniel Miller dmiller at dimagi.com
Tue Feb 14 11:52:59 EST 2017


Hi Luke,

Sorry for the late response and thanks for following up. I haven't seen it
happen since. At this point I'm going to wait and see if it happens again
and hopefully get more details about what might be causing it.

Daniel

On Thu, Feb 9, 2017 at 1:02 PM, Luke Bakken <lbakken at basho.com> wrote:

> Hi Daniel -
>
> I don't have any ideas at this point. Has this scenario happened again?
>
> --
> Luke Bakken
> Engineer
> lbakken at basho.com
>
>
> On Wed, Jan 25, 2017 at 2:11 PM, Daniel Miller <dmiller at dimagi.com> wrote:
> > Thanks for the quick response, Luke.
> >
> > There is nothing unusual about the keys. The format is a name + UUID +
> some
> > other random URL-encoded charaters, like most other keys in our cluster.
> >
> > There are no errors near the time of the incident in any of the logs (the
> > last [error] is from over a month before). I see lots of messages like
> this
> > in console.log:
> >
> > /var/log/riak/console.log
> > 2017-01-20 15:38:10.184 [info]
> > <0.22902.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys
> during
> > active anti-entropy exchange of
> > {776422744832042175295707567380525354192214163456,3} between
> > {776422744832042175295707567380525354192214163456,'riak-
> fake3 at fake3.fake.com'}
> > and
> > {822094670998632891489572718402909198556462055424,'riak-
> fake9 at fake9.fake.com'}
> > 2017-01-20 15:40:39.640 [info]
> > <0.21789.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 1 keys
> during
> > active anti-entropy exchange of
> > {936274486415109681974235595958868809467081785344,3} between
> > {959110449498405040071168171470060731649205731328,'riak-
> fake3 at fake3.fake.com'}
> > and
> > {981946412581700398168100746981252653831329677312,'riak-
> fake5 at fake5.fake.com'}
> > 2017-01-20 15:46:40.918 [info]
> > <0.13986.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys
> during
> > active anti-entropy exchange of
> > {662242929415565384811044689824565743281594433536,3} between
> > {685078892498860742907977265335757665463718379520,'riak-
> fake3 at fake3.fake.com'}
> > and
> > {707914855582156101004909840846949587645842325504,'riak-
> fake6 at fake6.fake.com'}
> > 2017-01-20 15:48:25.597 [info]
> > <0.29943.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys
> during
> > active anti-entropy exchange of
> > {776422744832042175295707567380525354192214163456,3} between
> > {776422744832042175295707567380525354192214163456,'riak-
> fake3 at fake3.fake.com'}
> > and
> > {799258707915337533392640142891717276374338109440,'riak-
> fake0 at fake0.fake.com'}
> >
> > Thanks!
> > Daniel
> >
> >
> >
> > On Wed, Jan 25, 2017 at 9:45 AM, Luke Bakken <lbakken at basho.com> wrote:
> >>
> >> Hi Daniel -
> >>
> >> This is a strange scenario. I recommend looking at all of the log
> >> files for "[error]" or other entries at about the same time as these
> >> PUTs or 404 responses.
> >>
> >> Is there anything unusual about the key being used?
> >> --
> >> Luke Bakken
> >> Engineer
> >> lbakken at basho.com
> >>
> >>
> >> On Wed, Jan 25, 2017 at 6:40 AM, Daniel Miller <dmiller at dimagi.com>
> wrote:
> >> > I have a 9-node Riak CS cluster that has been working flawlessly for
> >> > about 3
> >> > months. The cluster configuration, including backend and bucket
> >> > parameters
> >> > such as N-value are using default settings. I'm using the S3 API to
> >> > communicate with the cluster.
> >> >
> >> > Within the past week I had an issue where two objects were PUT
> resulting
> >> > in
> >> > a 200 (success) response, but all subsequent GET requests for those
> two
> >> > keys
> >> > return status of 404 (not found). Other than the fact that they are
> now
> >> > missing, there was nothing out of the ordinary with these particular
> to
> >> > PUTs. Maybe I'm missing something, but this seems like a scenario that
> >> > should never happen. All information included here about PUTs and GETs
> >> > comes
> >> > from reviewing the CS access logs. Both objects were PUT on the same
> >> > node,
> >> > however GET requests returning 404 have been observed on all nodes.
> >> > There is
> >> > plenty of other traffic on the cluster involving GETs and PUTs that
> are
> >> > not
> >> > failing. I'm unsure of how to troubleshoot further to find out what
> may
> >> > have
> >> > happened to those objects and why they are now missing. What is the
> best
> >> > approach to figure out why an object that was successfully PUT seems
> to
> >> > be
> >> > missing?
> >> >
> >> > Thanks!
> >> > Daniel Miller
> >> >
> >> > _______________________________________________
> >> > riak-users mailing list
> >> > riak-users at lists.basho.com
> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170214/8e186979/attachment-0002.html>


More information about the riak-users mailing list