Object not found after successful PUT on S3 API

Daniel Miller dmiller at dimagi.com
Wed Jan 25 17:11:47 EST 2017


Thanks for the quick response, Luke.

There is nothing unusual about the keys. The format is a name + UUID + some
other random URL-encoded charaters, like most other keys in our cluster.

There are no errors near the time of the incident in any of the logs (the
last [error] is from over a month before). I see lots of messages like this
in console.log:

/var/log/riak/console.log
2017-01-20 15:38:10.184 [info]
<0.22902.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 2 keys during active anti-entropy exchange of {
776422744832042175295707567380525354192214163456,3} between {
776422744832042175295707567380525354192214163456,'riak-fake3 at fake3.fake.com'}
and {822094670998632891489572718402909198556462055424,'riak-
fake9 at fake9.fake.com'}
2017-01-20 15:40:39.640 [info]
<0.21789.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 1 keys during active anti-entropy exchange of {
936274486415109681974235595958868809467081785344,3} between {
959110449498405040071168171470060731649205731328,'riak-fake3 at fake3.fake.com'}
and {981946412581700398168100746981252653831329677312,'riak-
fake5 at fake5.fake.com'}
2017-01-20 15:46:40.918 [info]
<0.13986.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 2 keys during active anti-entropy exchange of {
662242929415565384811044689824565743281594433536,3} between {
685078892498860742907977265335757665463718379520,'riak-fake3 at fake3.fake.com'}
and {707914855582156101004909840846949587645842325504,'riak-
fake6 at fake6.fake.com'}
2017-01-20 15:48:25.597 [info]
<0.29943.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 2 keys during active anti-entropy exchange of {
776422744832042175295707567380525354192214163456,3} between {
776422744832042175295707567380525354192214163456,'riak-fake3 at fake3.fake.com'}
and {799258707915337533392640142891717276374338109440,'riak-
fake0 at fake0.fake.com'}

Thanks!
Daniel


On Wed, Jan 25, 2017 at 9:45 AM, Luke Bakken <lbakken at basho.com> wrote:

> Hi Daniel -
>
> This is a strange scenario. I recommend looking at all of the log
> files for "[error]" or other entries at about the same time as these
> PUTs or 404 responses.
>
> Is there anything unusual about the key being used?
> --
> Luke Bakken
> Engineer
> lbakken at basho.com
>
>
> On Wed, Jan 25, 2017 at 6:40 AM, Daniel Miller <dmiller at dimagi.com> wrote:
> > I have a 9-node Riak CS cluster that has been working flawlessly for
> about 3
> > months. The cluster configuration, including backend and bucket
> parameters
> > such as N-value are using default settings. I'm using the S3 API to
> > communicate with the cluster.
> >
> > Within the past week I had an issue where two objects were PUT resulting
> in
> > a 200 (success) response, but all subsequent GET requests for those two
> keys
> > return status of 404 (not found). Other than the fact that they are now
> > missing, there was nothing out of the ordinary with these particular to
> > PUTs. Maybe I'm missing something, but this seems like a scenario that
> > should never happen. All information included here about PUTs and GETs
> comes
> > from reviewing the CS access logs. Both objects were PUT on the same
> node,
> > however GET requests returning 404 have been observed on all nodes.
> There is
> > plenty of other traffic on the cluster involving GETs and PUTs that are
> not
> > failing. I'm unsure of how to troubleshoot further to find out what may
> have
> > happened to those objects and why they are now missing. What is the best
> > approach to figure out why an object that was successfully PUT seems to
> be
> > missing?
> >
> > Thanks!
> > Daniel Miller
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170125/4884bf9c/attachment-0002.html>


More information about the riak-users mailing list