Object not found after successful PUT on S3 API

Luke Bakken lbakken at basho.com
Thu Feb 9 13:02:49 EST 2017


Hi Daniel -

I don't have any ideas at this point. Has this scenario happened again?

--
Luke Bakken
Engineer
lbakken at basho.com


On Wed, Jan 25, 2017 at 2:11 PM, Daniel Miller <dmiller at dimagi.com> wrote:
> Thanks for the quick response, Luke.
>
> There is nothing unusual about the keys. The format is a name + UUID + some
> other random URL-encoded charaters, like most other keys in our cluster.
>
> There are no errors near the time of the incident in any of the logs (the
> last [error] is from over a month before). I see lots of messages like this
> in console.log:
>
> /var/log/riak/console.log
> 2017-01-20 15:38:10.184 [info]
> <0.22902.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys during
> active anti-entropy exchange of
> {776422744832042175295707567380525354192214163456,3} between
> {776422744832042175295707567380525354192214163456,'riak-fake3 at fake3.fake.com'}
> and
> {822094670998632891489572718402909198556462055424,'riak-fake9 at fake9.fake.com'}
> 2017-01-20 15:40:39.640 [info]
> <0.21789.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 1 keys during
> active anti-entropy exchange of
> {936274486415109681974235595958868809467081785344,3} between
> {959110449498405040071168171470060731649205731328,'riak-fake3 at fake3.fake.com'}
> and
> {981946412581700398168100746981252653831329677312,'riak-fake5 at fake5.fake.com'}
> 2017-01-20 15:46:40.918 [info]
> <0.13986.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys during
> active anti-entropy exchange of
> {662242929415565384811044689824565743281594433536,3} between
> {685078892498860742907977265335757665463718379520,'riak-fake3 at fake3.fake.com'}
> and
> {707914855582156101004909840846949587645842325504,'riak-fake6 at fake6.fake.com'}
> 2017-01-20 15:48:25.597 [info]
> <0.29943.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys during
> active anti-entropy exchange of
> {776422744832042175295707567380525354192214163456,3} between
> {776422744832042175295707567380525354192214163456,'riak-fake3 at fake3.fake.com'}
> and
> {799258707915337533392640142891717276374338109440,'riak-fake0 at fake0.fake.com'}
>
> Thanks!
> Daniel
>
>
>
> On Wed, Jan 25, 2017 at 9:45 AM, Luke Bakken <lbakken at basho.com> wrote:
>>
>> Hi Daniel -
>>
>> This is a strange scenario. I recommend looking at all of the log
>> files for "[error]" or other entries at about the same time as these
>> PUTs or 404 responses.
>>
>> Is there anything unusual about the key being used?
>> --
>> Luke Bakken
>> Engineer
>> lbakken at basho.com
>>
>>
>> On Wed, Jan 25, 2017 at 6:40 AM, Daniel Miller <dmiller at dimagi.com> wrote:
>> > I have a 9-node Riak CS cluster that has been working flawlessly for
>> > about 3
>> > months. The cluster configuration, including backend and bucket
>> > parameters
>> > such as N-value are using default settings. I'm using the S3 API to
>> > communicate with the cluster.
>> >
>> > Within the past week I had an issue where two objects were PUT resulting
>> > in
>> > a 200 (success) response, but all subsequent GET requests for those two
>> > keys
>> > return status of 404 (not found). Other than the fact that they are now
>> > missing, there was nothing out of the ordinary with these particular to
>> > PUTs. Maybe I'm missing something, but this seems like a scenario that
>> > should never happen. All information included here about PUTs and GETs
>> > comes
>> > from reviewing the CS access logs. Both objects were PUT on the same
>> > node,
>> > however GET requests returning 404 have been observed on all nodes.
>> > There is
>> > plenty of other traffic on the cluster involving GETs and PUTs that are
>> > not
>> > failing. I'm unsure of how to troubleshoot further to find out what may
>> > have
>> > happened to those objects and why they are now missing. What is the best
>> > approach to figure out why an object that was successfully PUT seems to
>> > be
>> > missing?
>> >
>> > Thanks!
>> > Daniel Miller
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > riak-users at lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>




More information about the riak-users mailing list