Riak CS: Undeletable broken buckets

Andrew Stone astone at basho.com
Sun Jul 6 22:21:15 EDT 2014


Hi Toby,

We've seen this scenario before. It occurs because riak-cs stores bucket
information in 2 places on disk:
  1) Inside the user record (for bucket permissions)
  2) Inside a global list of buckets, since each bucket must be unique

What has happened most likely is that the bucket is no longer stored for
the given user, but still in the global list of bucket. It shows up in
bucket lists, but the current user doesn't have permission to actually do
anything with it. Essentially you have partially written (or partially
deleted) data. I believe the only time we saw this was when Riak was
configured with {allow_mult, false} which is an invalid setting when used
with riak-cs. Riak-cs uses siblings intelligently to merge conflicting
data, and without that it's possible to end up in these types of scenarios.
Later versions of riak-cs should refuse to run with {allow_mult, false}.
I'd check your riak config to see if that is the case here.

We actually have scripts to detect and remove the bad buckets that we've
used in support. We can probably get you a copy if you want. Just let me
know. And make sure when running in production that allow_mult = true.

-Andrew



On Sun, Jul 6, 2014 at 9:59 PM, Toby Corkindale <toby at dryft.net> wrote:

> Hi,
> At some point we've managed to create a couple of buckets that don't
> work and can't be deleted (in a development/testing cluster, not
> production).
> They show up with both 's3cmd ls' or by querying the HTTP API for a
> user's buckets.
> However attempting to list files in the bucket, or removing the
> bucket, or recreating the bucket, fails.
>
> It's not in a production cluster so it's not a huge concern to me, but
> thought I'd report the bug here in case it's of interest to you.
> Riak 1.4.9-1 and Riak-CS 1.4.5-1 on Ubuntu 12.04 LTS.
>
> $ s3cmd ls
> 2014-02-07 00:07  s3://test5403
> 2013-12-13 07:25  s3://test9857
>
> $ s3cmd ls s3://test5403
> ERROR: Bucket 'test5403' does not exist
> tobyc at adonai:~$ s3cmd ls s3://test9857
> ERROR: Bucket 'test9857' does not exist
>
> $ s3cmd rb s3://test5403
> ERROR: Bucket 'test5403' does not exist
> Bucket 's3://test5403/' removed
>
> $ s3cmd ls
> 2014-02-07 00:07  s3://test5403
> 2013-12-13 07:25  s3://test9857
>
> $ s3cmd mb s3://test5403
> Bucket 's3://test5403/' created
>
> $ s3cmd ls s3://test5403
> ERROR: Bucket 'test5403' does not exist
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140706/f068ad74/attachment.html>


More information about the riak-users mailing list