Expected vs Actual Bucket Behavior

John D. Rowell me at jdrowell.com
Wed Jul 21 10:17:17 EDT 2010

Justin, I think we could address both 1) and 2) in another way. The "real
world" need seems to be restricting the scope of costly operations like
walking a huge list of keys. Either having distinct buckets or reliable
lists of keys could solve the problem.

But simply looking up the (Dynamo) heritage for Riak shows that this is a
solved problem in other platforms. For instance Amazon's S3 service does
actual separation of keys per bucket (which is what improving Bitcask in the
way you described will achieve) but also has a more subtle, and
complementary solution: prefixes. When you search inside a bucket you can
specify a key prefix that will serve as a filter.

While I have no idea if their internal implementation of prefixes is
efficient, I see no reason why we can't have the same functionality in Riak
and guarantee it is efficient even in distributed cases. If keys are limited
to some manageable length (say 255 char strings) they could be indexed into
a distributed (only eventually consistent of course) ordered list or btree
that could be efficiently queried for ranges. Then to the application writer
it's just a matter of providing sane keys that will allows for scoping
searchs in several ways (e.g. per year, per year+month, per lot, etc.)

The suggestion above would not solve all indexing needs but would surely
allow for the current ones to be refactored into cases that would be
efficiently solvable.


2010/7/21 Justin Sheehy <justin at basho.com>

> I think that we are all (myself included) getting two different issues
> a bit mixed up in this discussion:
> 1: storing an implicit index of keys in the Riak key/value store
> 2: making buckets separate in that a per-bucket operation's
> performance would not be affected by the content of other buckets
> The thread started out with a request for #2, but included a
> suggestion to do #1.  These are actually two different topics.
> The first issue, implicitly storing a big index of keys, is
> impractical in a distributed key/value storage system that has Riak's
> availability goals.  We are very unlikely to implement this as
> described in the near future.  However, we very much recognize that
> there are many different ways that people would like to find their
> data.  In that light, we are working on multiple different efforts
> that will use the Riak core to provide data storage with more than
> just "simple" key/value access.
> The second issue, of isolating buckets, is a much simpler design
> choice and is also a per-backend implementation detail.  We can create
> and provide an alternative bitcask adapter that does this.  It will be
> a real tradeoff: in exchange for buckets not impacting each other as
> much, the system will consume more filehandles, be a bit less
> efficient at rebalancing, and will generally make buckets no longer
> "free".  This is a reasonable tradeoff in either direction for various
> applications, and I support making it available as a choice.  I have
> created a bugzilla entry to track it:
> https://issues.basho.com/show_bug.cgi?id=480
> I hope that this helps to clarify the issue.
> -Justin
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20100721/5e2165bb/attachment.html>

More information about the riak-users mailing list