List keys and multiple buckets

Izhar Ravid izhar at conduit.com
Tue Nov 15 14:10:04 EST 2011


So if I use a leveldb backend, and restrict the # of objects per bucket to dozens, and use the default bucket properties, I should be ok listing keys on the user buckets.
I guess the only open question here is Alexander's point about fd per leveldb bucket which rules out (or at least stretches) the 200M bucket option.

Thanks for your replies btw,
Izhar.
________________________________________
From: Alexander Sicular [siculars at gmail.com]
Sent: Tuesday, November 15, 2011 19:34
To: Kresten Krab Thorup
Cc: Izhar Ravid; riak-users
Subject: Re: List keys and multiple buckets

Refresh my memory, does leveldb open new files for each bucket? I'm thinking there may be some file descriptor penalty for this many buckets.

Otherwise you could set your default bucket properties to what you would want for these user buckets and then change properties if need be for a handful of other buckets you may need in your application.

Cheers,


Alexander Sicular
@siculars
http://sicuars.posterous.com


On Tuesday, November 15, 2011 at 12:17 PM, Kresten Krab Thorup wrote:

On Nov 15, 2011, at 5:07 PM, Izhar Ravid wrote:

Assuming I wish to store user information for some 200M users, and create a bucket per user. Each user bucket will contain several dozen objects.
- Will list-keys on such a user bucket be a reasonable action?

When using leveldb as the backend, list-keys is isolated to a bucket. As far as I can see, list_keys is a "blocking operation" meaning that a vnode doing list_keys can not respond to other requests while it is processing. So; in this case ... using leveldb and ~dozens of keys per bucket it does sound reasonable.

- Is list-keys isolated to a bucket?
- Is it reasonable to expect Riak to hold 200M buckets?

I'd say that it depends on your need to configure bucket properties.

Riak stores bucket properties in "the ring", which is a riak cluster's distributed state management.

The ring is not designed to have 200M entries as far as I know; and if you need to set bucket properties then the ring needs to hold those values (if you use the default properties it requires no state).

So, ... the answer is probably no in this case; 200M buckets is not reasonable, because you will likely eventually want to define properties for these buckets.

Kresten


Kind regards,
Izhar.

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list