auto-apply different backend config via bucket name prefix

Luke Bakken lbakken at basho.com
Mon May 19 11:13:59 EDT 2014


Hi Gavin,

You would have to create the tenant-specific backends in advance. I
would recommend creating a minimum set of backends necessary and using
a tenant-specific object prefix as the tenant-specific bucket. For
instance, obj_1 would be saved using key service1_bucket1_obj1.

In general, you should keep the number of buckets with custom
properties (like backend) to a minimum as having many of them could
cause gossip issues in your cluster. Going forward, using bucket types
in Riak 2.0 will help with this issue.
--
Luke Bakken
CSE
lbakken at basho.com


On Thu, May 8, 2014 at 7:32 PM, Gavin Huang <shuminghuang at gmail.com> wrote:
> Hi, Luke
>
>     Thanks for your reply, I realized that also after some investigation.
> I was wondering if there is other approaches to achieve my requirement.
>
> Here is my situation:
> 1. A tenant can save object to any bucket that with name start with tenant's
> name, for example service1_bucket1, service2_bucket2.
>
> 2. these bucket will not be created explicitly, it will leverage riak's
> feature, that saving an object to a non-exits bucket will have this bucket
> created.
>
> my requirement is have different tenant use different backend (both data
> storage type and data dir).
>
> seems the approach will not working in my case, and the pre-hook is for
> buckets that already exists, so is there any other way that i can control
> the behavior when a bucket is being created?
>
>
>
>
> Thanks.
> Gavin
>
>
>
>
> On Thu, May 8, 2014 at 10:43 PM, Luke Bakken <lbakken at basho.com> wrote:
>>
>> Hi Gavin,
>>
>> The multi_backend_prefix_list feature is for Riak CS's use and is not
>> intended for general use. That being said, the following two configuration
>> lines will direct you to how Riak CS accomplishes this:
>>
>> {add_paths, ["/usr/lib/riak-cs/lib/riak_cs-1.4.5/ebin"]},
>> {storage_backend, riak_cs_kv_multi_backend},
>>
>> --
>> Luke Bakken
>> CSE
>> lbakken at basho.com
>>
>>
>>
>> On Wed, Apr 30, 2014 at 2:21 AM, Gavin Huang <shuminghuang at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I know that Riak2 delivered a new feature of bucket type, buckets with
>> > same
>> > bucket type will have same bucket properties and different bucket type
>> > can
>> > have different bucket properties.
>> > however, since Riak2 is still on it's beta version, so think may be I
>> > should
>> > wait for a while before stable version released, so i would stick with
>> > 1.4.7
>> > for now.
>> >
>> > however, since more than one application is using same riak cluster
>> > right
>> > now, and different application need different storage engine, for
>> > example
>> > application1 need leveldb, while another one application2 need to use
>> > bitcask. so think it's better to use multi-backend in riak cluster.
>> >
>> > i can achieve that by change app.config:
>> > {storage_backend, riak_kv_multi_backend},
>> > %%{multi_backend_prefix_list, [{<<"service1_:">>, be_blocks}]},
>> > {multi_backend_default, <<"be_default">>},
>> > {multi_backend, [
>> >   {<<"be_default">>, riak_kv_eleveldb_backend, [
>> >     {max_open_files, 50},
>> >       {data_root, "/var/lib/riak/leveldb"}
>> >   ]},
>> >     {<<"be_blocks">>, riak_kv_bitcask_backend, [
>> >       {data_root, "/var/lib/riak/bitcask"}
>> >   ]},
>> >     {<<"for_service1">>, riak_kv_bitcask_backend, [
>> >       {data_root, "/var/lib/riak/service1"}
>> >   ]},
>> >     {<<"for_service2">>, riak_kv_eleveldb_backend, [
>> >       {data_root, "/var/lib/riak/service2"}
>> >   ]}
>> > ]},
>> > and then:
>> > curl -XPUT http://riak:8098/riak/manually_created/ -d
>> > '{"props":{"backend":"for_service1"}}'
>> >
>> > the problem is I could like to have riak automatically apply backend
>> > setting
>> > by the bucket name prefix, for example:
>> > service1_bucket11 would use <<"for_service1">> as backend, while
>> > service2_bucket21 would use <<"for_service2">> as backend.
>> >
>> > I know riakcs is using the similar mechanism from:
>> >
>> > http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Riak/
>> > {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
>> >
>> > but seems this does not work for me:
>> > {multi_backend_prefix_list, [{<<"service1">>, <<"for_service1">>}]},
>> >
>> > the newly created bucket did not use bitcask backend as expected, but
>> > using
>> > default leveldb.
>> >
>> > i'm not familiar with Erlang, can't read through the riak source code,
>> > don't
>> > why it does not work, or did i mis-understand how RiakCS achieve this?
>> >
>> > Thanks.
>> > Gavin
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > riak-users at lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>>
>




More information about the riak-users mailing list