Custom hashing (partitioning) functions

Sargun Dhillon sargun at sargun.me
Thu Jan 30 20:22:14 EST 2014


Cool, I went ahead and created an issue a little while ago:
https://github.com/basho/riak_kv/issues/811

It's not so much of a blocker for us, because the product team working
with custom partitioners and Riak. It's just something that people
might want to keep an eye out in the future when using MR, or
riak_pipe directly.

As as side-note, is there information around querying Riak directly
with Riak_pipe; I imagine it has some of the same downsides as
mapreduce. We're primarily using it as a method to avoid the
read-modify-write overheads on writing to Riak, and multi-key,
partition-specific reads (utilizing targeted 2i, because we know the
coverage as we know the partition key).

Has anyone else here done this in production?

On Tue, Jan 21, 2014 at 7:25 AM, Sean Cribbs <sean at basho.com> wrote:
> Sargun,
>
> That might indeed be a bug. Would you file it on the issue tracker?
> https://github.com/basho/riak_kv/issues
>
>
> On Mon, Jan 20, 2014 at 3:08 PM, Sargun Dhillon <sargun at sargun.me> wrote:
>>
>> So, I don't know how many people are aware of if, but Riak supports
>> custom hashing (partitioning) functions. It's exposed as a bucket
>> property (chash_keyfun), in which you can deploy your own code to hash
>> keys to ensure data locality to specific vnodes. This can come in
>> handy when doing custom mapreduce jobs, riak_pipe, or riak_core
>> applications.
>>
>> After playing around with this feature set quite a bit recently, it
>> seems like there are some places that don't entirely respect this.
>> Specifically, the place where I'm looking at right now is
>> riak_kv_pipe_get, when try_preflist gets called (if try_partition
>> falls through), it falls back to riak_core_util:chash_key (L188), as
>> opposed to looking at the riak_pipe chashfun, or the bucket chashfun.
>> What's the approach that most people have taken here with custom hash
>> functions?
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
> --
> Sean Cribbs <sean at basho.com>
> Software Engineer
> Basho Technologies, Inc.
> http://basho.com/




More information about the riak-users mailing list