Is it possible to include node load into the consistency hashing function in riak-core?
bryan at basho.com
Mon Oct 15 11:05:12 EDT 2012
On Tue, Oct 9, 2012 at 11:59 AM, Zhemzhitsky Sergey
<sergey_zhemzhitsky at sberbank-cib.ru> wrote:
> What I’d like to achieve is to have the preferred list be sorted based on
> the current load (maybe erlang:statistics(run_queue)) of the nodes.
> Is it possible to change the hashing function somehow so that the vnode on
> the node with the minimal load is returned as the first one?
Hi, Sergey. It would not be possible to change the hashing function to
take load into account. That function must *always* produce the same
hash for each key - that's the "consistent" part of consistent
hashing. If it didn't do that, then a request for a key might go to a
completely different vnode every time.
After the hash is computed and the preflist is generated, it would be
possible to re-sort the first N vnodes in the preference list, as you
describe. However, except for edge cases where the network is
overloaded, I don't think it will buy you much. Riak KV sends all of
those first N requests as fast as it can, and then starts waiting for
responses from them. So, getting a response to the least-loaded vnode
a few microseconds sooner is unlikely to change performance.
Although, that overloaded-network case could prove interesting and
valuable, and regular benchmarking could just prove me wrong. Also,
Pipe could almost certainly benefit from such a behavior, since its
operations usually target one vnode at a time instead of N. That is to
say, please share your results if you do try this!
Hope that helps,
More information about the riak-users