Is it possible to include node load into the consistency hashing function in riak-core?
sergey_zhemzhitsky at sberbank-cib.ru
Tue Oct 16 08:37:22 EDT 2012
Thanks a lot for the ideas!
In case of any results, I will necessarily share them.
From: Bryan Fink [mailto:bryan at basho.com]
Sent: Monday, October 15, 2012 7:05 PM
To: Zhemzhitsky Sergey
Subject: Re: Is it possible to include node load into the consistency hashing function in riak-core?
On Tue, Oct 9, 2012 at 11:59 AM, Zhemzhitsky Sergey <sergey_zhemzhitsky at sberbank-cib.ru> wrote:
> What I’d like to achieve is to have the preferred list be sorted based
> on the current load (maybe erlang:statistics(run_queue)) of the nodes.
> Is it possible to change the hashing function somehow so that the
> vnode on the node with the minimal load is returned as the first one?
Hi, Sergey. It would not be possible to change the hashing function to take load into account. That function must *always* produce the same hash for each key - that's the "consistent" part of consistent hashing. If it didn't do that, then a request for a key might go to a completely different vnode every time.
After the hash is computed and the preflist is generated, it would be possible to re-sort the first N vnodes in the preference list, as you describe. However, except for edge cases where the network is overloaded, I don't think it will buy you much. Riak KV sends all of those first N requests as fast as it can, and then starts waiting for responses from them. So, getting a response to the least-loaded vnode a few microseconds sooner is unlikely to change performance.
Although, that overloaded-network case could prove interesting and valuable, and regular benchmarking could just prove me wrong. Also, Pipe could almost certainly benefit from such a behavior, since its operations usually target one vnode at a time instead of N. That is to say, please share your results if you do try this!
Hope that helps,
CONFIDENTIALITY NOTICE: This email and any files attached to it may be conf idential. If you are not the intended recipient you are notified that using, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error please notify the sender and delete this email.
More information about the riak-users