build 393: adding server results in missing data
jonathan_lee at comcast.com
Fri Nov 13 15:46:16 EST 2009
I also am seeing the same thing. Sometimes, all of my objects
reappear after a few minutes, but usually only about 90% of my objects
In the handoff case, I imagine that a node that has freshly joined the
cluster should either be marked "offline" or pass on reads for keys it
doesn't have until handoff is complete, right? Are the in-progress
improvements along these lines?
On Fri, Nov 13, 2009 at 2:28 PM, B. Todd Burruss <bburruss at real.com> wrote:
> i let my instances run all night and the data still hasn't replicated
> completely. my dataset is 1000 2k blocks of data. it should be fairly
> quick i would assume. but even if it isn't fast (say i had 100,000,000
> 2k blocks) the data should always be available during a handoff. right?
> thx Justin for getting back to us
> On Fri, 2009-11-13 at 14:04 -0500, Justin Sheehy wrote:
>> My guess is that you're seeing the result of a large amount of handoff
>> in process.
>> When you add a new node, the old nodes will slowly notice the change
>> in ownership and begin sending the appropriate data to the new node as
>> it claims partitions and rebalances ownership. If you issue these
>> requests after ownership has changed but before handoff has completed,
>> there will be some cases where some of the ideal owning nodes for
>> given keys will not yet have received the data they are being asked
>> If that is the case, the problem should go away after some minutes
>> have passed. Is that what you observe?
>> It is also good timing that you mention this, as we are working on
>> some significant improvements to the performance and behavior of
>> handoff right now.
> riak-users mailing list
> riak-users at lists.basho.com
More information about the riak-users