failed get after successful put

Oleksiy Krivoshey oleksiyk at gmail.com
Thu May 1 04:57:47 EDT 2014


Thanks! Using bucket types solved the problem.


On 1 May 2014 01:44, Evan Vigil-McClanahan <emcclanahan at basho.com> wrote:

> The 2.0 solution to this is to create a bucket type
> (http://docs.basho.com/riak/2.0.0beta1/dev/advanced/bucket-types/).
> Once the type has been created pointed to that particular backend, any
> buckets of that type created later will inherit that setting, so you
> should be able to create as many of them as you like.
>
> On Wed, Apr 30, 2014 at 4:17 PM, Oleksiy Krivoshey <oleksiyk at gmail.com>
> wrote:
> > Yes, it really looks like the  above scenario. It started happening when
> I
> > only had no more than 20 custom buckets - is that too much? Whats the
> > suggested limit? Will there be other issues when using a lot of buckets
> or
> > is it just for the first gossip period?
> >
> >
> > On 1 May 2014 01:12, Evan Vigil-McClanahan <emcclanahan at basho.com>
> wrote:
> >>
> >> I suspect that, given the large heap messages, you're seeing the known
> >> issues where when a custom bucket is created and the moving that
> >> metadata around the ring takes increasingly long.  It isn't
> >> recommended to create a large number of custom buckets at the moment.
> >>
> >> What the process likely looks like is this:
> >>
> >> 1. you do the set.
> >> 2. bucket metadata starts being gossip around the ring.
> >> 3. you do the put.  it succeeds, but on some nodes, the metadata
> >> hasn't been committed, so it's put into the default backend.
> >> 4. gossip finishes.
> >> 5. you do the get.  it fails because now the bucket for this backend
> >> has made 2 replicas unreachable.
> >> 6. read repair happens, repopulating the 2 missing replicas.
> >> 7. you re-get and it works.
> >>
> >> On Wed, Apr 30, 2014 at 4:04 PM, Oleksiy Krivoshey <oleksiyk at gmail.com>
> >> wrote:
> >> > The objects are really small (50 - 500 bytes).
> >> >
> >> > It doesn't happen if the bucket was already created.
> >> > It also does't happen if I don't call SetBucket at all (so using
> default
> >> > backend and options).
> >> > And it seems it doesn't happen if I call SetBucket but don't set
> >> > `backend`
> >> > property.
> >> >
> >> >
> >> >
> >> > On 1 May 2014 00:51, Evan Vigil-McClanahan <emcclanahan at basho.com>
> >> > wrote:
> >> >>
> >> >> Does this continue if the bucket hasn't been created recently?  Does
> >> >> it matter how large the object is?  Is it particularly large in this
> >> >> case?
> >> >>
> >> >> On Wed, Apr 30, 2014 at 3:47 PM, Oleksiy Krivoshey <
> oleksiyk at gmail.com>
> >> >> wrote:
> >> >> > Hi guys,
> >> >> >
> >> >> > can someone please suggest what can be the reason for 'get'
> >> >> > immediately
> >> >> > following successful 'put' to fail?
> >> >> >
> >> >> > I'm running a fully connected, healthy, 5 node Riak 2.0-beta1
> >> >> > cluster.
> >> >> > Using
> >> >> > a multiple backend feature, so the order of operations is:
> >> >> >
> >> >> > 1. 'SetBucket' for a new bucket with a backend name. Wait for
> >> >> > successful
> >> >> > reply
> >> >> > 2. 'Put' object to bucket. Wait for successful reply (return_head:
> >> >> > true,
> >> >> > so
> >> >> > I get clock, vtag, etc back)
> >> >> > 3. 'Get' object from step 2. Returns empty response (missing
> object)
> >> >> >
> >> >> > If I repeat 'Get' after just few moments - I will successfully
> >> >> > retrieve
> >> >> > the
> >> >> > object.
> >> >> >
> >> >> > CAP options are all defaults (n: 3, etc). Happens with both custom
> >> >> > bitcask
> >> >> > and custom leveldb backends.
> >> >> >
> >> >> > There are no errors in riak logs, but a lot of the following:
> >> >> >
> >> >> > 2014-04-30 21:41:05.656 [info]
> >> >> > <0.95.0>@riak_core_sysmon_handler:handle_event:92 monitor
> large_heap
> >> >> > <0.10689.74>
> >> >> >
> >> >> >
> >> >> >
> [{initial_call,{erlang,apply,2}},{almost_current_function,{eval_bits,expr_grp,4}},{message_queue_len,0}]
> >> >> >
> >> >> >
> >> >> >
> [{old_heap_block_size,0},{heap_block_size,79467343},{mbuf_size,0},{stack_size,394},{old_heap_size,0},{heap_size,28858704}]
> >> >> >
> >> >> > --
> >> >> > Oleksiy
> >> >> >
> >> >> > _______________________________________________
> >> >> > riak-users mailing list
> >> >> > riak-users at lists.basho.com
> >> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Oleksiy Krivoshey
> >
> >
> >
> >
> > --
> > Oleksiy Krivoshey
>



-- 
Oleksiy Krivoshey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140501/e66df07c/attachment.html>


More information about the riak-users mailing list