Using $bucket index for listing keys

Russell Brown russell.brown at me.com
Fri Mar 11 11:33:22 EST 2016


But still requires ordered keys to do pagination. the $bucket index is only available if you’re using 2i, which requires leveldb.

On 11 Mar 2016, at 16:32, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:

> I know that 2i requires leveldb, but I'm not using my custom index, I'm using $bucket, I thought $bucket index is some kind of special internal index?
> 
> On 11 March 2016 at 18:28, Russell Brown <russell.brown at me.com> wrote:
> You can’t…can you? I mean, 2i requires level. It requires ordered keys. Which explains your problem, but you should have failed a lot earlier.
> 
> On 11 Mar 2016, at 16:26, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:
> 
> > Hi Magnus,
> >
> > The bucket type has the following properties:
> >
> > '{"props":{"backend":"fs_chunks","allow_mult":"false","r":1,"notfound_ok":"false","basic_quorum":"false"}}'
> >
> > fs_chunks backend is configured as part of riak_kv_multi_backend:
> >
> > {<<"fs_chunks">>, riak_kv_bitcask_backend, [
> >                 {data_root, "/var/lib/riak/fs_chunks"}
> >             ]},
> >
> > Objects stored are chunks of binary data (ContentType=binary/octet-stream) with a maximum size of 256Kb. Each object also has a single key/value pair in user metadata (usermeta).
> >
> > I'm not able to replicate this on my local single node setup, it only happens on a production 5 node cluster.
> >
> >
> > On 11 March 2016 at 15:53, Magnus Kessler <mkessler at basho.com> wrote:
> > Hi Oleksiy,
> >
> > Could you please share the bucket or bucket-type properties for that small bucket? If you open an issue on github, please add the properties there, too.
> >
> > Many Thanks,
> >
> > On 11 March 2016 at 13:46, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:
> > I got the recursive behavior with other, larger buckets but I had no logging so when I enabled debugging this was the first bucket to replicate the problem. I have a lot of buckets of the same type, some have many thousands keys some are small. My task is to iterate the keys (once only) of all buckets. Either with 2i or with Yokozuna.
> > On Fri, Mar 11, 2016 at 15:32 Russell Brown <russell.brown at me.com> wrote:
> > Not the answer, by why pagination for 200 keys? Why the cost of doing the query 20 times vs once?
> >
> > On 11 Mar 2016, at 13:28, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:
> >
> > > Unfortunately there are just 200 keys in that bucket. So with larger max_results I just get all the keys without continuation. I'll try to replicate this with a bigger bucket.
> > > On Fri, Mar 11, 2016 at 15:21 Russell Brown <russell.brown at me.com> wrote:
> > > That seems very wrong. Can you do me a favour and try with a larger max_results. I remember a bug with small results set, I thought it was fixed, I’m looking into the past issues, but can you try “max_results=1000” or something, and let me know what you see?
> > >
> > > On 11 Mar 2016, at 13:03, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:
> > >
> > > > Here it is without the `value` part of request:
> > > >
> > > > curl 'http://127.0.0.1:8098/types/fs_chunks/buckets/0r0e5wahrhsgpolk9stbnrqmp77fjjye.chunks/index/$bucket/_?max_results=10&continuation=g20AAAAja1AzdzJwOXpYcVoyb0F4NDhTMVNnRUpBbGJ0ZkhVdkk6MjU='
> > > >
> > > > {"keys":["4rpG2PwRTs3YqasGGYrhACBvZqTg7mQW:0","4rpG2PwRTs3YqasGGYrhACBvZqTg7mQW:2","FSEky50kr2TLkBuo1JKv6sphINYwnJfV:1","F3KcwtjG9VAtM5u8vbwBuCjuGBrPTnfq:0","RToMNlsnVKvXcawQK6BGnCAKx58pC9xX:1","UMiHx4qDR5pHWT9OgLAu1KMlFeEKbISm:0","F3KcwtjG9VAtM5u8vbwBuCjuGBrPTnfq:2","YQlRWkJPFYiLlAwhvgqOysJC3ycmQ9OA:0","kP3w2p9zXqZ2oAx48S1SgEJAlbtfHUvI:15","kP3w2p9zXqZ2oAx48S1SgEJAlbtfHUvI:25"],"continuation":"g20AAAAja1AzdzJwOXpYcVoyb0F4NDhTMVNnRUpBbGJ0ZkhVdkk6MjU="}
> > > >
> > > > On 11 March 2016 at 14:58, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:
> > > > I'm actually using PB interface, but I can replicate the problem with HTTP as in my previous email. Request with '&continuation=AAAA' returns the result set with the same continuation AAAA.
> > > >
> > > > On 11 March 2016 at 14:55, Magnus Kessler <mkessler at basho.com> wrote:
> > > > Hi Oleksiy,
> > > >
> > > > How are you performing your 2i-based key listing? Querying with pagination as shown in the documentation[0] should work.
> > > >
> > > > As an example here is the HTTP invocation:
> > > >
> > > > curl "https://localhost:8098/types/default/buckets/test/index/\$bucket/_?max_results=10&continuation=g20AAAACNTM="
> > > >
> > > > Once the end of the key list is reached, the server returns an empty keys list and no further continuation value.
> > > >
> > > > Please let me know if this works for you.
> > > >
> > > > Kind Regards,
> > > >
> > > > Magnus
> > > >
> > > >
> > > > [0]: http://docs.basho.com/riak/latest/dev/using/2i/#Querying
> > > >
> > > > On 11 March 2016 at 10:06, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:
> > > > Anyone?
> > > >
> > > > On 4 March 2016 at 19:11, Oleksiy Krivoshey <oleksiyk at gmail.com> wrote:
> > > > I have a bucket with ~200 keys in it and I wanted to iterate them with the help of $bucket index and 2i request, however I'm facing the recursive behaviour, for example I send the following 2i request:
> > > >
> > > > {
> > > > bucket: 'BUCKET_NAME',
> > > > type: 'BUCKET_TYPE',
> > > > index: '$bucket',
> > > > key: 'BUCKET_NAME',
> > > > qtype: 0,
> > > > max_results: 10,
> > > > continuation: 'AAAA'
> > > > }
> > > >
> > > > I receive 10 keys and continuation 'BBBB', I then repeat the request with continuation 'BBBB' and at this point I can receive a reply with continuation 'CCCC' or 'AAAA' or even 'BBBB' and its going in never ending recursion.
> > > >
> > > > I'm running this on a 5 node 2.1.3 cluster.
> > > >
> > > > What I'm doing wrong? Or is this not supported at all?
> > > >
> > > > Thanks!
> > > >
> > > >
> > > > _______________________________________________
> > > > riak-users mailing list
> > > > riak-users at lists.basho.com
> > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Magnus Kessler
> > > > Client Services Engineer
> > > > Basho Technologies Limited
> > > >
> > > > Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
> > > >
> > > >
> > > > _______________________________________________
> > > > riak-users mailing list
> > > > riak-users at lists.basho.com
> > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > >
> >
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> >
> >
> >
> > --
> > Magnus Kessler
> > Client Services Engineer
> > Basho Technologies Limited
> >
> > Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
> >
> 
> 





More information about the riak-users mailing list