stream_list_keys

tom kelly ttom.kelly at gmail.com
Wed Apr 17 05:29:36 EDT 2013


Hi Christian,

Thanks for the reply!

That code snippet is very similar to what I had, except I was missing the
ack_keys call. That leads to the trivial question of why is there two keys
messages, {Id, From, {keys,Keys}} and {Id, {keys,Keys}}? I guess the ack
has been recently added for flow control?

My keys will be MD5s of data objects and I can't know them beforehand so I
need to cycle through them. Cycling through all the buckets is a bad idea
alright but I need to do it only very infrequently, if there's no other way
I'll profile it before deciding to use it or not. So the important question
for my application, is there a recommended way to retrieve all the keys in
a bucket?

//Tom.


On Tue, Apr 16, 2013 at 9:45 PM, Christian Dahlqvist <christian at basho.com>wrote:

> Hi Tom,
>
> Here is an example that runs from the Riak console and dumps all keys of a
> bucket to a file and shows how stream_list_keys can be used:
>
> -module(keylister).
>
> -export([list_keys_to_file/2]).
>
> -define(TIMEOUT, 10000000).
>
> %% @spec list_keys_to_file(binary(), string()) ->
> %%       ok | {error, term()}
> list_keys_to_file(Bucket, File) when is_binary(Bucket) andalso is_list(File) ->
>     {ok, C} = riak:local_client(),
>     case file:open(File, [write]) of
>         {ok, IoDev} ->
>             C:stream_list_keys(Bucket, ?TIMEOUT),
>             write_keys_to_file(IoDev);
>         {error, Reason} ->
>             {error, Reason}
>     end.
>
> write_keys_to_file(IoDev) ->
>     receive
>         {_, {keys,Keys}} ->
>             Output = [[K, <<"\n">>] || K <- Keys],
>           file:write(IoDev, Output),
>             write_keys_to_file(IoDev);
>         {_, From, {keys,Keys}} ->
>             riak_kv_keys_fsm:ack_keys(From),
>             Output = [[K, <<"\n">>] || K <- Keys],
>     	    file:write(IoDev, Output),
>             write_keys_to_file(IoDev);
>         {_, done} ->
>             file:close(IoDev);
>         M ->
>             {error, unexpected_message, M}
>     end.
>
>
> Although there are times when this is useful to use, it is NOT recommended
> for production use as it has to traverse ALL keys stored in the cluster,
> see:
> http://docs.basho.com/riak/latest/references/apis/http/HTTP-List-Keys/
>
> Best regards,
>
> Christian
>
>
>
>
> On 16 Apr 2013, at 15:49, tom kelly <ttom.kelly at gmail.com> wrote:
>
> Hi List,
> I'm experimenting with riak 1.3.1 and I've started a cluster with two
> nodes, connected from a dev node and wrote & read a few test keys and
> everything was looking good. I was in the process of writing a few
> functions we'll need in our system, one of which needs to cycle through all
> the keys at startup, stream_list_keys looked like it would do the job but
> I've just noticed this strange behavior:
>
> (a at dev1.bfast.com)41>
> (a at dev1.bfast.com)41> DB:stream_list_keys(<<"test">>).
> {ok,2583625}
> (a at dev1.bfast.com)42> DB:stream_list_keys(<<"test">>).
> {ok,98542259}
> (a at dev1.bfast.com)43> DB:stream_list_keys(<<"test">>).
> {ok,83513611}
> (a at dev1.bfast.com)44> DB:stream_list_keys(<<"test">>).
> {ok,112529022}
> (a at dev1.bfast.com)45> DB:stream_list_keys(<<"test">>).
> {ok,66267591}
> (a at dev1.bfast.com)46> DB:stream_list_keys(<<"test">>).
> {ok,90620806}
> (a at dev1.bfast.com)47> DB:stream_list_keys(<<"test">>).
> {ok,24108838}
> (a at dev1.bfast.com)48> DB:stream_list_keys(<<"test">>).
> {ok,17013899}
> (a at dev1.bfast.com)49> DB:stream_list_keys(<<"test">>).
> {ok,48399864}
> (a at dev1.bfast.com)50> flush().
> Shell got {2583625,{<5272.17627.0>,#Ref<5272.0.0.95523>},{keys,[<<1>>]}}
> Shell got {98542259,{<5272.17656.0>,#Ref<5272.0.0.95733>},{keys,[<<1>>]}}
> Shell got {83513611,{<5272.17670.0>,#Ref<5272.0.0.95901>},{keys,[<<1>>]}}
> Shell got {112529022,{<5272.17683.0>,#Ref<5272.0.0.96064>},{keys,[]}}
> Shell got {66267591,{<5272.17698.0>,#Ref<5272.0.0.96224>},{keys,[]}}
> Shell got {90620806,{<5272.17708.0>,#Ref<5272.0.0.96375>},{keys,[<<1>>]}}
> Shell got {24108838,{<5272.17717.0>,#Ref<5272.0.0.96482>},{keys,[<<1>>]}}
> Shell got {17013899,{<5272.17731.0>,#Ref<5272.0.0.96638>},{keys,[<<1>>]}}
> Shell got {48399864,{<5272.17736.0>,#Ref<5272.0.0.96767>},{keys,[]}}
> ok
>
> The key <<1>> was written over an hour ago and these successive calls to
> stream_list_keys are ~2 seconds apart, The key isn't always in the first
> keys message returned and there's only ever one message returned per call.
> From the comments in the source I expected all the keys to be returned
> followed by the done message.
> Am I misunderstanding or misusing anything here? Any pointers welcomed.
> Thanks,
> //TTom.
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130417/3791f7c4/attachment.html>


More information about the riak-users mailing list