stream_list_keys

Christian Dahlqvist christian at basho.com
Tue Apr 16 16:45:34 EDT 2013


Hi Tom,

Here is an example that runs from the Riak console and dumps all keys of a bucket to a file and shows how stream_list_keys can be used:

-module(keylister).
 
-export([list_keys_to_file/2]).
 
-define(TIMEOUT, 10000000).
 
%% @spec list_keys_to_file(binary(), string()) ->
%%       ok | {error, term()}
list_keys_to_file(Bucket, File) when is_binary(Bucket) andalso is_list(File) ->
    {ok, C} = riak:local_client(),
    case file:open(File, [write]) of 
        {ok, IoDev} ->
            C:stream_list_keys(Bucket, ?TIMEOUT),
            write_keys_to_file(IoDev);
        {error, Reason} ->
            {error, Reason}
    end.
 
write_keys_to_file(IoDev) ->
    receive
        {_, {keys,Keys}} ->
            Output = [[K, <<"\n">>] || K <- Keys],
          file:write(IoDev, Output),
            write_keys_to_file(IoDev);
        {_, From, {keys,Keys}} ->
            riak_kv_keys_fsm:ack_keys(From),
            Output = [[K, <<"\n">>] || K <- Keys],
    	    file:write(IoDev, Output),
            write_keys_to_file(IoDev);
        {_, done} ->
            file:close(IoDev);
        M ->
            {error, unexpected_message, M}
    end.

Although there are times when this is useful to use, it is NOT recommended for production use as it has to traverse ALL keys stored in the cluster, see: http://docs.basho.com/riak/latest/references/apis/http/HTTP-List-Keys/

Best regards,

Christian




On 16 Apr 2013, at 15:49, tom kelly <ttom.kelly at gmail.com> wrote:

> Hi List,
> I'm experimenting with riak 1.3.1 and I've started a cluster with two nodes, connected from a dev node and wrote & read a few test keys and everything was looking good. I was in the process of writing a few functions we'll need in our system, one of which needs to cycle through all the keys at startup, stream_list_keys looked like it would do the job but I've just noticed this strange behavior:
> 
> (a at dev1.bfast.com)41> 
> (a at dev1.bfast.com)41> DB:stream_list_keys(<<"test">>).
> {ok,2583625}
> (a at dev1.bfast.com)42> DB:stream_list_keys(<<"test">>).
> {ok,98542259}
> (a at dev1.bfast.com)43> DB:stream_list_keys(<<"test">>).
> {ok,83513611}
> (a at dev1.bfast.com)44> DB:stream_list_keys(<<"test">>).
> {ok,112529022}
> (a at dev1.bfast.com)45> DB:stream_list_keys(<<"test">>).
> {ok,66267591}
> (a at dev1.bfast.com)46> DB:stream_list_keys(<<"test">>).
> {ok,90620806}
> (a at dev1.bfast.com)47> DB:stream_list_keys(<<"test">>).
> {ok,24108838}
> (a at dev1.bfast.com)48> DB:stream_list_keys(<<"test">>).
> {ok,17013899}
> (a at dev1.bfast.com)49> DB:stream_list_keys(<<"test">>).
> {ok,48399864}
> (a at dev1.bfast.com)50> flush().
> Shell got {2583625,{<5272.17627.0>,#Ref<5272.0.0.95523>},{keys,[<<1>>]}}
> Shell got {98542259,{<5272.17656.0>,#Ref<5272.0.0.95733>},{keys,[<<1>>]}}
> Shell got {83513611,{<5272.17670.0>,#Ref<5272.0.0.95901>},{keys,[<<1>>]}}
> Shell got {112529022,{<5272.17683.0>,#Ref<5272.0.0.96064>},{keys,[]}}
> Shell got {66267591,{<5272.17698.0>,#Ref<5272.0.0.96224>},{keys,[]}}
> Shell got {90620806,{<5272.17708.0>,#Ref<5272.0.0.96375>},{keys,[<<1>>]}}
> Shell got {24108838,{<5272.17717.0>,#Ref<5272.0.0.96482>},{keys,[<<1>>]}}
> Shell got {17013899,{<5272.17731.0>,#Ref<5272.0.0.96638>},{keys,[<<1>>]}}
> Shell got {48399864,{<5272.17736.0>,#Ref<5272.0.0.96767>},{keys,[]}}
> ok
> 
> The key <<1>> was written over an hour ago and these successive calls to stream_list_keys are ~2 seconds apart, The key isn't always in the first keys message returned and there's only ever one message returned per call. From the comments in the source I expected all the keys to be returned followed by the done message.
> Am I misunderstanding or misusing anything here? Any pointers welcomed.
> Thanks,
> //TTom.
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130416/a2a7a86b/attachment.html>


More information about the riak-users mailing list