Slow performance on getting via 2i

Russell Brown russell.brown at me.com
Tue Mar 13 10:05:12 EDT 2012


Hi Daniel,
Sorry for the slow reply.

On 12 Mar 2012, at 11:36, ivenhov wrote:

> Excuse my doubel-posting and replying to my own thread.
> I found this in PBClientAdapter class in fetchIndex method
> 
>        final MapReduce mr = new IndexMapReduce(this, indexQuery);
> 
>        mr.addReducePhase(NamedErlangFunction.REDUCE_IDENTITY,
> Args.REDUCE_PHASE_ONLY_1);
>        // only return the key, to match the http rest api
>        mr.addReducePhase(new JSSourceFunction("function(v) { return
> v.map(function(e) { return e[1]; }); }"), Args.REDUCE_PHASE_ONLY_1);
> 
> This means it uses map reduce and JavaScript for protocol buffers.
> Would that be the reason of slow performance with thousands of keys?
> In my case it more performant to list all the keys than do 2i query.

You can measure if this *is* the problem if you are using Riak 1.1.*+ as an index MapReduce no longer requires a reduce identity call in Riak 1.1 onwards. Using the java client you can create an Index MapReduce without the reduce phases:

    IndexQuery = new BinIndexQuery(BinIndex.named("idx_key"), BUCKET_NAME, filter.toString());
    client.mapReduce(indexQuery).execute();

Your result will be a list of bucket/key pairs (different from the HTTP API's list of keys.) If that executes faster the problem is with the reduce phases, you could even try with just the first reduce phase, and then try with the second to discover if the JS phase is the issue.

The reason for the extra JS reduce phase was a (perhaps poor?) choice I made. In order to provide parity between the HTTP and PB API I made the choice to have the PB call return a list of Keys only (like the HTTP API.) I guess I could have iterated on the client to reformat the result, or had Jackson do it, or even just returned different output between the APIs…it is a hard choice and really the problem is that Riak has a special API for HTTP returning results in one format, and requires a MapReduce to get at indexes over protocol buffers.

Please let me know if executing an index MapReduce without reduce phases performs more in line with your expectations.

Cheers

Russell

> 
> I'm using java client 1.0.4 and riak cluster 1.1.1
> Any feedback appreciated.
> 
> Daniel
> 
> --
> View this message in context: http://riak-users.197444.n3.nabble.com/Slow-performance-on-getting-via-2i-tp3812664p3818984.html
> Sent from the Riak Users mailing list archive at Nabble.com.
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list