Riak Map / Reduce / Map capacity limit reached at 20000 keys / Doubt about reduce step

Matthew Tovbin matthew at tovbin.com
Tue May 22 18:45:05 EDT 2012


Claude,

This should work:

 function(v) {return [v.reduce(function(total, value) {if
(isNaN(parseInt(value))) { return total + 1;} else{return total + value;}},
0)];}


(It's an example from http://riakhandbook.com/)


-Matthew



On Tue, May 22, 2012 at 9:09 AM, <claudef at br.ibm.com> wrote:

> Dear Matthew,
>
> The capacity adjustment at the Java script engines worked fine. I was able
> to run a clean-up on the database. I also decided o double the JS memory
> space to 16 MB, as RAM is not the issue on our server,  There is still a
> doubt about the Map & Reduce syntax which is not giving a matching result:
>
> My simple JavaScript, pattern scan & count works OK and delivers a counter
> for a pattern search :
>
> query.map("""function(v) { var str = '"""+search_string+"""'; var data =
> JSON.parse(v.values[0].data); if
> (v.values[0].data.toString().toUpperCase().search(str) != -1) { return
> [[v.key, data]]; } return []; }""")
> query.reduce("function(v) { var counteri = 0;  for(var i in v) {counteri
> += 1;} return [['counter_found', counteri]]; } ")
>
> The map function as you sent below returns a list of  keys and 1 like this
>
>
> query.map("""function(v) {if
> (v.values[0].data.search('"""+search_string+"""') != -1) { return [[v.key,
> 1]]; } return []; }""")
>
> How many keys in the bucket: 1000
> agFa82470070 - 1
> sDko48992336 - 1
> eFxO66269553 - 1
> Sovt26572593 - 1
> ..
>
> This looks good. By using the JavaScript *reduce* function I get a zero
> result
>
> Source:  query.reduce("function(v) { return
> [v.reduce(function(acc,value){return acc + value;},0)]}")
>
> Output:   0 - 0
>
> Thanks in advance for clarification.
>
> Regards,
> Claude
> *
> Claude Falbriard
> Certified IT Specialist L2 - Middleware
> AMS Hortolândia / SP - Brazil
> phone:    +55 19 9837 0789
> cell:         +55 13 8117 3316
> e-mail:    claudef at br.ibm.com
> blog:        **https://w3.tap.ibm.com/weblogs/inovarMF/*<https://w3.tap.ibm.com/weblogs/inovarMF/>
> *
> **
> Project CallOwn Blue      *
>
>
>
> From:        Matthew Tovbin <matthew at tovbin.com>
> To:        claudef at br.ibm.com
> Cc:        riak-users at lists.basho.com
> Date:        21/05/2012 17:01
> Subject:        Re: Riak Map / Reduce / Map capacity limit reached at
> 20000 keys
> Sent by:        tovbinm at gmail.com
> ------------------------------
>
>
>
> Claude,
>
> 1.
> try to increase js_vm_counts as follows:
>
> {riak_kv, [
>              ...
>             {map_js_vm_count, 24 },
>             {reduce_js_vm_count, 18 },
>             ...]
>
> 2. optimize your js function:
> query.map("""function(v) {if
> (v.values[0].data.search('"""+search_string+"""') != -1) { return [[v.key,
> 1]]; } return []; }""")
> query.reduce("function(v) { return [v.reduce(function(acc,value){return
> acc + value;},0]; } ")
>
>
> -Matthew
>
>
>
> On Mon, May 21, 2012 at 10:02 AM, <*claudef at br.ibm.com*<claudef at br.ibm.com>>
> wrote:
> Dear colleagues,
>
>  During my pilot testing I've got an error message during a text pattern
> search operation coded as a Riak Client "map" logic. This error appeared at
> a volume of approximately 20.000 keys, each with an 1 KB free text data
> block associated. The Map function was coded in JavaScript. Here the
> source:
>
> query.map("""function(v) { var str = '"""+search_string+"""'; var data =
> JSON.parse(v.values[0].data); if (v.values[0].data.toString().search(str)
> != -1) { return [[v.key, data]]; } return []; }""")
> query.reduce("function(v) { var counteri = 0;  for(var i in v) {counteri
> += 1;} return [['counter_found', counteri]]; } ")
>
> The error shown is the following : *[preflist_exhausted]*
>
> zbra:/opt/python/myprojects/riak # python pilot3.py
> starts at : 2012-05-21 07:35:49.070470
> How many keys in the bucket: 19760
> Riak client exception: Error running MapReduce operation. Status: 500 :
> {"phase":0,"error":"[preflist_exhausted]","input":"{ok,{r_object,<<\"ramtest\">>,<<\"XDHt25887344\">>,[{r_content,{dict,6,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[[<<\"Links\">>]],[],[],[],[],[],[],[],[[<<\"content-type\">>,97,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110],[<<\"X-Riak-VTag\">>,53,86,72,48,120,77,53,99,114,86,111,77,77,54,109,114,79,70,54,52,49,109]],[[<<\"index\">>]],[],[[<<\"X-Riak-Last-Modified\">>|{1337,595548,374416}]],[],[[<<\"X-Riak-Meta\">>]]}}},<<\"{\"is_valid\":
> true, \"cEDO\":
> \"GQZumfdWBCfX...\">>}],...},...}","type":"forward_preflist","stack":"[]"}
>
> Any suggestions how to improve the pattern scan capability in the in the
> RiakClient  map function, or how enhance the JavaScript code for better
> capacity?
> Are there any servers setup tuning options to adjust?
>
> Thanks in advance for your support.
>
> Regards,
> Claude*
>
> Claude Falbriard
> Certified IT Specialist L2 - Middleware
> AMS Hortolândia / SP - Brazil
> phone:    **+55 19 9837 0789* <%2B55%2019%209837%200789>*
> cell:         **+55 13 8117 3316* <%2B55%2013%208117%203316>*
> e-mail:    **claudef at br.ibm.com* <claudef at br.ibm.com>
>
> _______________________________________________
> riak-users mailing list*
> **riak-users at lists.basho.com* <riak-users at lists.basho.com>*
> **http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com*<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120522/a5de5b43/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3383 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120522/a5de5b43/attachment.jpe>


More information about the riak-users mailing list