Riak Map / Reduce / Map capacity limit reached at 20000 keys / Doubt about reduce step

claudef at br.ibm.com claudef at br.ibm.com
Tue May 22 12:09:59 EDT 2012


Dear Matthew, 

The capacity adjustment at the Java script engines worked fine. I was able 
to run a clean-up on the database. I also decided o double the JS memory 
space to 16 MB, as RAM is not the issue on our server,  There is still a 
doubt about the Map & Reduce syntax which is not giving a matching result: 
 

My simple JavaScript, pattern scan & count works OK and delivers a counter 
for a pattern search : 

query.map("""function(v) { var str = '"""+search_string+"""'; var data = 
JSON.parse(v.values[0].data); if 
(v.values[0].data.toString().toUpperCase().search(str) != -1) { return 
[[v.key, data]]; } return []; }""")
query.reduce("function(v) { var counteri = 0;  for(var i in v) {counteri 
+= 1;} return [['counter_found', counteri]]; } ") 
 
The map function as you sent below returns a list of  keys and 1 like this 
 

query.map("""function(v) {if 
(v.values[0].data.search('"""+search_string+"""') != -1) { return [[v.key, 
1]]; } return []; }""")

How many keys in the bucket: 1000
agFa82470070 - 1
sDko48992336 - 1
eFxO66269553 - 1
Sovt26572593 - 1
.. 

This looks good. By using the JavaScript reduce function I get a zero 
result 

Source:  query.reduce("function(v) { return 
[v.reduce(function(acc,value){return acc + value;},0)]}")

Output:   0 - 0 

Thanks in advance for clarification. 

Regards, 
Claude

Claude Falbriard 
Certified IT Specialist L2 - Middleware
AMS Hortolândia / SP - Brazil
phone:    +55 19 9837 0789
cell:         +55 13 8117 3316
e-mail:    claudef at br.ibm.com
blog:        https://w3.tap.ibm.com/weblogs/inovarMF/

Project CallOwn Blue 



From:   Matthew Tovbin <matthew at tovbin.com>
To:     claudef at br.ibm.com
Cc:     riak-users at lists.basho.com
Date:   21/05/2012 17:01
Subject:        Re: Riak Map / Reduce / Map capacity limit reached at 
20000 keys
Sent by:        tovbinm at gmail.com



Claude,

1.
try to increase js_vm_counts as follows:

{riak_kv, [
             ...
            {map_js_vm_count, 24 },
            {reduce_js_vm_count, 18 },
            ...]

2. optimize your js function:
query.map("""function(v) {if 
(v.values[0].data.search('"""+search_string+"""') != -1) { return [[v.key, 
1]]; } return []; }""")
query.reduce("function(v) { return [v.reduce(function(acc,value){return 
acc + value;},0]; } ") 


-Matthew



On Mon, May 21, 2012 at 10:02 AM, <claudef at br.ibm.com> wrote:
Dear colleagues, 

 During my pilot testing I've got an error message during a text pattern 
search operation coded as a Riak Client "map" logic. This error appeared 
at a volume of approximately 20.000 keys, each with an 1 KB free text data 
block associated. The Map function was coded in JavaScript. Here the 
source: 

query.map("""function(v) { var str = '"""+search_string+"""'; var data = 
JSON.parse(v.values[0].data); if (v.values[0].data.toString().search(str) 
!= -1) { return [[v.key, data]]; } return []; }""") 
query.reduce("function(v) { var counteri = 0;  for(var i in v) {counteri 
+= 1;} return [['counter_found', counteri]]; } ") 

The error shown is the following : [preflist_exhausted] 

zbra:/opt/python/myprojects/riak # python pilot3.py 
starts at : 2012-05-21 07:35:49.070470 
How many keys in the bucket: 19760 
Riak client exception: Error running MapReduce operation. Status: 500 : 
{"phase":0,"error":"[preflist_exhausted]","input":"{ok,{r_object,<<\"ramtest\">>,<<\"XDHt25887344\">>,[{r_content,{dict,6,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[[<<\"Links\">>]],[],[],[],[],[],[],[],[[<<\"content-type\">>,97,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110],[<<\"X-Riak-VTag\">>,53,86,72,48,120,77,53,99,114,86,111,77,77,54,109,114,79,70,54,52,49,109]],[[<<\"index\">>]],[],[[<<\"X-Riak-Last-Modified\">>|{1337,595548,374416}]],[],[[<<\"X-Riak-Meta\">>]]}}},<<\"{\"is_valid\": 
true, \"cEDO\": 
\"GQZumfdWBCfX...\">>}],...},...}","type":"forward_preflist","stack":"[]"} 


Any suggestions how to improve the pattern scan capability in the in the 
RiakClient  map function, or how enhance the JavaScript code for better 
capacity? 
Are there any servers setup tuning options to adjust? 

Thanks in advance for your support. 

Regards, 
Claude

Claude Falbriard 
Certified IT Specialist L2 - Middleware
AMS Hortolândia / SP - Brazil
phone:    +55 19 9837 0789
cell:         +55 13 8117 3316
e-mail:    claudef at br.ibm.com

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120522/b8ddcf3e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3383 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120522/b8ddcf3e/attachment.jpe>


More information about the riak-users mailing list