<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hey Santi, Baskar,<div class=""><br class=""></div><div class="">Are you noticing increased CPU load as you create more and more indexes? Running `riak-admin top -interval 2` a few times may bring sometime to light.</div><div class=""><br class=""></div><div class="">I’d see how you could increase resources or think more critically on how you’re indexing data for Solr. Does the data share most fields? Can you reuse indexes for some of the data and filter certain queries?</div><div class=""><br class=""></div><div class="">You may also wanted to look at this thread, <a href="https://groups.google.com/forum/#!topic/nosql-databases/9ECQpVS0QjE" class="">https://groups.google.com/forum/#!topic/nosql-databases/9ECQpVS0QjE</a>, which discusses modeling Riak Search data and the issues you’ll have with the overhead with gossiping so much metadata and the what Solr can handle.</div><div class=""><br class=""><div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: 'Ubuntu Mono for Powerline'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: 'Ubuntu Mono'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Zeeshan Lakhani<br class="">programmer | </div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">software engineer at @basho | </div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">org. member/founder of @papers_we_love | <a href="http://paperswelove.org" class="">paperswelove.org</a><br class="">twitter => @zeeshanlakhani</div></span></div></span></div></span></div></div></div></div></div></div></div></div></div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Mar 9, 2015, at 8:25 AM, Santi Kumar <<a href="mailto:santi@veradocs.com" class="">santi@veradocs.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Zeeshan,<div class=""><br class=""></div><div class="">We have typically seen this issue when we have lots of indexes created in that instance. On a t2.medium machine we already have around 512+ indexes created in data folder. In such case, if we trying to create any new indexes it's taking time. Association of Index to Bucket is failing even after  the FetchIndex operation returning sucess as shown in the below code.</div><div class=""><br class=""></div><div class="">is there any limitation of the number of Indexes? Any thing related to FileSystem handlers causing this issue?</div><div class=""><br class=""></div><div class=""><p class=""><span class="">while</span><span class="">(!</span>isCreated<span class="">){</span></p><p class=""><span class=""> </span>    <span class="">      </span><span class="">    </span>FetchIndex <span class="">fetchIndex</span> = <span class="">new</span> FetchIndex.Builder(<span class="">indexName</span>).build();</p><p class=""><span class="">        </span>    <span class="">      </span><span class="">    </span>RiakFuture<com.basho.riak.client.core.operations.YzFetchIndexOperation.Response, String> <span class="">fetchIndexFuture</span> = <span class="">client</span>.executeAsync(<span class="">fetchIndex</span>);</p><p class=""><span class="">        </span>    <span class="">      </span><span class="">    </span><span class="">try</span>{</p><p class=""><span class="">      </span><span class="">    </span>    <span class="">      </span><span class="">    </span><span class="">fetchIndexFuture</span>.await();</p><p class=""><span class=""> </span><span class="">    </span>    <span class="">      </span><span class="">    </span>com.basho.riak.client.core.operations.YzFetchIndexOperation.Response <span class="">response</span> = <span class="">fetchIndexFuture</span>.get();</p><p class=""><span class="">       </span><span class="">    </span>    <span class="">      </span><span class="">    </span>List<YokozunaIndex> <span class="">indexes</span> = <span class="">response</span>.getIndexes();</p><p class=""><span class="">    </span><span class="">    </span>    <span class="">      </span><span class="">    </span><span class="">for</span>(YokozunaIndex <span class="">index</span>:<span class="">indexes</span>){</p><p class=""><span class=""> </span><span class="">    </span>    <span class="">      </span><span class="">    </span><span class="">    </span><span class="">if</span>(<span class="">indexName</span>.equals(<span class="">index</span>.getName())){</p><p class=""><span class="">    </span><span class="">    </span>    <span class="">      </span><span class="">    </span><span class="">    </span><span class="">    </span><span class="">isCreated</span>=<span class="">true</span>;</p><p class=""><span class="">       </span><span class="">    </span>    <span class="">      </span><span class="">    </span><span class="">    </span><span class="">    </span><span class="">logger</span>.info(<span class="">"Index "</span>+<span class="">indexName</span>+<span class="">" created "</span>);</p><p class=""><span class="">      </span><span class="">    </span>    <span class="">      </span><span class="">    </span><span class="">    </span><span class="">    </span><span class="">continue</span>;</p><p class=""><span class=""> </span><span class="">    </span>    <span class="">      </span><span class="">    </span><span class="">    </span>}</p><p class=""><span class="">     </span><span class="">    </span>    <span class="">      </span><span class="">    </span>}</p><p class=""><span class="">     </span>    <span class="">      </span><span class="">    </span>}<span class="">catch</span>(Exception <span class="">e</span>){</p><p class=""><span class=""><span class="">   </span>    <span class="">      </span><span class="">    </span><span class="">    </span></span><span class="">logger</span><span class="">.warn(</span>"Unable to get "<span class="">+</span><span class="">indexName</span><span class="">+</span>" Still trying"<span class="">);</span></p><p class=""><span class="">     </span>    <span class="">      </span><span class="">    </span><span class="">    </span><span class="">isCreated</span>=<span class="">false</span>;</p><p class=""><span class="">      </span>    <span class="">      </span><span class="">    </span>}</p><p class="">    <span class="">       </span><span class="">    </span>}</p></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Mar 6, 2015 at 2:11 AM, Zeeshan Lakhani <span dir="ltr" class=""><<a href="mailto:zlakhani@basho.com" target="_blank" class="">zlakhani@basho.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Hello Baskar, Santi,</div><div class=""><br class=""></div><div class="">2-15 minutes is a long while, and we’ve not seen index creation/propagation be so slow. I’d definitely take a closer look at how you’re creating these indexes dynamically on the fly, as index creation is typically a more straightforward admin task.</div><div class=""><br class=""></div><div class="">We’ve added defaults to solrconfig.xml to handle most typical use-cases. You can read more about solrconfig.xml at <a href="http://wiki.apache.org/solr/SolrConfigXml#mainIndex_Section" target="_blank" class="">http://wiki.apache.org/solr/SolrConfigXml#mainIndex_Section</a>. You may want to take another look and optimize/improve your schema design to prevent such issues. You can read more about Solr’s performance factors here -> <a href="http://wiki.apache.org/solr/SolrPerformanceFactors" target="_blank" class="">http://wiki.apache.org/solr/SolrPerformanceFactors</a>. </div><span class=""><div class=""><br class=""></div><div class="">Thanks.</div><div class=""><br class=""></div><br class=""><div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="font-family: 'Ubuntu Mono for Powerline'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="font-family: 'Ubuntu Mono'; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><span style="border-collapse:separate;border-spacing:0px" class=""><div style="word-wrap:break-word" class=""><span style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; border-spacing: 0px;" class=""><div style="word-wrap:break-word" class=""><span style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; border-spacing: 0px;" class=""><div style="word-wrap:break-word" class="">Zeeshan Lakhani<br class="">programmer | </div><div style="word-wrap:break-word" class="">software engineer at @basho | </div><div style="word-wrap:break-word" class="">org. member/founder of @papers_we_love | <a href="http://paperswelove.org/" target="_blank" class="">paperswelove.org</a><br class="">twitter => @zeeshanlakhani</div></span></div></span></div></span></div></div></div></div></div></div></div></div></div></div>
</div>
<br class=""></span><div class=""><div class="h5"><div class=""><blockquote type="cite" class=""><div class="">On Mar 5, 2015, at 3:00 PM, Baskar Srinivasan <<a href="mailto:baskar@veradocs.com" target="_blank" class="">baskar@veradocs.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Hello Zeeshan,<div class=""><br class=""></div><div class="">Thanks for the pointer regarding waiting for index creation in each node in the cluster.</div><div class=""><br class=""></div><div class="">Presently, when the indices get created on one node, it takes a full 2-15 minutes for it to get created on other nodes in the cluster. Following are the timestamps on 3 nodes for a single index:</div><div class=""><br class=""></div><div class="">#Create index request from our server via load balancer</div><div class=""><p class="">11:16:52.999 [http-bio-8080-exec-3] INFO  c.v.s.u.RiakClientUtil - Created index for bsr-test-fromlocal-1-Access_index</p></div><div class="">#1st node, immediate creation (12 secs) once call is issued from our server</div><div class=""><p class="">2015-03-05 19:17:04.135 [info] <0.17388.104>@yz_index:local_create:189 Created index bsr-test-fromlocal-1-Access_index with schema</p><p class="">#2nd node, takes another 4 minutes for creation request to propagate</p><div class="">







<br class=""></div><p class="">2015-03-05 19:21:17.879 [info] <0.20606.449>@yz_index:local_create:189 Created index bsr-test-fromlocal-1-Access_index</p><p class="">#3rd node, takes 15 minutes for creation request to propagate</p><div class="">







<br class=""></div><p class="">2015-03-05 19:32:32.172 [info] <0.14715.94>@yz_index:local_create:189 Created index bsr-test-fromlocal-1-Access_index</p><p class="">Is there a solr config we can tune to make the 2nd and 3rd node propagation more immediate in the order of < 60 seconds?</p><p class="">Thanks,</p><p class="">Baskar</p></div>







</div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Mar 5, 2015 at 9:11 AM, Zeeshan Lakhani <span dir="ltr" class=""><<a href="mailto:zlakhani@basho.com" target="_blank" class="">zlakhani@basho.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class=""><div class=""><div style="word-wrap:break-word" class=""><span style="font-family:UbuntuMono-RegularForPowerline" class="">Hello Santi, Baskar. Please keep your messages on the user group mailing list, btw. Thanks.</span><div style="font-family:UbuntuMono-RegularForPowerline" class=""><br class=""></div><div style="font-family:UbuntuMono-RegularForPowerline" class="">Here’s an example of our testing harness’s wait_for_index function, <a href="https://github.com/basho/yokozuna/blob/develop/riak_test/yz_rt.erl#L420" target="_blank" class="">https://github.com/basho/yokozuna/blob/develop/riak_test/yz_rt.erl#L420</a>. We check for the index on each of the nodes, which is an approach you can take. </div><div style="font-family:UbuntuMono-RegularForPowerline" class=""><br class=""></div><div style="font-family:UbuntuMono-RegularForPowerline" class="">And, as I mentioned, I’m currently working on making Index creation synchronous to make this easier.</div><div style="font-family:UbuntuMono-RegularForPowerline" class=""><br class=""></div><div style="font-family:UbuntuMono-RegularForPowerline" class="">If your logs are not pointing to any errors and being that your bucket, index contains so few objects, I’d delete or mv the search-root/index directory (<span style="font-family:Verdana,Geneva,Helvetica,Arial,sans-serif;font-size:13.4399995803833px;background-color:rgb(255,255,255)" class="">./data/yz/<<index_name>>)</span> and let AAE resync the data, which should then give you consistent results.</div><div style="font-family:UbuntuMono-RegularForPowerline" class=""><br class=""></div><span style="font-family:UbuntuMono-RegularForPowerline" class="">Thanks.</span><br class=""></div></div></div></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></body></html>