<div dir="ltr"><div><div><div><div><div><div><div>Hi,<br><br></div>so I did what you suggested, decreased the n_val to 2 both on the bucket as well as on the search index props.<br></div>Created some entries again and it seems that it happens again, although this time I'm not 100% sure. <br></div>I didn't find a specific entry yet, but when I look on solr admin at the index statistics, I see a difference on the numFound on the nodes (See screenshots).<br></div>As suggested, I read through the responses and we e.g. removed the yz_anti_entropy directory and ran the `yz_entropy_mgr:init([])` function. Solr error logs we don't have, as well as no riak error logs.<br><br></div>I'm not sure if the numbers reported on the solr admin page actually should match or not, i.e. when we have a difference on the nodes there, does it mean that we have a problem with the index or not?<br><br></div><div>What I didn't do yet but probably will do now is to run a re-save of all the objects in the bucket and see what happens with the index.<br></div><div><br></div>Thanks again,<br></div><div>Michael <br></div><div><div><div><div><img alt="Inline-Bild 4" src="cid:ii_14c5b8c90b56fe5f" height="161" width="578"><img alt="Inline-Bild 3" src="cid:ii_14c5b8c8efb5ff25" height="169" width="578"><br></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-03-25 17:33 GMT+01:00 Zeeshan Lakhani <span dir="ltr"><<a href="mailto:zlakhani@basho.com" target="_blank">zlakhani@basho.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Ok, no worries.<div><br></div><div>Also, sometimes, AAE may take a little time to complete the exchanges. Definitely read through my responses in <a href="http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html" target="_blank">http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html</a> if you continue to see issues with Riak Search being out-of-sync.</div><div><br></div><div>Thanks.</div><div><span class=""><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);font-family:'Ubuntu Mono for Powerline';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);font-family:'Ubuntu Mono';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">Zeeshan Lakhani<br>programmer | </div><div style="word-wrap:break-word">software engineer at @basho | </div><div style="word-wrap:break-word">org. member/founder of @papers_we_love | <a href="http://paperswelove.org" target="_blank">paperswelove.org</a><br>twitter => @zeeshanlakhani</div></span></div></span></div></span></div></div></div></div></div></div></div></div></div></div>
</div>
<br></span><div><div class="h5"><div><blockquote type="cite"><div>On Mar 25, 2015, at 12:27 PM, Michael Weibel <<a href="mailto:michael.weibel@gmail.com" target="_blank">michael.weibel@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div><div><div><div>Hi Zeeshan,<br><br></div>Ok, will do that and report back as soon as I have it.. Might take a while though because I first also have to figure out whether I have still the same issue or not.. ;) <br></div><br></div>Thanks!<br></div>Michael<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">2015-03-25 16:56 GMT+01:00 Zeeshan Lakhani <span dir="ltr"><<a href="mailto:zlakhani@basho.com" target="_blank">zlakhani@basho.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Hey Michael,</div><div><br></div><div>Ideally, for this “testing" setup, n_val=2 would be the effective choice. I’d create a new bucket_type/bucket and re-PUT your data in and test search again to be sure.</div><div><br></div><div>Let me know. Thanks.</div><span><br><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="font-family:'Ubuntu Mono for Powerline';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="font-family:'Ubuntu Mono';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">Zeeshan Lakhani<br>programmer | </div><div style="word-wrap:break-word">software engineer at @basho | </div><div style="word-wrap:break-word">org. member/founder of @papers_we_love | <a href="http://paperswelove.org/" target="_blank">paperswelove.org</a><br>twitter => @zeeshanlakhani</div></span></div></span></div></span></div></div></div></div></div></div></div></div></div></div>
</div>
<br></span><div><blockquote type="cite"><div><div><div>On Mar 25, 2015, at 11:02 AM, Michael Weibel <<a href="mailto:michael.weibel@gmail.com" target="_blank">michael.weibel@gmail.com</a>> wrote:</div><br></div></div><div><div><div><div dir="ltr" style="font-family:UbuntuMono-RegularForPowerline;font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div>Hi Zeeshan,<br><br></div>Thanks for your answer.<div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>Just to be sure, does your custom schema include the required fields, as mentioned in the docs: <a href="http://docs.basho.com/riak/latest/dev/advanced/search-schema/#Custom-Schemas?" target="_blank">http://docs.basho.com/riak/latest/dev/advanced/search-schema/#Custom-Schemas?</a></div></div></blockquote><div><br></div><div>Yes, I double checked that now to make sure, and the schema includes the required fields.<br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div> Are these Riak nodes joined? What’s your ring size, n_val value?</div></div></blockquote><div><br></div><div>They run in a cluster, yes. Output of "riak-admin status":<br><br>ring_creation_size : 64<br>ring_members : ['riak@IPADDRESS','riak@IPADDRESS']<br>ring_num_partitions : 64<br>ring_ownership : <<"[{'riak@IPADDRESS',32},{'riak@IPADDRESS',32}]">><br>rings_reconciled : 0<br>rings_reconciled_total : 33<br><br></div><div>n_val is 3, the initial one. According to the docs we should probably either add another node or reduce it to "2" though..correct? (Sorry, newbie in riak here :D)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>How are you querying the Solr nodes to know which node has the data and which one doesn't? Coverage is R=1, so you would be getting a different number on some search queries (using the standard /search/query/<index_name>?...) if its only on one of the Solr cores.</div></div></blockquote><div><br></div><div>Yes exactly. That's how I figured out that there's a difference, later on I queried the separate Solr instances using the solr admin interface itself. I also then fetched the missing key on both the riak nodes (without going to solr, just fetching it directly using the riak HTTP API) and they exist on both nodes.<span> </span><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>Can you also post me a screenshot of your search AAE exchanges, e.g. `riak-admin search aae-status`? You could look at this thread, <a href="http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html" target="_blank">http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html</a>, for answers on how to perform read-repair/repair the AAE tree.</div></div></blockquote><div><br></div><div>aae-status is in the .log file attached.<br><br></div><div>So you'd propose to perform a read-repair on the AAE tree?<br><br></div><div>Best,<br></div><div>Michael<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Thanks.</div><div><br><div><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="font-family:'Ubuntu Mono for Powerline';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="font-family:'Ubuntu Mono';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;border-spacing:0px"><div style="word-wrap:break-word">Zeeshan Lakhani<br>programmer | </div><div style="word-wrap:break-word">software engineer at @basho | </div><div style="word-wrap:break-word">org. member/founder of @papers_we_love |<span> </span><a href="http://paperswelove.org/" target="_blank">paperswelove.org</a><br>twitter => @zeeshanlakhani</div></span></div></span></div></span></div></div></div></div></div></div></div></div></div></div></div><br><div><blockquote type="cite"><div><div><div>On Mar 25, 2015, at 6:42 AM, Michael Weibel <<a href="mailto:michael.weibel@gmail.com" target="_blank">michael.weibel@gmail.com</a>> wrote:</div><br></div></div><div><div><div><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Hi all,<br><br></div>I have on a test environment two riak nodes and each of them has solr activated which index 3 buckets using a custom schema.<br></div>After testing a bit back and forth, I have the case that on one solr node, an entry is not in the index (I know in which node though).<span> </span><br></div>Fetching the specific key in the bucket works however, both nodes have the respective entry.<br><br></div>1) How can this happen? I don't see any error/warning in the logs (neither solr nor riak logs).<span> </span><br></div>2) Is there a possibility to fix this without having to do e.g. a PUT on the specific key with the same content in order to update it?<br><br></div>I tried to run a repair on the failing node using the guide:<span> </span><a href="http://docs.basho.com/riak/1.4.7/ops/running/recovery/repairing-indexes/#Repairing-Search-Indexes" target="_blank">http://docs.basho.com/riak/1.4.7/ops/running/recovery/repairing-indexes/#Repairing-Search-Indexes</a><br></div>When running the repair command on the partitions I received, it gave me however a [{<PartitionId>, down}, {...}] response, which gives me an uncomfortable feeling, but I didn't really figure out yet what this means exactly.<span> </span><br><br></div>Thanks a lot for your help :)<br><br></div>Best,<br></div>Michael<br></div></div></div>_______________________________________________<br>riak-users mailing list<br><a href="mailto:riak-users@lists.basho.com" target="_blank">riak-users@lists.basho.com</a><br><a href="http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com" target="_blank">http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com</a><br></div></blockquote></div><br></div></div></blockquote></div><br></div></div></div></div><span><riak-admin-aae-status.log></span></div></blockquote></div><br></div></blockquote></div><br></div>
</div></blockquote></div><br></div></div></div></div></blockquote></div><br></div>