Riak2.0 with Solr Search: index on one node contains not all entries

Michael Weibel michael.weibel at gmail.com
Fri Mar 27 10:04:48 EDT 2015


Hi,

so I did what you suggested, decreased the n_val to 2 both on the bucket as
well as on the search index props.
Created some entries again and it seems that it happens again, although
this time I'm not 100% sure.
I didn't find a specific entry yet, but when I look on solr admin at the
index statistics, I see a difference on the numFound on the nodes (See
screenshots).
As suggested, I read through the responses and we e.g. removed the
yz_anti_entropy directory and ran the `yz_entropy_mgr:init([])` function.
Solr error logs we don't have, as well as no riak error logs.

I'm not sure if the numbers reported on the solr admin page actually should
match or not, i.e. when we have a difference on the nodes there, does it
mean that we have a problem with the index or not?

What I didn't do yet but probably will do now is to run a re-save of all
the objects in the bucket and see what happens with the index.

Thanks again,
Michael
[image: Inline-Bild 4][image: Inline-Bild 3]

2015-03-25 17:33 GMT+01:00 Zeeshan Lakhani <zlakhani at basho.com>:

> Ok, no worries.
>
> Also, sometimes, AAE may take a little time to complete the exchanges.
> Definitely read through my responses in
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html if
> you continue to see issues with Riak Search being out-of-sync.
>
> Thanks.
>
> Zeeshan Lakhani
> programmer |
> software engineer at @basho |
> org. member/founder of @papers_we_love | paperswelove.org
> twitter => @zeeshanlakhani
>
> On Mar 25, 2015, at 12:27 PM, Michael Weibel <michael.weibel at gmail.com>
> wrote:
>
> Hi Zeeshan,
>
> Ok, will do that and report back as soon as I have it.. Might take a while
> though because I first also have to figure out whether I have still the
> same issue or not.. ;)
>
> Thanks!
> Michael
>
>
> 2015-03-25 16:56 GMT+01:00 Zeeshan Lakhani <zlakhani at basho.com>:
>
>> Hey Michael,
>>
>> Ideally, for this “testing" setup, n_val=2 would be the effective choice.
>> I’d create a new bucket_type/bucket and re-PUT your data in and test search
>> again to be sure.
>>
>> Let me know. Thanks.
>>
>> Zeeshan Lakhani
>> programmer |
>> software engineer at @basho |
>> org. member/founder of @papers_we_love | paperswelove.org
>> twitter => @zeeshanlakhani
>>
>> On Mar 25, 2015, at 11:02 AM, Michael Weibel <michael.weibel at gmail.com>
>> wrote:
>>
>> Hi Zeeshan,
>>
>> Thanks for your answer.
>>
>>> Just to be sure, does your custom schema include the required fields, as
>>> mentioned in the docs:
>>> http://docs.basho.com/riak/latest/dev/advanced/search-schema/#Custom-Schemas?
>>>
>>
>> Yes, I double checked that now to make sure, and the schema includes the
>> required fields.
>>
>>
>>>  Are these Riak nodes joined? What’s your ring size, n_val value?
>>>
>>
>> They run in a cluster, yes. Output of "riak-admin status":
>>
>> ring_creation_size : 64
>> ring_members : ['riak at IPADDRESS','riak at IPADDRESS']
>> ring_num_partitions : 64
>> ring_ownership : <<"[{'riak at IPADDRESS',32},{'riak at IPADDRESS',32}]">>
>> rings_reconciled : 0
>> rings_reconciled_total : 33
>>
>> n_val is 3, the initial one. According to the docs we should probably
>> either add another node or reduce it to "2" though..correct? (Sorry, newbie
>> in riak here :D)
>>
>>
>>> How are you querying the Solr nodes to know which node has the data and
>>> which one doesn't? Coverage is R=1, so you would be getting a different
>>> number on some search queries (using the standard
>>> /search/query/<index_name>?...) if its only on one of the Solr cores.
>>>
>>
>> Yes exactly. That's how I figured out that there's a difference, later on
>> I queried the separate Solr instances using the solr admin interface
>> itself. I also then fetched the missing key on both the riak nodes (without
>> going to solr, just fetching it directly using the riak HTTP API) and they
>> exist on both nodes.
>>
>>
>>> Can you also post me a screenshot of your search AAE exchanges, e.g.
>>> `riak-admin search aae-status`? You could look at this thread,
>>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html,
>>> for answers on how to perform read-repair/repair the AAE tree.
>>>
>>
>> aae-status is in the .log file attached.
>>
>> So you'd propose to perform a read-repair on the AAE tree?
>>
>> Best,
>> Michael
>>
>>
>>
>>>
>>> Thanks.
>>>
>>> Zeeshan Lakhani
>>> programmer |
>>> software engineer at @basho |
>>> org. member/founder of @papers_we_love | paperswelove.org
>>> twitter => @zeeshanlakhani
>>>
>>> On Mar 25, 2015, at 6:42 AM, Michael Weibel <michael.weibel at gmail.com>
>>> wrote:
>>>
>>> Hi all,
>>>
>>> I have on a test environment two riak nodes and each of them has solr
>>> activated which index 3 buckets using a custom schema.
>>> After testing a bit back and forth, I have the case that on one solr
>>> node, an entry is not in the index (I know in which node though).
>>> Fetching the specific key in the bucket works however, both nodes have
>>> the respective entry.
>>>
>>> 1) How can this happen? I don't see any error/warning in the logs
>>> (neither solr nor riak logs).
>>> 2) Is there a possibility to fix this without having to do e.g. a PUT on
>>> the specific key with the same content in order to update it?
>>>
>>> I tried to run a repair on the failing node using the guide:
>>> http://docs.basho.com/riak/1.4.7/ops/running/recovery/repairing-indexes/#Repairing-Search-Indexes
>>> When running the repair command on the partitions I received, it gave me
>>> however a [{<PartitionId>, down}, {...}] response, which gives me an
>>> uncomfortable feeling, but I didn't really figure out yet what this means
>>> exactly.
>>>
>>> Thanks a lot for your help :)
>>>
>>> Best,
>>> Michael
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>>
>> <riak-admin-aae-status.log>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150327/220d217b/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node01.png
Type: image/png
Size: 96238 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150327/220d217b/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: node02.png
Type: image/png
Size: 41981 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150327/220d217b/attachment-0001.png>


More information about the riak-users mailing list