Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime

Vitaliy E 13vitamins at gmail.com
Wed Apr 1 06:48:22 EDT 2015


Hi Michael,

It can't be the same in my case because we run with n_val = 3 on 5 nodes,
that is not all keys exist on each node. And I don't know how to check Sorl
cores of all nodes for consistency, sorry. What I did was to write a simple
client that ran the same search request multiple times and counted
occurrences of each key in the results. If the number of executions is big
enough, a key missing from ~33% of results means it's not indexed on one
node (out of three), ~66% - not indexed on two nodes. But again, it was a
dev environment with just a few keys, so I could run this safely. BTW the
indexes eventually converged, within 24 hours I think.

Regards,
Vitaliy

On Wed, Apr 1, 2015 at 12:38 PM, Michael Weibel <michael.weibel at gmail.com>
wrote:

> Hey Vitaly,
>
> experienced the same issue recently (see
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016972.html).
>
> When you have a look at your now working solr instances admin pages, do
> you have everywhere the index counts the same? (i.e. the numDocs, maxDocs,
> deletedDocs)
>
> Thanks,
> Michael
>
> 2015-04-01 9:50 GMT+02:00 Vitaliy E <13vitamins at gmail.com>:
>
>> Hello everyone,
>>
>> I've just joined the list, and am a bit late to the party. Sorry about
>> that. Thought I would contribute an answer anyway.
>>
>> Santi, what is you n_val?
>>
>> I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
>> two cases:
>>
>> 1. One of the nodes was not part of the cluster although the cluster was
>> thinking it was. Don't ask me how that happened. Obviously, when a request
>> hit that node part of the entries could not be found there.
>>
>> 2. Look for indexing errors in Solr console and Riak logs. Each Riak node
>> has its own Solr "repository", so if an entry fails to be indexed on any of
>> them, search results will be inconsistent depending on which set of nodes
>> returns it. Let's say you have replicas on nodes A, B, and C. Entry X
>> failed to be indexed on A, entry Y failed to be indexed on A and B, and
>> entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
>> {Z} as your search results.
>>
>> In our case the indexing failures were caused by disk/filesystem errors.
>>
>> Regards,
>> Vitaly
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150401/bb714a7b/attachment-0002.html>


More information about the riak-users mailing list