Fwd: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime

Zeeshan Lakhani zlakhani at basho.com
Thu Mar 5 11:53:09 EST 2015


Hello Baskar, Santi.

I’d definitely like to know what your AAE status is like, as per my last email. The fix here may just be clearing the AAE trees and rebuilding them.

Yes, index creation is currently async and you have to wait for it to be created before sending data. You can poll to check for the index, similar to this function used in the search java-client tests, https://github.com/basho/riak-java-client/blob/89778ae5a2274ce8693709bcc4c5ea3558d72971/src/test/java/com/basho/riak/client/core/operations/itest/ITestBase.java#L264 <https://github.com/basho/riak-java-client/blob/89778ae5a2274ce8693709bcc4c5ea3558d72971/src/test/java/com/basho/riak/client/core/operations/itest/ITestBase.java#L264>. We do something similarly in our erlang tests for search. 

I’m not aware of a better way right now. Currently, I’m actually working on making index creation synchronous before returning a response (within a timeout), which could solve your issues.

Thanks.


> Begin forwarded message:
> 
> Date: March 5, 2015 at 11:21:59 AM EST
> Subject: Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
> From: Baskar Srinivasan <baskar at veradocs.com>
> To: Santi Kumar <santi at veradocs.com>
> Cc: Zeeshan Lakhani <zlakhani at basho.com>
> 
> Hello Zeeshan,
> 
> I work with Santi on the same server backend component. Our primary issue seems to be that Index creation and Bucket association with the indices is being done via load balancer. After we do that, we move onto actually creating data. When data is created is getting into Riak fine. However not all data make it to the indices. 
> 
> It appears that depending when a node gets the Riak data persist call it may not have completed index creation and bucket to index association for that node which in turns means that particular data doesn't get indexed anywhere.
> 
> Is it possible to ensure that the call to create indices and associate buckets with indices guarantees that this action has been completed on all nodes of a cluster?
> 
> The API we are using are as follows:
> 
> 1. Index creation
> YokozunaSchema yschema = getSchema();
> 
> YokozunaIndex vdIndex = new YokozunaIndex(indexName,yschema.getName());
> 
>         	
>         StoreIndex storeIndex =new StoreIndex.Builder(vdIndex).build();
> 
>         RiakFuture<Void, YokozunaIndex> storeIndexFuture = client.executeAsync(storeIndex);
> 
>         storeIndexFuture.await();
> 
>          if(!storeIndexFuture.isSuccess()){
> 
>         throw new Exception("....");
> 
> 
>         	}
> 
> 
> 
> 2. Bucket association
> 
> 
> 
> StoreBucketProperties storeBucketProps = new StoreBucketProperties.Builder(bucket)
> 
>         .withSearchIndex(indexName)
> 
>         .build();
> 
>         	
>         RiakFuture<Void,Namespace> storeBucketPropsFuture = client.executeAsync(storeBucketProps);
> 
>         try{
> 
>         storeBucketPropsFuture.await();
> 
>         if(storeBucketPropsFuture.isDone() && storeBucketPropsFuture.isSuccess()){
> 
>         logger.info(" Associated index "+indexName+" With bucket "+bucketName);
> 
>         }else{
> 
>         logger.warn(" Unable to associate bucket "+bucketName);
> 
>         }
> 
>         }catch(Exception e){
> 
>         logger.warn("Issues with ");
> 
> 
>     	}
> 
> 
> 
> Regards,
> 
> Baskar
> 
> 
> On Thu, Mar 5, 2015 at 8:06 AM, Santi Kumar <santi at veradocs.com <mailto:santi at veradocs.com>> wrote:
> This issue is happening for *:* query only. It doesn't have lot of objects so we are doing *:* query. Ever time it fetches different number when I switched to list all keys and do a multi get its returning all objects
> 
> Another thing what we do is we create indexes on the fly and wait for index creation by checking Listindex operation. Once it found the index then we are associating the Bucket with index. After that we write some entries into that bucket.
> 
> I fetched the key and tried searching again but it's the same again. So read repair would have happened so that wouldn't be the issue I can check AAE
> 
> Can you suggest any better way for these index creations and bucket association? I suspect that might be the reason
> Is there any way to check what's the data each shard has? I tried solr query with shards param 
> 
> Thanks 
> Santi
> 
> On Mar 5, 2015 8:05 PM, "Zeeshan Lakhani" <zlakhani at basho.com <mailto:zlakhani at basho.com>> wrote:
> Hello Santi, 
> 
> Have you deleted an object in that bucket/index at some point? 
> 
> Please make sure AAE is running by checking search’s AAE status, `riak-admin search aae-status`, and that data exists in the correct directory, `./data/yz_anti_entropy` (http://docs.basho.com/riak/latest/ops/advanced/configs/search/ <http://docs.basho.com/riak/latest/ops/advanced/configs/search/>). 
> 
> You may just need to perform a read-repair by performing a fetch of the object itself first, before performing search queries again.
> 
> Thanks.
> 
> Zeeshan Lakhani
> programmer | 
> software engineer at @basho | 
> org. member/founder of @papers_we_love | paperswelove.org <http://paperswelove.org/>
> twitter => @zeeshanlakhani
> 
>> On Mar 4, 2015, at 9:06 PM, Santi Kumar <santi at veradocs.com <mailto:santi at veradocs.com>> wrote:
>> 
>> But in our case we didnt have spaces in keys. All our keys are UUID's so I wouldn't suspect that. AAE I haven't verified .
>> 
>> On Thu, Mar 5, 2015 at 7:16 AM, John O'Brien <boardom at gmail.com <mailto:boardom at gmail.com>> wrote:
>> I'd lean towards AAE issues on Yokozuna... Same problems we were having with our 'spaces-in-keys' issue... Once we cleaned those up, things were great again.
>> 
>> On Wed, Mar 4, 2015 at 8:36 PM, Santi Kumar <santi at veradocs.com <mailto:santi at veradocs.com>> wrote:
>> Raik 2.0.0
>> 
>> On Mar 5, 2015 12:31 AM, "Christopher Meiklejohn" <cmeiklejohn at basho.com <mailto:cmeiklejohn at basho.com>> wrote:
>> 
>> > On Mar 4, 2015, at 1:15 PM, Santi Kumar <santi at veradocs.com <mailto:santi at veradocs.com>> wrote:
>> >
>> > Hi,
>> > We are running into a strange issue with Riak Search. Our setup is with 3 nodes of Riak (with search enabled) in 3 different  ec2 instances behind ELB. App server talks to the cluster through ELB. We are querying for list of objects through *:* query instead of list bucket keys and every time the result is different. We have only 4 objects in that bucket / index, only few times it's giving all 4 but ofther it returns 2 or 3.
>> >
>> > This wasn't happening when we have one instance of Riak. Any insight?
>> 
>> Hi Santi,
>> 
>> Can you provide information regarding what versions of Riak you are running?
>> 
>> - Chris
>> 
>> Christopher Meiklejohn
>> Senior Software Engineer
>> Basho Technologies, Inc.
>> cmeiklejohn at basho.com <mailto:cmeiklejohn at basho.com>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>> 
>> 
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150305/cfc03561/attachment-0002.html>


More information about the riak-users mailing list