Searching

Jason Ryan jason.ryan at trustev.com
Sat Mar 7 11:06:26 EST 2015


Just as some FYI - we've just had another crash as of 15mins ago.

[image: photo]
*Jason Ryan*
VP Engineering

Trustev
Real Time, Online Identity Verification

email: jason.ryan at trustev.com
skype: jason_j_ryan
web: www.trustev.com

Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland.

On 7 March 2015 at 15:54, Zeeshan Lakhani <zlakhani at basho.com> wrote:

> Hello Jason,
>
> I initially was thinking that you’re issues were similar to what we’ve
> found here,
> https://github.com/basho/yokozuna/issues/442#issuecomment-77233636, but
> seeing {error, retry_later} and the 500s seem to place the problems more on
> the Solr end. Just to make sure, are you seeing `yz_events` crashes in your
> logs?
>
> Can you tell me how much RAM each node has and have you adjusted/increased
> the search.sol.jmv_options max heap size at all (more info on
> issues/factors w/ Solr performance here ->
> http://docs.basho.com/riak/latest/ops/advanced/configs/search/#Solr-for-Operators
> )?
>
> What kind of issues are you getting in your solr logs? Anything related to
> bad/corrupted data (which will index into _yz_err fields when dealing with
> failed extractions) or possible sibling explosion (duplicates of the same
> object with different _yz_vtag fields)?
>
> Thanks.
>
>
> Zeeshan Lakhani
> programmer |
> software engineer at @basho |
> org. member/founder of @papers_we_love | paperswelove.org
> twitter => @zeeshanlakhani
>
> On Mar 7, 2015, at 5:43 AM, Jason Ryan <jason.ryan at trustev.com> wrote:
>
> Hi all,
>
> We're having real trouble with Riak Search.
> We are seeing an awful lot of errors, which leads to alot of logging for
> solr and the disk IO reaching 95% + which is causing lots of issues.
>
> - We consistently see errors around Riak trying to create indexes which
> already exist - only a restart of each Riak node stops this for a period of
> time - indexes are only ever created manually, not by software.
>
> - We see lots of errors around failing to index objects - details of what
> appears in the log are:
>
> 2015-03-07 10:30:26.871 [error] <0.2538.0>@yz_kv:index:215 failed to index
> object
> {{<<"Production">>,<<"Grains.Domain.Case">>},<<"455203890918dfc6fd3c7da49dd6adb00300000043a46a51">>}
> with error {"Failed to index docs",{error,retry_later}} because
> [{yz_solr,index,3,[{file,"src/yz_solr.erl"},{line,192}]},{yz_kv,index,7,[{file,"src/yz_kv.erl"},{line,267}]},{yz_kv,index,3,[{file,"src/yz_kv.erl"},{line,202}]},{riak_kv_vnode,actual_put,6,[{file,"src/riak_kv_vnode.erl"},{line,1418}]},{riak_kv_vnode,perform_put,3,[{file,"src/riak_kv_vnode.erl"},{line,1406}]},{riak_kv_vnode,do_put,7,[{file,"src/riak_kv_vnode.erl"},{line,1201}]},{riak_kv_vnode,handle_command,3,[{file,"src/riak_kv_vnode.erl"},{line,486}]},{riak_core_vnode,vnode_command,3,[{file,"src/riak_core_vnode.erl"},{line,345}]}]
>
> - We are also starting to see 500's being returned for search queries -
> the response looks like this:
>
> <html><head><title>500 Internal Server
> Error</title></head><body><h1>Internal Server Error</h1>The server
> encountered an error while processing this request:<br><pre>{error,
>
>     {throw,
>
>         {"Failed to search",
>
>          "http://localhost:8093/internal_solr/sessions/select",
>
>          {error,retry_later}},
>
>         [{yz_solr,search,3,[{file,"src/yz_solr.erl"},{line,278}]},
>
>
> {yz_wm_search,search,2,[{file,"src/yz_wm_search.erl"},{line,129}]},
>
>          {webmachine_resource,resource_call,3,
>
>              [{file,"src/webmachine_resource.erl"},{line,186}]},
>
>          {webmachine_resource,do,3,
>
>              [{file,"src/webmachine_resource.erl"},{line,142}]},
>
>          {webmachine_decision_core,resource_call,1,
>
>              [{file,"src/webmachine_decision_core.erl"},{line,48}]},
>
>          {webmachine_decision_core,decision,1,
>
>              [{file,"src/webmachine_decision_core.erl"},{line,558}]},
>
>          {webmachine_decision_core,handle_request,2,
>
>              [{file,"src/webmachine_decision_core.erl"},{line,33}]},
>
>          {webmachine_mochiweb,loop,2,
>
> * Connection #0 to host 10.0.5.6 left intact
>
>
> [{file,"src/webmachine_mochiweb.erl"},{line,74}]}]}}</pre><P><HR><ADDRESS>mochiweb+webmachine
> web server</ADDRESS></body></html>
>
>
> Anyone that could point us in the right direction of where to look and
> debug, as this is becoming a huge issue for us.
>
>
> Thanks,
>
> Jason
>
>
>
>
> This message is for the named person's use only. If you received this
> message in error, please immediately delete it and all copies and notify
> the sender. You must not, directly or indirectly, use, disclose,
> distribute, print, or copy any part of this message if you are not the
> intended recipient. Any views expressed in this message are those of the
> individual sender and not Trustev Ltd. Trustev is registered in Ireland
> No. 516425 and trades from 2100 Cork Airport Business Park, Cork, Ireland.
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>

-- 


This message is for the named person's use only. If you received this 
message in error, please immediately delete it and all copies and notify 
the sender. You must not, directly or indirectly, use, disclose, 
distribute, print, or copy any part of this message if you are not the 
intended recipient. Any views expressed in this message are those of the 
individual sender and not Trustev Ltd. Trustev is registered in Ireland No. 
516425 and trades from 2100 Cork Airport Business Park, Cork, Ireland.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150307/cb752291/attachment-0002.html>


More information about the riak-users mailing list