Periodically solr down issue.

Luke Bakken lbakken at basho.com
Sat Feb 4 10:30:54 EST 2017


Hi Alex -

What is in solr.log ?
--
Luke Bakken
Engineer
lbakken at basho.com


On Sat, Feb 4, 2017 at 3:37 AM, Alex Feng <sweden.feng at gmail.com> wrote:
> Hello Riak users,
>
> I recently found our solr system got down after running some time, it got
> recovered after a restart. But then it happened again.
>
> Below is the error output, do you guys have any clue about this ?
>
>
> 2017-02-04 18:12:22.329 [error] <0.830.0>@yz_kv:index_internal:237 failed to
> index object
> {{<<"process_test_result">>,<<83,104,105,76,111,110,103,90,105,9
> 5,77,66,80,65,67,75,45,231,148,159,228,186,167,232,189,166,233,151,180,50,48,49,55,50,52>>},<<"9H5AlII0OZm8xwnGbI">>}
> with error {"Failed to index docs",o
> ther,{error,{conn_failed,{error,econnrefused}}}} because
> [{yz_solr,index,3,[{file,"src/yz_solr.erl"},{line,205}]},{yz_kv,index,7,[{file,"src/yz_kv.erl"},{
> line,293}]},{yz_kv,index_internal,5,[{file,"src/yz_kv.erl"},{line,224}]},{riak_kv_vnode,actual_put,6,[{file,"src/riak_kv_vnode.erl"},{line,1619}]},{riak_k
> v_vnode,perform_put,3,[{file,"src/riak_kv_vnode.erl"},{line,1607}]},{riak_kv_vnode,do_put,7,[{file,"src/riak_kv_vnode.erl"},{line,1398}]},{riak_kv_vnode,h
> andle_command,3,[{file,"src/riak_kv_vnode.erl"},{line,558}]},{riak_core_vnode,vnode_command,3,[{file,"src/riak_core_vnode.erl"},{line,346}]}]
>
>
>
> restart node, got this>
>
>
> 2017-02-04 18:13:47.026 [error] <0.4340.0>@yz_pb_search:maybe_process:111
> {solr_error,{500,"http://localhost:8093/internal_solr/process_test_result/select
> ",<<"{\"error\":{\"msg\":\"org.apache.solr.client.solrj.SolrServerException:
> Server refused connection at:
> http://nosql-3.dsdb:8093/internal_solr/process_
> test_result\",\"trace\":\"org.apache.solr.common.SolrException:
> org.apache.solr.client.solrj.SolrServerException: Server refused connection
> at: http://nos
> ql-3.dsdb:8093/internal_solr/process_test_result\\n\\tat
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:308)\\n\\tat
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\\n\\tat
> org.apache.solr.core.SolrCore.execute(SolrCore.java:1916)\\
> n\\tat
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780)\\n\\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDis
> patchFilter.java:427)\\n\\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)\\n\\tat
> org.eclipse.jetty.servlet.ServletHa
> ndler$CachedChain.doFilter(ServletHandler.java:1419)\\n\\tat
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)\\n\\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)\\n\\tat
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)\\n\\tat
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)\\n\\tat
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)\\n\\tat
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)\\n\\tat
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)\\n\\tat
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)\\n\\tat
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)\\n\\tat
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)\\n\\tat
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)\\n\\tat
> org.eclipse.jetty.server.handler.HandlerWrap...">>}}
> [{yz_solr,search,3,[{file,"src/yz_solr.erl"},{line,292}]},{yz_pb_search,maybe_process,3,[{file,"src/yz_pb_search.erl"},{line,76}]},{riak_api_pb_server,process_message,4,[{file,"src/riak_api_pb_server.erl"},{line,388}]},{riak_api_pb_server,connected,2,[{file,"src/riak_api_pb_server.erl"},{line,226}]},{riak_api_pb_server,decode_buffer,2,[{file,"src/riak_api_pb_server.erl"},{line,364}]},{gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,505}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]
> 2017-02-04 18:17:02.756 [error]
> <0.6871.0>@riak_core_handoff_receiver:handle_info:97 Handoff receiver for
> partition 0 exited abnormally after processing 0 objects from
> {"192.168.9.247",39483}:
> {error,{vnode_timeout,60000,844,<<131,104,2,100,0,10,101,110,99,111,100,101,95,114,97,119,104,3,104,2,109,0,0,0,19,112,114,111,99,101,115,115,95,116,101,115,116,95,114,101,115,117,108,116,109,0,0,0,35,83,104,105,76,111,110,103,90,105,95,77,66,80,65,67,75,45,231,148,159,228,186,167,232,189,166,233,151,180,50,48,49,55,50,52,109,0,0,0,18,57,72,53,65,109,84,122,54,79,109,99,108,87,107,87,68,80,85,109,0,0,2,220,53,1,0,0,0,38,131,108,0,0,0,1,104,2,109,0>>}}
>
>
> Br,
>
> Alex
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>




More information about the riak-users mailing list