Crash Error

Jason Ryan jason.ryan at trustev.com
Wed Mar 4 11:37:04 EST 2015


We're seeing regular errors around indexes trying to be created - but they
already exist - indexes are only ever created manually, and no new indexes
have been created in quite a while. We have however added new nodes and
removed other nodes due to some node failures.

We also have large object warnings, >5MB (generally around 7MB) - which we
address each time we see them and mitigate the errors for the future...

Other than that we don't see many other errors, so we're unsure why the
cluster tends to fail.



[image: photo]
*Jason Ryan*
VP Engineering

Trustev
Real Time, Online Identity Verification

email: jason.ryan at trustev.com
skype: jason_j_ryan
web: www.trustev.com

Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland.

On 4 March 2015 at 16:14, Sean Cribbs <sean at basho.com> wrote:

> Are there any other significant error messages in the logs? I find it hard
> to believe that that specific crash would make your cluster unusable.
>
> On Wed, Mar 4, 2015 at 10:07 AM, Jason Ryan <jason.ryan at trustev.com>
> wrote:
>
>> This is what we have on our haproxy right now:
>>
>> backend riak_rest_backend
>>         mode               http
>>         option             httplog
>>         balance            roundrobin
>>         option             httpchk GET /ping
>>         server EU-RIAK-5 10.0.4.12:8098 weight 1 maxconn 1024  check
>>         server EU-RIAK-6 10.0.4.13:8098 weight 1 maxconn 1024  check
>>         server EU-RIAK-7 10.0.4.14:8098 weight 1 maxconn 1024  check
>>         server EU-RIAK-8 10.0.4.15:8098 weight 1 maxconn 1024  check
>>         server EU-RIAK-9 10.0.4.16:8098 weight 1 maxconn 1024  check
>>>>
>>
>>
>> [image: photo]
>> *Jason Ryan*
>> VP Engineering
>>
>> Trustev
>> Real Time, Online Identity Verification
>>
>> email: jason.ryan at trustev.com
>> skype: jason_j_ryan
>> web: www.trustev.com
>>
>> Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland.
>>
>> On 4 March 2015 at 15:58, Sean Cribbs <sean at basho.com> wrote:
>>
>>> We recently saw this on the PB interface as well and have mitigated it
>>> [1], but we have not addressed it on HTTP. In the meantime, I would
>>> recommend changing your health check to use a regular request, for example:
>>>
>>> option httpchk /ping
>>>
>>> [1] https://github.com/basho/riak_api/pull/54
>>>
>>> On Wed, Mar 4, 2015 at 9:46 AM, Jason Ryan <jason.ryan at trustev.com>
>>> wrote:
>>>
>>>> Hi Sean - yes we are using HAProxy - and there are health checks.
>>>>
>>>> [image: photo]
>>>> *Jason Ryan*
>>>> VP Engineering
>>>>
>>>> Trustev
>>>> Real Time, Online Identity Verification
>>>>
>>>> email: jason.ryan at trustev.com
>>>> skype: jason_j_ryan
>>>> web: www.trustev.com
>>>>
>>>> Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland.
>>>>
>>>> On 4 March 2015 at 15:41, Sean Cribbs <sean at basho.com> wrote:
>>>>
>>>>> Are you using a load balancer in front of Riak, like HAProxy, with
>>>>> health checks turned on?
>>>>>
>>>>> On Wed, Mar 4, 2015 at 7:24 AM, Jason Ryan <jason.ryan at trustev.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We're getting the following crash report messages in the log very
>>>>>> regularly in the last day - it renders our Riak Cluster pretty much
>>>>>> unusable - until we restart Riak on each node. All nodes report the same
>>>>>> type of error.
>>>>>>
>>>>>> 2015-03-04 13:17:43.106 [error] <0.31063.9> CRASH REPORT Process
>>>>>> <0.31063.9> with 0 neighbours crashed with reason: no function clause
>>>>>> matching webmachine_request:peer_from_peername({error,enotconn},
>>>>>> {webmachine_request,{wm_reqstate,#Port<0.89869>,[],undefined,undefined,undefined,{wm_reqdata,'GET',...},...}})
>>>>>> line 150
>>>>>>
>>>>>>
>>>>>> Anyone seen this before, know what the cause might be?
>>>>>>
>>>>>> Thanks,
>>>>>> Jason
>>>>>>
>>>>>> [image: photo]
>>>>>> *Jason Ryan*
>>>>>> VP Engineering
>>>>>>
>>>>>> Trustev
>>>>>> Real Time, Online Identity Verification
>>>>>>
>>>>>> email: jason.ryan at trustev.com
>>>>>> skype: jason_j_ryan
>>>>>> web: www.trustev.com
>>>>>>
>>>>>> Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland.
>>>>>>
>>>>>> This message is for the named person's use only. If you received this
>>>>>> message in error, please immediately delete it and all copies and notify
>>>>>> the sender. You must not, directly or indirectly, use, disclose,
>>>>>> distribute, print, or copy any part of this message if you are not the
>>>>>> intended recipient. Any views expressed in this message are those of the
>>>>>> individual sender and not Trustev Ltd. Trustev is registered in
>>>>>> Ireland No. 516425 and trades from 2100 Cork Airport Business Park, Cork,
>>>>>> Ireland.
>>>>>>
>>>>>> _______________________________________________
>>>>>> riak-users mailing list
>>>>>> riak-users at lists.basho.com
>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sean Cribbs <sean at basho.com>
>>>>> Sr. Software Engineer
>>>>> Basho Technologies, Inc.
>>>>> http://basho.com/
>>>>>
>>>>
>>>>
>>>> This message is for the named person's use only. If you received this
>>>> message in error, please immediately delete it and all copies and notify
>>>> the sender. You must not, directly or indirectly, use, disclose,
>>>> distribute, print, or copy any part of this message if you are not the
>>>> intended recipient. Any views expressed in this message are those of the
>>>> individual sender and not Trustev Ltd. Trustev is registered in
>>>> Ireland No. 516425 and trades from 2100 Cork Airport Business Park, Cork,
>>>> Ireland.
>>>>
>>>
>>>
>>>
>>> --
>>> Sean Cribbs <sean at basho.com>
>>> Sr. Software Engineer
>>> Basho Technologies, Inc.
>>> http://basho.com/
>>>
>>
>>
>> This message is for the named person's use only. If you received this
>> message in error, please immediately delete it and all copies and notify
>> the sender. You must not, directly or indirectly, use, disclose,
>> distribute, print, or copy any part of this message if you are not the
>> intended recipient. Any views expressed in this message are those of the
>> individual sender and not Trustev Ltd. Trustev is registered in Ireland
>> No. 516425 and trades from 2100 Cork Airport Business Park, Cork, Ireland.
>>
>
>
>
> --
> Sean Cribbs <sean at basho.com>
> Sr. Software Engineer
> Basho Technologies, Inc.
> http://basho.com/
>

-- 


This message is for the named person's use only. If you received this 
message in error, please immediately delete it and all copies and notify 
the sender. You must not, directly or indirectly, use, disclose, 
distribute, print, or copy any part of this message if you are not the 
intended recipient. Any views expressed in this message are those of the 
individual sender and not Trustev Ltd. Trustev is registered in Ireland No. 
516425 and trades from 2100 Cork Airport Business Park, Cork, Ireland.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150304/69a12785/attachment-0002.html>


More information about the riak-users mailing list