Node unreachable even though the nodes are pingable

Shrikar archak shrikar84 at gmail.com
Tue Feb 12 12:34:43 EST 2013


Hi Richard,
Thanks for getting back. Yes the nodes which were not able to see each
other in my case riak4 and riak6 showed increased latency. Other node
latency
was fine.Yes I will start running mtr[0] and will update if this thing
happens again.

Thanks,
Shrikar


On Tue, Feb 12, 2013 at 7:54 AM, Richard Shaw <richard at basho.com> wrote:

> Hi Shrikar,
>
> I'd like to confirm there isn't underlying network weirdness going on.
>
> You said in your mail that "During this time the latency increases to 1
> second or more than that." can you elaborate, was that from running ping
> $node manually that you saw the latency increase?
>
> What network monitoring do you have in place on your network at the
> moment? If nothing formal, leave mtr[0] running in a screen/tmux session
> between your nodes then review the latency and loss.
>
> [0]http://www.bitwizard.nl/mtr/
>
> Regards
>
> Richard
>
> On Tue, Feb 12, 2013 at 6:46 AM, Shrikar archak <shrikar84 at gmail.com>wrote:
>
>> Sorry guys for the delay in response.
>>
>> To answer dimitry's question
>> This is what I observed
>>
>> Actual nodes ping : Works ( I literally mean ping riak1 , ping riak2 ....)
>> I didnt try riak ping .. But from each Riak node I tried riak-admin
>> ring-status , this is were I saw the some nodes not reachable.
>>
>> Example
>> from riak6(192.168.8.136) I tried the above command and it replied back
>> saying riak4(192.168.8.134) was not reachable
>>
>> And the same command from riak4( 192.168.8.134) replied back saying riak6
>> not reachable.. So I assume the riak ping on those nodes would
>> have returned with pong reply.
>>
>> To answer Mark's question .. That was a typo in the mail I gave the
>> actual ip correctly.
>> Thanks,
>> Shrikar
>>
>>
>> On Mon, Feb 11, 2013 at 3:45 PM, Mark Phillips <mark at basho.com> wrote:
>>
>>> Hi Shrikar,
>>>
>>> I suspect you've gotten to the bottom of this (as we haven't heard
>>> from you in a few days), but upon re-reading this I noticed the
>>> following:
>>>
>>> >
>>> >
>>> The following nodes are unreachable: ['riak at 192.168.8.134',
>>>                                       'riak at 192.168.8.135']
>>>
>>> electic at riak-6:/var/lib/riak/leveldb$ ping 92.168.8.134
>>> PING 92.168.8.134 (92.168.8.134) 56(84) bytes of data.
>>> ^C
>>> --- 92.168.8.134 ping statistics ---
>>> 1 packets transmitted, 0 received, 100% packet loss, time 0ms
>>> >
>>> >
>>>
>>> It looks like you're trying to ping "92.168.8.134", which is different
>>> from the "192.168.8.134" address above.
>>>
>>> That may not solve the issue, but it might not be a bad place to start.
>>>
>>> Hope that helps. Let us know if you're still chasing this.
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>> On Mon, Feb 4, 2013 at 4:20 PM, Shrikar archak <shrikar84 at gmail.com>
>>> wrote:
>>> > Hi All,
>>> > We have a 6 node riak cluster running on riak 1.2.1
>>> > I am often seeing this issue where the nodes become unreachable
>>> > when I run the riak-admin command
>>> >
>>> > electic at riak-6:/var/lib/riak/leveldb$ riak-admin ring-status
>>> > Attempting to restart script through sudo -H -u riak
>>> > ================================== Claimant
>>> > ===================================
>>> > Claimant:  'riak at 192.168.8.132'
>>> > Status:     up
>>> > Ring Ready: true
>>> >
>>> > ============================== Ownership Handoff
>>> > ==============================
>>> > No pending changes.
>>> >
>>> > ============================== Unreachable Nodes
>>> > ==============================
>>> > The following nodes are unreachable: ['riak at 192.168.8.134',
>>> >                                       'riak at 192.168.8.135']
>>> >
>>> > electic at riak-6:/var/lib/riak/leveldb$ ping 92.168.8.134
>>> > PING 92.168.8.134 (92.168.8.134) 56(84) bytes of data.
>>> > ^C
>>> > --- 92.168.8.134 ping statistics ---
>>> > 1 packets transmitted, 0 received, 100% packet loss, time 0ms
>>> >
>>> > electic at riak-6:/var/lib/riak/leveldb$ ping 192.168.8.134
>>> > PING 192.168.8.134 (192.168.8.134) 56(84) bytes of data.
>>> > 64 bytes from 192.168.8.134: icmp_req=1 ttl=64 time=0.062 ms
>>> > 64 bytes from 192.168.8.134: icmp_req=2 ttl=64 time=0.057 ms
>>> >
>>> > electic at riak-6:/var/lib/riak/leveldb$ ping 192.168.8.135
>>> > PING 192.168.8.135 (192.168.8.135) 56(84) bytes of data.
>>> > 64 bytes from 192.168.8.135: icmp_req=1 ttl=64 time=0.065 ms
>>> > 64 bytes from 192.168.8.135: icmp_req=2 ttl=64 time=0.057 ms
>>> >
>>> > Has anyone faced similar problems like these?
>>> >
>>> > During this time the latency increases to 1 second or more than that.
>>> >
>>> > Please let me know if there is a way to get around this problem.
>>> >
>>> > Thanks,
>>> > Shrikar
>>> >
>>> > _______________________________________________
>>> > riak-users mailing list
>>> > riak-users at lists.basho.com
>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> >
>>>
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130212/a936369e/attachment.html>


More information about the riak-users mailing list