Node unreachable even though the nodes are pingable

Richard Shaw richard at basho.com
Tue Feb 12 10:54:22 EST 2013


Hi Shrikar,

I'd like to confirm there isn't underlying network weirdness going on.

You said in your mail that "During this time the latency increases to 1
second or more than that." can you elaborate, was that from running ping
$node manually that you saw the latency increase?

What network monitoring do you have in place on your network at the moment?
If nothing formal, leave mtr[0] running in a screen/tmux session between
your nodes then review the latency and loss.

[0]http://www.bitwizard.nl/mtr/

Regards

Richard

On Tue, Feb 12, 2013 at 6:46 AM, Shrikar archak <shrikar84 at gmail.com> wrote:

> Sorry guys for the delay in response.
>
> To answer dimitry's question
> This is what I observed
>
> Actual nodes ping : Works ( I literally mean ping riak1 , ping riak2 ....)
> I didnt try riak ping .. But from each Riak node I tried riak-admin
> ring-status , this is were I saw the some nodes not reachable.
>
> Example
> from riak6(192.168.8.136) I tried the above command and it replied back
> saying riak4(192.168.8.134) was not reachable
>
> And the same command from riak4( 192.168.8.134) replied back saying riak6
> not reachable.. So I assume the riak ping on those nodes would
> have returned with pong reply.
>
> To answer Mark's question .. That was a typo in the mail I gave the actual
> ip correctly.
> Thanks,
> Shrikar
>
>
> On Mon, Feb 11, 2013 at 3:45 PM, Mark Phillips <mark at basho.com> wrote:
>
>> Hi Shrikar,
>>
>> I suspect you've gotten to the bottom of this (as we haven't heard
>> from you in a few days), but upon re-reading this I noticed the
>> following:
>>
>> >
>> >
>> The following nodes are unreachable: ['riak at 192.168.8.134',
>>                                       'riak at 192.168.8.135']
>>
>> electic at riak-6:/var/lib/riak/leveldb$ ping 92.168.8.134
>> PING 92.168.8.134 (92.168.8.134) 56(84) bytes of data.
>> ^C
>> --- 92.168.8.134 ping statistics ---
>> 1 packets transmitted, 0 received, 100% packet loss, time 0ms
>> >
>> >
>>
>> It looks like you're trying to ping "92.168.8.134", which is different
>> from the "192.168.8.134" address above.
>>
>> That may not solve the issue, but it might not be a bad place to start.
>>
>> Hope that helps. Let us know if you're still chasing this.
>>
>> Mark
>>
>>
>>
>>
>> On Mon, Feb 4, 2013 at 4:20 PM, Shrikar archak <shrikar84 at gmail.com>
>> wrote:
>> > Hi All,
>> > We have a 6 node riak cluster running on riak 1.2.1
>> > I am often seeing this issue where the nodes become unreachable
>> > when I run the riak-admin command
>> >
>> > electic at riak-6:/var/lib/riak/leveldb$ riak-admin ring-status
>> > Attempting to restart script through sudo -H -u riak
>> > ================================== Claimant
>> > ===================================
>> > Claimant:  'riak at 192.168.8.132'
>> > Status:     up
>> > Ring Ready: true
>> >
>> > ============================== Ownership Handoff
>> > ==============================
>> > No pending changes.
>> >
>> > ============================== Unreachable Nodes
>> > ==============================
>> > The following nodes are unreachable: ['riak at 192.168.8.134',
>> >                                       'riak at 192.168.8.135']
>> >
>> > electic at riak-6:/var/lib/riak/leveldb$ ping 92.168.8.134
>> > PING 92.168.8.134 (92.168.8.134) 56(84) bytes of data.
>> > ^C
>> > --- 92.168.8.134 ping statistics ---
>> > 1 packets transmitted, 0 received, 100% packet loss, time 0ms
>> >
>> > electic at riak-6:/var/lib/riak/leveldb$ ping 192.168.8.134
>> > PING 192.168.8.134 (192.168.8.134) 56(84) bytes of data.
>> > 64 bytes from 192.168.8.134: icmp_req=1 ttl=64 time=0.062 ms
>> > 64 bytes from 192.168.8.134: icmp_req=2 ttl=64 time=0.057 ms
>> >
>> > electic at riak-6:/var/lib/riak/leveldb$ ping 192.168.8.135
>> > PING 192.168.8.135 (192.168.8.135) 56(84) bytes of data.
>> > 64 bytes from 192.168.8.135: icmp_req=1 ttl=64 time=0.065 ms
>> > 64 bytes from 192.168.8.135: icmp_req=2 ttl=64 time=0.057 ms
>> >
>> > Has anyone faced similar problems like these?
>> >
>> > During this time the latency increases to 1 second or more than that.
>> >
>> > Please let me know if there is a way to get around this problem.
>> >
>> > Thanks,
>> > Shrikar
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > riak-users at lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130212/499181f5/attachment.html>


More information about the riak-users mailing list