Riak 1.4.2 10G Ethernet Performance Problems

Chris Read chris.read at gmail.com
Mon Oct 28 10:59:33 EDT 2013


Yes, we have have jumbo frames on.

Kernel and systems are in a good state. I've got the settings from the
Linux perf tuning guide on the riak docs. Testing with iperf(1) shows our
nodes can saturate the 10G interfaces without breaking a sweat - as long as
the packets are large enough...

1) http://iperf.sourceforge.net/


On Sat, Oct 26, 2013 at 11:36 AM, Sean Cribbs <sean at basho.com> wrote:

> Chris,
>
> Are you using jumbo frames on your network? (MTU 9000 or w/e) That's just
> one aspect, but we've also found there are many other kernel/TCP-stack
> tunings that need to be done to get Riak to push the limits of the network.
>
>
> On Fri, Oct 25, 2013 at 9:10 PM, Chris Read <chris.read at gmail.com> wrote:
>
>> Greetings all...
>>
>> Context: We're running Riak 1.4.2 on Ubuntu 12.04 using the Basho build
>> .deb package. Out object size varies between 1K and 512K
>>
>> I've been having some trouble trying to figure out why handoffs between
>> our nodes takes so long and never pushes the network card over 2.5 Gb/s.
>> Testing communication between nodes using iperf shows that the
>> configuration we have can sustain 10G.
>>
>> Further investigation shows that communication between nodes never uses
>> messages > 4k which gets the network cards to top out on packages per
>> second before they reach the top bits per second.
>>
>> Possibly related: I see exactly the same problem when using Basho Bench
>> and the pb driver - I can never get the client to go over 2.5Gb/s. This is
>> pushing me to suspect erlang vm or something as the problem.
>>
>> Anyone seen this before? Is there a magic "use bigger messages" flag
>> hidden somewhere?
>>
>> Thanks,
>>
>> Chris
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> Sean Cribbs <sean at basho.com>
> Software Engineer
> Basho Technologies, Inc.
> http://basho.com/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131028/4d2b080c/attachment.html>


More information about the riak-users mailing list