slow 2 node cluster

Aphyr aphyr at aphyr.com
Sun Nov 20 14:27:04 EST 2011


On 11/20/2011 05:19 AM, Catalin Constantin wrote:
> The connection between servers is 10MBytes / sec not 10Mbit / sec.

Are you sure? To my knowledge almost no ethernet gear runs at 10 MB/s. 
It's almost always 10, 100, 1000, or 10000 Mb/s.

It may be your n_val. If it's the default (3), one of your two machines 
has to handle that third replica. That'll cut your throughput 
significantly. Finally, your disks may be the bottleneck. I'd take a 
look at iostat and look for significant (on our servers, 2.6% meant we 
were thrashing) IO_WAIT time on the riak beam.smp process.

--Kyle

> One row of data looks like this:
> 309819178dazoot at gmail.com
> <mailto:dazoot at gmail.com>55942dzt1home2011-05-3116:22:102011-09-07
> 17:03:48127.0.0.111
>
> I use Protobuf transport.
> If dw = 0 and w = 0 there is no wait for other replica nodes, right ?
>
> This should improve the write speed, correct ?
>
> On Sun, Nov 20, 2011 at 2:21 PM, Erik Søe Sørensen <ess at trifork.com
> <mailto:ess at trifork.com>> wrote:
>
>     This depends quite a bit on the sizes of your objects.
>     Supposing an average size of 2KB, and n=3: on each write, on average
>     1.5 of the replicas would be on the other node, implying inter-node
>     network traffic of 1.5*2KB=3KB (and this is just in one direction).
>     If your inter-node network connection is indeed 10Mbit ~  1MB/s,
>     then 300 writes/s * 3KB = 0.9MB/s would just about saturate the
>     connection.
>
>     You may want to check the network utilization.
>     ________________________________________
>     From: riak-users-bounces at lists.basho.com
>     <mailto:riak-users-bounces at lists.basho.com>
>     [riak-users-bounces at lists.basho.com
>     <mailto:riak-users-bounces at lists.basho.com>] On Behalf Of Catalin
>     Constantin [dazoot at gmail.com <mailto:dazoot at gmail.com>]
>     Sent: 20 November 2011 11:10
>     To: riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>     Subject: slow 2 node cluster
>
>     Hello,
>
>     I am trying to evaluate / run some tests on a 4 mil dataset.
>     I have 2 nodes setup (different machines - 8GB ram each, I7 cpus).
>     10 MB connection between them.
>
>     I am trying to insert data into riak using w=1 and dw=1 (also tried
>     with dw = 0, w = 0).
>     For each risk object i have 4 indexes (2 binary 2 int).
>     Riak backend is leveldb.
>
>     I can't get more than 300 inserts per second.
>     I have also tried running 2 threads each hitting the different node.
>     Nothing changed too much.
>
>     Is this normal behavior ?
>
>     --
>     Catalin Constantin
>
>
>
>
> --
> Catalin Constantin
> Dazoot Software
> http://www.dazoot.eu/
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




More information about the riak-users mailing list