slow 2 node cluster

Catalin Constantin dazoot at gmail.com
Sun Nov 20 15:14:14 EST 2011


I am 100% sure the transfer rate is 10MBytes / second. This is not the
problem.
IOWAIT is also pretty low. iostat shows: iowait 4.69%

I have retried the test with one node, a new bucket newly created where i
have set: n to 1.
bucket.set_n_val(1)

Results are the same. Less than 300 inserts / second.

Any idea why riak is so slow on inserting data ?


On Sun, Nov 20, 2011 at 9:27 PM, Aphyr <aphyr at aphyr.com> wrote:

> On 11/20/2011 05:19 AM, Catalin Constantin wrote:
>
>> The connection between servers is 10MBytes / sec not 10Mbit / sec.
>>
>
> Are you sure? To my knowledge almost no ethernet gear runs at 10 MB/s.
> It's almost always 10, 100, 1000, or 10000 Mb/s.
>
> It may be your n_val. If it's the default (3), one of your two machines
> has to handle that third replica. That'll cut your throughput
> significantly. Finally, your disks may be the bottleneck. I'd take a look
> at iostat and look for significant (on our servers, 2.6% meant we were
> thrashing) IO_WAIT time on the riak beam.smp process.
>
> --Kyle
>
>  One row of data looks like this:
>> 309819178dazoot at gmail.com
>> <mailto:dazoot at gmail.com>**55942dzt1home2011-05-3116:22:**102011-09-07
>> 17:03:48127.0.0.111
>>
>>
>> I use Protobuf transport.
>> If dw = 0 and w = 0 there is no wait for other replica nodes, right ?
>>
>> This should improve the write speed, correct ?
>>
>> On Sun, Nov 20, 2011 at 2:21 PM, Erik Søe Sørensen <ess at trifork.com
>> <mailto:ess at trifork.com>> wrote:
>>
>>    This depends quite a bit on the sizes of your objects.
>>    Supposing an average size of 2KB, and n=3: on each write, on average
>>    1.5 of the replicas would be on the other node, implying inter-node
>>    network traffic of 1.5*2KB=3KB (and this is just in one direction).
>>    If your inter-node network connection is indeed 10Mbit ~  1MB/s,
>>    then 300 writes/s * 3KB = 0.9MB/s would just about saturate the
>>    connection.
>>
>>    You may want to check the network utilization.
>>    ______________________________**__________
>>    From: riak-users-bounces at lists.**basho.com<riak-users-bounces at lists.basho.com>
>>    <mailto:riak-users-bounces@**lists.basho.com<riak-users-bounces at lists.basho.com>
>> >
>>    [riak-users-bounces at lists.**basho.com<riak-users-bounces at lists.basho.com>
>>    <mailto:riak-users-bounces@**lists.basho.com<riak-users-bounces at lists.basho.com>>]
>> On Behalf Of Catalin
>>    Constantin [dazoot at gmail.com <mailto:dazoot at gmail.com>]
>>
>>    Sent: 20 November 2011 11:10
>>    To: riak-users at lists.basho.com <mailto:riak-users at lists.**basho.com<riak-users at lists.basho.com>
>> >
>>
>>    Subject: slow 2 node cluster
>>
>>    Hello,
>>
>>    I am trying to evaluate / run some tests on a 4 mil dataset.
>>    I have 2 nodes setup (different machines - 8GB ram each, I7 cpus).
>>    10 MB connection between them.
>>
>>    I am trying to insert data into riak using w=1 and dw=1 (also tried
>>    with dw = 0, w = 0).
>>    For each risk object i have 4 indexes (2 binary 2 int).
>>    Riak backend is leveldb.
>>
>>    I can't get more than 300 inserts per second.
>>    I have also tried running 2 threads each hitting the different node.
>>    Nothing changed too much.
>>
>>    Is this normal behavior ?
>>
>>    --
>>    Catalin Constantin
>>
>>
>>
>>
>> --
>> Catalin Constantin
>> Dazoot Software
>> http://www.dazoot.eu/
>>
>>
>>
>> ______________________________**_________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/**mailman/listinfo/riak-users_**lists.basho.com<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>>
>


-- 
Catalin Constantin
Dazoot Software
http://www.dazoot.eu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111120/ddee9317/attachment.html>


More information about the riak-users mailing list