slow 2 node cluster

Catalin Constantin dazoot at gmail.com
Sun Nov 20 16:34:15 EST 2011


To make it simple. No more networking. Just one node (with n = 1) and local
tests.

The producing of data is a simple CSV file read (ruled out too cause this
is fast).
Switched to bitcask:
{storage_backend, riak_kv_bitcask_backend},

Removed the index from the RiakObject (so now we have simple k/v).
Created a new bucket (just to be "clean").
And rerun the test.

Now it's better:
~450 inserts / second

I still think it's slow.

Hardware is pretty good:

Intel® Core™ i7-920 Quadcore incl. Hyper-Threading Technology

RAM: 8GB DDR3

HDD: 2 x 750 GB SATA 2 (RAID1)


What insert rate should i expect on a normal Debian 6.0 64 bit installation
(no tweaks) ?

I can only compare it with other DBs i have tested on the same machine:
ex: mongodb, kyototycoon

On Sun, Nov 20, 2011 at 10:56 PM, Aphyr <aphyr at aphyr.com> wrote:

> On 11/20/2011 12:14 PM, Catalin Constantin wrote:
>
>> I am 100% sure the transfer rate is 10MBytes / second. This is not the
>> problem.
>>
>
> In ten years of network administration I have never encountered an
> ethernet device with a wire rate of 10 MBps. I have, however, encountered
> frequent confusion over units. Perhaps you understand my suspicion here. :)
>
>
>  IOWAIT is also pretty low. iostat shows: iowait 4.69%
>>
>
> If you read my email, I suggested that even values as low as 2.6% may
> suggest contention. It depends on your CPU arch and utilization. I would
> investigate your disks more closely. Are they spinning or solid-state?
> Median seek time? 95/99 seek times? Disk cache? Does the riak process spend
> disproportionate time in IO_WAIT relative to USER? Filesystem
> atime/relatime/noatime? FS block size properly aligned for your disk?
> Insufficient filesystem or leveldb cache? hdparm options? Is riak on an
> independent disk or competing with other processes, i.e. syslog, file
> servers, etc? Does strace show an unusual amount of time spent in certain
> system calls?
>
> It might just be leveldb, too. I only have experience with bitcask in
> production.
>
>
>  I have retried the test with one node, a new bucket newly created where
>> i have set: n to 1.
>> bucket.set_n_val(1)
>>
>> Results are the same. Less than 300 inserts / second.
>>
>
> This is good; it rules out replication.
>
>
>  Any idea why riak is so slow on inserting data ?
>>
>
> Disk, disk disk disk, disk disk? Disk!
>
> You could also look at the client. Are you writing to a local node or a
> remote one? Is your client's threading model getting in the way? Can you
> actually produce data fast enough to insert it? Is your client fighting for
> the same resources as the riak process? Presuming you've ruled these out,
> it's almost certainly network or disk.
>
> --Kyle
>



-- 
Catalin Constantin
Dazoot Software
http://www.dazoot.eu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111120/a6dea499/attachment.html>


More information about the riak-users mailing list