Tune Riak for fast inserts - populate DB

Bogdan Flueras flueras.bogdan at gmail.com
Thu Feb 14 06:05:16 EST 2013


On 02/14/2013 12:19 PM, Erik Søe Sørensen wrote:
> Those 18.4MB/s is utilized bandwidth, I presume?
Yes. What I find odd is that riak is writing max 6-700KB/sec on disk.
> And how about the net between client and Riak?
They're on the same machine: the Java client and the riak process (one 
node).
> Further questions:
I've already answered, but it's ok to refresh again:
> - Which backend are you using - LevelDB?
Bitcask, many writer threads each with it's own riakClient instance 
configured to use all nodes in cluster.
>   (If LevelDB: how does it look over time - is it faster at first, but 
> slowing down?)
> - And how many writes do you do in parallel?
>   If you're only writing from one thread, your problem is probably not 
> bandwidth, but latency.
Regards,
>
> On 14-02-2013 10:21, Bogdan Flueras wrote:
>> What I have is:
>>
>> Network bandwidth between nodes: 18.4 MB/sec
>> Network cards: 1.000Mbps full duplex
>> Riak process: IO: avg 600 K/s for disk write, 7% RAM used, 17% CPU used
>>
>>
>> ing. Bogdan Flueras
>>
>>
>>
>> On Thu, Feb 14, 2013 at 10:17 AM, Erik Søe Sørensen <ess at trifork.com 
>> <mailto:ess at trifork.com>> wrote:
>>
>>     Just to rule something out: what is the capacity of the network
>>     between the cluster and the client - and among the cluster nodes?
>>
>>
>>     -------- Oprindelig meddelelse --------
>>     Fra: Shuhao <shuhao at shuhaowu.com <mailto:shuhao at shuhaowu.com>>
>>     Dato:
>>     Til: riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>     Emne: Re: Tune Riak for fast inserts - populate DB
>>
>>
>>     Hi,
>>
>>     I've had similar experiences with riak. I used the Riak Python client
>>     with HTTP and PBC (even rolled my own client! That only had a
>>     ~10% speed
>>     up though, and it was HTTP only). I've also examined that the
>>     Riak host
>>     machine's CPU is only at around 70% when I insert. Other NoSQL
>>     dbs uses
>>     about 100% when I just continuous insert. Some DBs even have bulk
>>     insert.
>>
>>     Maybe this is an issue with Riak itself... I don't really know.
>>     Maybe a
>>     Riak dev could chip in on the conversation
>>
>>     Cheers,
>>
>>     Shuhao
>>
>>     On 13-02-13 02:37 AM, Bogdan Flueras wrote:
>>     > Hello all,
>>     > I've got a 5 node cluster with Riak 1.2.1, all machines are
>>     multicore,
>>     > with min 4GB RAM.
>>     >
>>     > I want to insert something like 50 million records in Riak with
>>     the java
>>     > client (Protobuf used) with default settings.  I've tried also
>>     with HTTP
>>     > protocol and set w = 1 but got some problems.
>>     >
>>     > However the process is very slow: it doesn't write more than
>>     6GB/ hour or
>>     > aprox. 280 KB/second.
>>     > To have all my data filled in, it would take aprox 2 days !!
>>     >
>>     > What can I do to have the data filled into Riak ASAP?
>>     > How should I configure the cluster ? (vm.args/ app.config) I
>>     don't care so
>>     > much about consistency at this point.
>>     >
>>     > Thank you,
>>     > ing. Bogdan Flueras
>>     >
>>
>
> -- 
> Mobile: + 45 26 36 17 55 | Skype: eriksoesorensen | Twitter: @eriksoe
> Trifork A/S  |  Margrethepladsen 4  |  DK-8000 Aarhus C | 
> www.trifork.com <http://www.trifork.com/>


-- 
ing. Bogdan Flueras

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130214/39f08d05/attachment.html>


More information about the riak-users mailing list