Tune Riak for fast inserts - populate DB
flueras.bogdan at gmail.com
Thu Feb 14 06:05:16 EST 2013
On 02/14/2013 12:19 PM, Erik Søe Sørensen wrote:
> Those 18.4MB/s is utilized bandwidth, I presume?
Yes. What I find odd is that riak is writing max 6-700KB/sec on disk.
> And how about the net between client and Riak?
They're on the same machine: the Java client and the riak process (one
> Further questions:
I've already answered, but it's ok to refresh again:
> - Which backend are you using - LevelDB?
Bitcask, many writer threads each with it's own riakClient instance
configured to use all nodes in cluster.
> (If LevelDB: how does it look over time - is it faster at first, but
> slowing down?)
> - And how many writes do you do in parallel?
> If you're only writing from one thread, your problem is probably not
> bandwidth, but latency.
> On 14-02-2013 10:21, Bogdan Flueras wrote:
>> What I have is:
>> Network bandwidth between nodes: 18.4 MB/sec
>> Network cards: 1.000Mbps full duplex
>> Riak process: IO: avg 600 K/s for disk write, 7% RAM used, 17% CPU used
>> ing. Bogdan Flueras
>> On Thu, Feb 14, 2013 at 10:17 AM, Erik Søe Sørensen <ess at trifork.com
>> <mailto:ess at trifork.com>> wrote:
>> Just to rule something out: what is the capacity of the network
>> between the cluster and the client - and among the cluster nodes?
>> -------- Oprindelig meddelelse --------
>> Fra: Shuhao <shuhao at shuhaowu.com <mailto:shuhao at shuhaowu.com>>
>> Til: riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>> Emne: Re: Tune Riak for fast inserts - populate DB
>> I've had similar experiences with riak. I used the Riak Python client
>> with HTTP and PBC (even rolled my own client! That only had a
>> ~10% speed
>> up though, and it was HTTP only). I've also examined that the
>> Riak host
>> machine's CPU is only at around 70% when I insert. Other NoSQL
>> dbs uses
>> about 100% when I just continuous insert. Some DBs even have bulk
>> Maybe this is an issue with Riak itself... I don't really know.
>> Maybe a
>> Riak dev could chip in on the conversation
>> On 13-02-13 02:37 AM, Bogdan Flueras wrote:
>> > Hello all,
>> > I've got a 5 node cluster with Riak 1.2.1, all machines are
>> > with min 4GB RAM.
>> > I want to insert something like 50 million records in Riak with
>> the java
>> > client (Protobuf used) with default settings. I've tried also
>> with HTTP
>> > protocol and set w = 1 but got some problems.
>> > However the process is very slow: it doesn't write more than
>> 6GB/ hour or
>> > aprox. 280 KB/second.
>> > To have all my data filled in, it would take aprox 2 days !!
>> > What can I do to have the data filled into Riak ASAP?
>> > How should I configure the cluster ? (vm.args/ app.config) I
>> don't care so
>> > much about consistency at this point.
>> > Thank you,
>> > ing. Bogdan Flueras
> Mobile: + 45 26 36 17 55 | Skype: eriksoesorensen | Twitter: @eriksoe
> Trifork A/S | Margrethepladsen 4 | DK-8000 Aarhus C |
> www.trifork.com <http://www.trifork.com/>
ing. Bogdan Flueras
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users