Tune Riak for fast inserts - populate DB

Erik Søe Sørensen ess at trifork.com
Thu Feb 14 05:19:09 EST 2013


Those 18.4MB/s is utilized bandwidth, I presume?
And how about the net between client and Riak?

Further questions:
- Which backend are you using - LevelDB?
   (If LevelDB: how does it look over time - is it faster at first, but 
slowing down?)
- And how many writes do you do in parallel?
   If you're only writing from one thread, your problem is probably not 
bandwidth, but latency.

On 14-02-2013 10:21, Bogdan Flueras wrote:
> What I have is:
>
> Network bandwidth between nodes: 18.4 MB/sec
> Network cards: 1.000Mbps full duplex
> Riak process: IO: avg 600 K/s for disk write, 7% RAM used, 17% CPU used
>
>
> ing. Bogdan Flueras
>
>
>
> On Thu, Feb 14, 2013 at 10:17 AM, Erik Søe Sørensen <ess at trifork.com 
> <mailto:ess at trifork.com>> wrote:
>
>     Just to rule something out: what is the capacity of the network
>     between the cluster and the client - and among the cluster nodes?
>
>
>     -------- Oprindelig meddelelse --------
>     Fra: Shuhao <shuhao at shuhaowu.com <mailto:shuhao at shuhaowu.com>>
>     Dato:
>     Til: riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>     Emne: Re: Tune Riak for fast inserts - populate DB
>
>
>     Hi,
>
>     I've had similar experiences with riak. I used the Riak Python client
>     with HTTP and PBC (even rolled my own client! That only had a ~10%
>     speed
>     up though, and it was HTTP only). I've also examined that the Riak
>     host
>     machine's CPU is only at around 70% when I insert. Other NoSQL dbs
>     uses
>     about 100% when I just continuous insert. Some DBs even have bulk
>     insert.
>
>     Maybe this is an issue with Riak itself... I don't really know.
>     Maybe a
>     Riak dev could chip in on the conversation
>
>     Cheers,
>
>     Shuhao
>
>     On 13-02-13 02:37 AM, Bogdan Flueras wrote:
>     > Hello all,
>     > I've got a 5 node cluster with Riak 1.2.1, all machines are
>     multicore,
>     > with min 4GB RAM.
>     >
>     > I want to insert something like 50 million records in Riak with
>     the java
>     > client (Protobuf used) with default settings.  I've tried also
>     with HTTP
>     > protocol and set w = 1 but got some problems.
>     >
>     > However the process is very slow: it doesn't write more than
>     6GB/ hour or
>     > aprox. 280 KB/second.
>     > To have all my data filled in, it would take aprox 2 days !!
>     >
>     > What can I do to have the data filled into Riak ASAP?
>     > How should I configure the cluster ? (vm.args/ app.config) I
>     don't care so
>     > much about consistency at this point.
>     >
>     > Thank you,
>     > ing. Bogdan Flueras
>     >
>

-- 
Mobile: + 45 26 36 17 55 | Skype: eriksoesorensen | Twitter: @eriksoe
Trifork A/S  |  Margrethepladsen 4  |  DK-8000 Aarhus C | 
www.trifork.com <http://www.trifork.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130214/c4d4c9d2/attachment.html>


More information about the riak-users mailing list