innostore performance tuning

Jonathan Langevin jlangevin at loomlearning.com
Tue Aug 30 12:52:47 EDT 2011


So, essentially, creation of new keys/objects is much slower than an update
to already existing keys/objects?
Just want to make sure I'm following your description properly.
*

 <http://www.loomlearning.com/>
 Jonathan Langevin
Systems Administrator
Loom Inc.
Wilmington, NC: (910) 241-0433 - jlangevin at loomlearning.com -
www.loomlearning.com - Skype: intel352
*


On Tue, Aug 30, 2011 at 12:30 PM, Kresten Krab Thorup <krab at trifork.com>wrote:

> If you can do the inserts in sorted (ascending) key order, then innostore
> will be significantly faster.
>
> Kresten
>
>
> Mobile: + 45 2343 4626 | Skype: krestenkrabthorup | Twitter: @drkrab
> Trifork A/S   |  Margrethepladsen 4  | DK- 8000 Aarhus C |  Phone : +45
> 8732 8787  |  www.trifork.com
>
> Trifork organizes the world class conference on software development: GOTO
> Aarhus - check it out!
>
>
>
> On Aug 30, 2011, at 6:14 PM, David Koblas wrote:
>
> > I'm currently working on importing a very large dataset (800M) into Riak
> and running into some serious performance problems.  Hopefully this is just
> configuration issues and nothing deeper...
> >
> > Hardware -
> >   * 8 proc box
> >   * 32 Gb ram
> >   * 5TB disk - RAID10
> >
> > Have a cluster of 4 for these boxes all running riak - riak configuration
> options that are different from stock:
> >
> >   * Listening on all IP address "0.0.0.0"
> >   * {storage_backend, riak_kv_innostore_backend},
> >   * innostore section - {buffer_pool_size, 17179869184}, %% 16GB
> >   * innostore section - {flush_method, "O_DIRECT"}
> >
> > What I see is that the performance of my import script runs at about
> 200...300 keys per/second for keys that it's seen recently (e.g. re-runs)
> then drops to 20ish keys per/sec for new keys.
> > STATS: 1000 keys handled in 3 seconds 250.75 keys/sec
> > STATS: 1000 keys handled in 3 seconds 258.20 keys/sec
> > STATS: 1000 keys handled in 4 seconds 240.11 keys/sec
> > STATS: 1000 keys handled in 5 seconds 177.63 keys/sec
> > STATS: 1000 keys handled in 4 seconds 246.26 keys/sec
> > STATS: 1000 keys handled in 5 seconds 184.79 keys/sec
> > STATS: 1000 keys handled in 5 seconds 195.95 keys/sec
> > STATS: 1000 keys handled in 47 seconds 21.02 keys/sec
> > STATS: 1000 keys handled in 44 seconds 22.63 keys/sec
> > STATS: 1000 keys handled in 42 seconds 23.64 keys/sec
> > STATS: 1000 keys handled in 43 seconds 22.88 keys/sec
> > STATS: 1000 keys handled in 45 seconds 22.12 keys/sec
> > STATS: 1000 keys handled in 43 seconds 22.83 keys/sec
> > STATS: 1000 keys handled in 43 seconds 23.11 keys/sec
> > Of course with 800M records to import a performance of 20 keys/sec is not
> useful, plus as time goes on having an insert rate at that level is going to
> be problematic.
> >
> > Questions -
> >   Is there additional things to change for imports and datasets on this
> scale?
> >   Is there a way to get additional debugging to see where the performance
> issues are?
> >
> > Thanks,
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110830/493bad2e/attachment.html>


More information about the riak-users mailing list