Store() Performance

Mark Steele msteele at beringmedia.com
Fri Apr 8 10:01:36 EDT 2011


If using HTTP, make sure you're using keep-alives. That will be a gigantic
speed boost.

The protocol buffer API is much faster if you're client language supports
it.


Mark Steele
Bering Media Inc.


On Thu, Apr 7, 2011 at 10:58 PM, matthew hawthorne <mhawthorne at gmail.com>wrote:

> Hi Gui,
>
> I recently pushed 70 million records of size 1K each into a 5-node
> Riak cluster (which was replicating to another 5-node cluster) at
> around 1000 writes/second using basho_bench and the REST interface.  I
> probably could have pushed it further, but I wanted to confirm that it
> could maintain the load for the entire data set, which it did.
>
> My point being that your speed-limit of 200 writes/second is likely
> specific to your configuration.
>
> I wonder:
> 1) what's your average write latency?
> 2) how big is your connection pool?
>
> Because it's possible that you don't have enough connections available
> to handle your desired load.
>
> -matt
>
>
> On Thu, Apr 7, 2011 at 6:01 PM, Gui Pinto <gpinto at chitika.com> wrote:
> > Hey guys,
> > I'm attempting to importing 300M+ objects into a Riak cluster, but have
> > quickly reached the REST API's speed-limit at 200-store()'s per second..
> > At the rate of 200/s, I'm looking at 20-days to import this data set!
> That
> > can't be the fastest method to do this..
> >
> > Any recommendations?
> >
> > Thanks!
> > Gui Pinto
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> >
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110408/cb7f8b24/attachment.html>


More information about the riak-users mailing list