Store() Performance

Gui Pinto gpinto at chitika.com
Fri Apr 8 12:23:13 EDT 2011


Hey Everyone, thanks for all of the recommendations.

I've tried importing using the example load_data
script<http://wiki.basho.com/Loading-Data-and-Running-MapReduce-Queries.html>available
on the Fast Track, and have last tried the PHP library.

Both of these execute a straight-foward CURL -X PUT request.. which makes me
think Mark might have just guessed it..
Keep-alive not being used definitely explains the 200-writes/second cap.

I'm going to take a look into the PHP library and test this theory.

Gui Pinto
Software Engineer at Chitika


On Fri, Apr 8, 2011 at 10:01 AM, Mark Steele <msteele at beringmedia.com>wrote:

> If using HTTP, make sure you're using keep-alives. That will be a gigantic
> speed boost.
>
> The protocol buffer API is much faster if you're client language supports
> it.
>
>
> Mark Steele
> Bering Media Inc.
>
>
>
> On Thu, Apr 7, 2011 at 10:58 PM, matthew hawthorne <mhawthorne at gmail.com>wrote:
>
>> Hi Gui,
>>
>> I recently pushed 70 million records of size 1K each into a 5-node
>> Riak cluster (which was replicating to another 5-node cluster) at
>> around 1000 writes/second using basho_bench and the REST interface.  I
>> probably could have pushed it further, but I wanted to confirm that it
>> could maintain the load for the entire data set, which it did.
>>
>> My point being that your speed-limit of 200 writes/second is likely
>> specific to your configuration.
>>
>> I wonder:
>> 1) what's your average write latency?
>> 2) how big is your connection pool?
>>
>> Because it's possible that you don't have enough connections available
>> to handle your desired load.
>>
>> -matt
>>
>>
>> On Thu, Apr 7, 2011 at 6:01 PM, Gui Pinto <gpinto at chitika.com> wrote:
>> > Hey guys,
>> > I'm attempting to importing 300M+ objects into a Riak cluster, but have
>> > quickly reached the REST API's speed-limit at 200-store()'s per second..
>> > At the rate of 200/s, I'm looking at 20-days to import this data set!
>> That
>> > can't be the fastest method to do this..
>> >
>> > Any recommendations?
>> >
>> > Thanks!
>> > Gui Pinto
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > riak-users at lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>> >
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110408/efdc1f12/attachment.html>


More information about the riak-users mailing list