tuning riak for 100s of concurrent connections

Les Mikesell lesmikesell at gmail.com
Fri Feb 25 13:41:41 EST 2011


Those settings shouldn't make a big difference in how the number of 
connections scale up, though.  There is a theoretical maximum rate limit 
for creating new connections as each socket is supposed to sit in 
TIME_WAIT for a packet round-trip time to ensure that nothing 
outstanding will collide with that socket number when it is reused for 
the same IP address.  Maybe your test is hitting that limit.  Can you 
set SO_REUSEADDR?


On 2/25/2011 12:25 PM, Sean Cribbs wrote:
> Now you're talking about tweaking kernel-level settings -- something well outside my expertise (although TCP slow-start comes to mind).  That said, it's kind of hard not to use TCP if you want a reliable connection, warts and all.
>
> Sean Cribbs<sean at basho.com>
> Developer Advocate
> Basho Technologies, Inc.
> http://basho.com/
>
> On Feb 25, 2011, at 1:06 PM, Wilson MacGyver wrote:
>
>> any advise to solve this?
>>
>> On Fri, Feb 25, 2011 at 1:05 PM, Sean Cribbs<sean at basho.com>  wrote:
>>> Yes, the majority of the cost is probably in TCP setup. That pain is happening at the TCP stack level, not in Erlang.  It's actually really cheap and easy to spawn a new process in Erlang, so there's no reason to have "standby workers", especially when they don't need to maintain state between requests.
>>>
>>> Sean Cribbs<sean at basho.com>
>>> Developer Advocate
>>> Basho Technologies, Inc.
>>> http://basho.com/
>>>
>>> On Feb 25, 2011, at 12:45 PM, Wilson MacGyver wrote:
>>>
>>>> I've tried it, it didn't have much impact. A bit more info on how I'm
>>>> doing the test.
>>>>
>>>> I'm using apachebench. I purposely have it grab the same key/value over and over
>>>> again.
>>>>
>>>> if I use concurrent connection of 10, 1000 requests each. 50% of the
>>>> request complets
>>>> within 7ms, longest request is 38ms. this is quite good.
>>>>
>>>> now, if I increase the concurrent connection to 100, 50% of the
>>>> request complets at 77ms.
>>>>
>>>> it "feels" like a paying for startup cost of connection problem to me.
>>>> Is there a way to purposely
>>>> startup riak with a bunch of standby workers, or something to that effect?
>>>>
>>>> On Fri, Feb 25, 2011 at 11:37 AM, Sean Cribbs<sean at basho.com>  wrote:
>>>>> You can disable Nagle on the riak side (at least on 0.14 and later). Put this in the riak_core section of app.config:
>>>>>
>>>>> {disable_http_nagle, true}
>>>>>
>>>>> Sean Cribbs<sean at basho.com>
>>>>> Developer Advocate
>>>>> Basho Technologies, Inc.
>>>>> http://basho.com/
>>>>>
>>>>> On Feb 25, 2011, at 9:33 AM, Wilson MacGyver wrote:
>>>>>
>>>>>> TCP_NODELAY is something you'd set when you use the socket API call,
>>>>>> not a global tuning setting on the OS as I recall.
>>>>>>
>>>>>> On Fri, Feb 25, 2011 at 9:28 AM, Nico Meyer<nico.meyer at adition.com>  wrote:
>>>>>>> Whenever I see latencies which are roughly multiples of 40ms it screams
>>>>>>> to me 'nagle algorithm'. I have seen this so often now, that the first
>>>>>>> thing I check is, if the TCP_NODELAY option is set on the TCP socket on
>>>>>>> both ends.
>>>>>>
>>>>
>>>> --
>>>> Omnem crede diem tibi diluxisse supremum.
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>>
>> --
>> Omnem crede diem tibi diluxisse supremum.
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list