tuning riak for 100s of concurrent connections

Joseph Blomstedt dklayn at gmail.com
Fri Feb 25 15:17:40 EST 2011


On Fri, Feb 25, 2011 at 11:31 AM, Wilson MacGyver <wmacgyver at gmail.com> wrote:
> Another possibility that comes to mind to avoid the setup cost is to
> do connection pool
> of some sorts.
>
> Does any of the riak client lib supports connection pooling?

Connection pooling is going to depend on several factors.

If you're going with REST, I'm not sure of any client libs that do
connection pooling. However, even if a library provided pooling there
is the underlying issue of HTTP keep-alive/persistent connections.
Ripple, for example, will use persistent connections when using the
excon backend, but won't when using the default Ruby HTTP backend.
Although, your overall stack will effect persistent connections
independently from what the client is trying to do. For example, if
you're using HAProxy between your client connections and your riak
cluster, you're subject to the fact that HAProxy doesn't maintain
persistent connections to backend severs (ie. riak) and will create a
new connection for each request. If that's where you bottleneck lies,
then there's not much you can do short of tuning TCP or adding more
riak nodes. On the other hand, if the bottleneck is between the
clients and the HAProxy front-end, then connection pooling along with
a version of HAProxy that supports client-side keep-alive may help.

For protocol buffers there are client libraries that support
connection pooling. Of the top of my head I know riak-js does, and
most of us using Erlang have implemented our own connection pooling
schemes -- several which were linked to on this mailing list a few
weeks back. Other client libraries may as well.

-Joe




More information about the riak-users mailing list