Connection Pool with Erlang PB Client Necessary?

Bryan O'Sullivan bos at mailrank.com
Tue Jul 26 12:39:43 EDT 2011


On Mon, Jul 25, 2011 at 4:03 PM, Andrew Berman <rexxe98 at gmail.com> wrote:

> I know that this subject has been brought up before, but I'm still
> wondering what the value of a connection pool is with Riak.


It's a big deal:

   - It amortises TCP and PBC connection setup overhead over a number of
   requests, thereby reducing average query latency.
   - It greatly reduces the likelihood that very busy clients and servers
   will run out of limited resources that are effectively invisible, e.g.
   closed TCP connections stuck in TIME_WAIT.

Each of the above is a pretty big deal. Of course, connection pooling isn't
free.

   - If you have many clients talking to a server sporadically, you may end
   up with large numbers of open-and-idle connections on a server, which will
   both consume resources and increase latency for all other clients. This is
   usually only a problem with a very large number (many thousands) of clients
   per server, and it usually only arises with poorly written and tuned
   connection pooling libraries. But ...
   - ... Most connection pooling libraries are poorly written and tuned, so
   they'll behave pathologically just when you need them not to.
   - Since you don't set up a connection per request, the requests where you
   *do* need to set up a connection are going to be more expensive than those
   where you don't, so you'll see jitter in your latency profile. About 99.9%
   of users will never, ever care about this.

Since Erlang processes are so small and fast to
> create, is there really any overhead in having the gen_server create a
> new connection (with the same client id) each time it needs to access
> Riak?
>

Of course. The overhead of Erlang processes has nothing to do with the cost
of setting up a connection.

Also, you really don't want to be using the same client ID repeatedly across
different connections. That's an awesome way to cause bugs with vclock
resolution that end up being very very hard to diagnose.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110726/bcc01b10/attachment.html>


More information about the riak-users mailing list