Riak behind a Load Balancer

Swinney, Austin Austin at vimeo.com
Mon Jun 25 14:41:15 EDT 2012


I use the Amazon Elastic Load Balancer (ELB) on my ec2 riak cluster.  I understand the concerns of LB fail, but for me, using Riak is largely about ease of use.  Easy to deploy, grow, shrink, replace, etc.  Using an the ELB allows me to do those things without consulting those who manage the services that use cluster.  And in the turbulent cloud space, if a node goes unresponsive, it just gets kicked out without any action on my part.

A few weeks ago, I had some episode where I replaced all the nodes in the riak cluster.  It sucked for me (because I don't know what I'm doing!), but the service owner who actually uses the cluster came to my desk a week later and asked, "So how is everything going with the riak cluster?"  He hadn't even noticed, nor had anyone else unless I told them about it.  To me, that is pure ops gold.

Performance wise, I get what I need out of it.  Response times through the ELB are all pretty similar across the board.  I never tested it without the ELB, so I am not aware of any performance hit using ELB.

My ELB health check settings:

Ping Target:
HTTP:8098/ping

Timeout:
5 seconds

Interval:
30 seconds

Unhealthy Threshold:
2

Healthy Threshold:
10



If I didn't have ELB, I'd look at the zookeeper route.

On Jun 25, 2012, at 2:21 PM, Anthony Molinaro wrote:

This is almost exactly what we do with our riak clusters (as well as actually
all our subclusters).  It's actually nice for developer because when they
need to talk to a network service they just reference 127.0.0.1 and some
port, and haproxy gets them to the right place.  This also allows an
operations team to make network changes without impacting applications
(via the use of the haproxy reload).

The only downside is that if your connections are long lived and you restart
your riak nodes, all requests will end up on the last node.  And if you restart
haproxy they will all end up on the first node initially.  I sort of wish
there was a configuration parameter to haproxy to randomize the start
point for the roundrobin so you could keep things a little more balanced
across many machines running haproxy.

-Anthony

On Mon, Jun 25, 2012 at 07:44:05AM -0400, Sean Cribbs wrote:
Another typical setup is to have each client node have its own haproxy, and
when Riak nodes are added or removed (not a common occurrence, mind you), a
configuration management tool like Chef/Puppet/cfengine/etc can adjust the
config and signal the process to reload it (I think it's `kill -HUP`). Then
your client code also only ever needs to connect to localhost, and doesn't
have to have itself reconfigured.

On Mon, Jun 25, 2012 at 4:40 AM, Samuel Elliott <sam at lenary.co.uk<mailto:sam at lenary.co.uk>> wrote:

On Mon, Jun 25, 2012 at 7:36 AM, Matt Black <matt.black at jbadigital.com<mailto:matt.black at jbadigital.com>>
wrote:
Dear list,

Does anyone have an opinion on the concept of putting a Riak cluster
behind
a load balancer?

It has been done before. there are various results when searching
"riak haproxy" in your favourite search engine.


We wish to be able to automatically add/remove nodes from the cluster, so
adding an extra layer at the front is desirable. We should also benefit
for
incoming requests behind shared across all nodes.

Can anyone see any drawbacks / problems with doing this?

If your load balancer falls over, what do you do then? Highly
available may go down the pan. Have more than one would be the obvious
answer.

What do you do when you want to transparently add more machines to
your load balancer?

Maybe it might be better to have a list of riak nodes stored in a
separate registry (I'm thinking something like zookeeper), that your
application servers can then poll for changes (or even subscribe to
changes) to the list of servers.

Sam


Thanks
Matt


_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
Samuel Elliott
sam at lenary.co.uk<mailto:sam at lenary.co.uk>
http://lenary.co.uk/
+44 (0)7891 993 664

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
Sean Cribbs <sean at basho.com<mailto:sean at basho.com>>
Software Engineer
Basho Technologies, Inc.
http://basho.com/

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


--
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym at alumni.caltech.edu<mailto:anthonym at alumni.caltech.edu>>

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120625/9fea9eb0/attachment.html>


More information about the riak-users mailing list