christopher.meiklejohn at gmail.com
Wed Jul 13 16:22:52 EDT 2016
On Wed, Jul 13, 2016 at 10:08 PM, Jon Brisbin <jbrisbin at basho.com> wrote:
> I guess to answer at least one part of your question on whether it's
> "encouraged" or not: it depends on what you want to do with it.
> I can't speak for the support side of things but on the engineering side
> anyway we're using Docker containers fairly happily to do things like create
> ephemeral clusters for doing adhoc testing. It makes for a very nice way to
> run a small cluster on a laptop or on a Mesos slave in a Jenkins build with
> very similar expectations (I won't say identical because networking issues
> can rear their head). I created images based on the latest TS and KV
> packages. There are docs in the READMEs:
> Running a production Riak cluster, however, raises some concerns that I'm
> pretty sure no one really has a good handle on. Networking issues can become
> difficult unless you use something like Weave. Volumes could also be an
> issue. docker-compose and Docker Swarm should help with some of this but we
> don't have any docs that demonstrate how to use them.
I'd say there's two classes of issues here:
Most of the networking issues arise from the use of distributed
Erlang: the Erlang port mapper deamon requires that you operate in
host networking mode, on a known port visible to all other nodes,
because of the way node discovery and clustering works in Erlang.
Outside of that, if you're running multiple containers on the same
machine, you'll also need to modify Riak to listen on different ports
for the Protocol Buffer and HTTP interfaces, because under host
networking only one server is going to be able to bind.
The second is the lifecycle around the containers: where is the
persistent storage going to be for the cluster? How will cluster
orchestrations in Riak be handled? Mesos, in it's current iteration
does not have a mechanism for issuing a callback when a container
terminates, so an explicit operator command is still required to force
remove a node from the cluster if a container is to die. How will
nodes be added to the cluster when a new container is launched? Will
an orchestration framework such as Mesos or Kubernetes be used? The
operational side is very unclear at this point.
> I'd say we also don't have a comprehensive picture of performance
> characteristics of running a Dockerized Riak cluster since there's several
> moving parts to consider. We need to investigate bridged vs host networking
> mode and using different volume drivers, etc... There is some valid concern
> that the key reasons you'd want to use Riak to begin with are somewhat
> ameliorated by a Docker container's ephemeral nature; e.g. it's really easy
> to kill off containers which might lead to accidental data loss.
As someone who's been doing a significant amount of Erlang in Docker,
I'd also be worried about the oom-killer and cgroups with Docker
containers if one is considering Mesos: you don't want to lose a node
and risk data loss if your node happens to have a memory fluctuation.
I've personally made some headway in this for my own research when
migrating some of our application off of Riak Core and it's involved
finding solutions for many of these things: it's lead us down the path
of completely disabling distributed Erlang and doing explicit TCP
connection handling between nodes; using a custom membership protocol;
and writing a system for container lifecycle management tied into
membership information being exported from Kubernetes/Mesos (and, I'm
sure similar could be done for swarm, weave, etc.)
More information about the riak-users