riak_core nodes on Heroku?

Ciprian Dorin Craciun ciprian.craciun at gmail.com
Sun Jun 17 07:47:05 EDT 2012


On Thu, Jun 14, 2012 at 4:37 PM, Samuel Elliott <sam at lenary.co.uk> wrote:
> It's actually very much not possible.
>
> Erlang nodes need to listen on one port for the EPMD, and then another
> for HTTP (and maybe more for other coordination). Heroku only lets you
> listen on a single node per process, and you can't guarantee scaled
> processes occur on the same dyno or even machine to another, so you
> can't explicitly run epmd separately.

    Although not strictly related to Heroku's dyno's (for which I
don't have too much "in-depth" knowledge), in a project I'm currently
working, that provides a similar environment with Heroku, a major
problem in running distributed Erlang nodes (epecially Riak), is the
EPMD process. (From my point of view this is quite a serious
limitation for Erlang and the OTP...)

    The problem is as follows (I make one assertion presented below,
but which I'm certain is FALSE, thus this paragraph is solely for
"theoretical" purposes):
    * all the Erlang nodes in a distributed application must use the
same EPMD port (as specified by using `ERL_EPMD_PORT`);
    * now if Heroku provides you with an unique IP per dyno then this
is no problem, as each Erlang node dyno could run his own `epmd`
process;
    * if Heroku doesn't provide an uinque IP per dyno then "game
over"; (although it could seem it works, but if two dynos of your own
run on the same machine, then when one dies, the other remains
"invisible" from further discoveries;)

    The assertion I've made is that Heroku doesn't firewall
communication on private IP's between arbitrary dyno's. WHICH I DON'T
THINK IS THE CASE, as I would be an important security issue...


> Yes, distribution via other means seems sensible.

    As Sam said, you could however solve these issues by:
    * providing your own implementation of an `erl_epmd` solver;
    * and your own implementation of an `inet_tcp_dist`;
    * both using as communication an HTTP module (like `mochiweb` or
`misultin`) multiplexing between the two;

    But this road is very hard... If interested you could read one of
my posts to the Erlang list exactly about this topic, but which
remained unanswered...
      http://erlang.org/pipermail/erlang-questions/2011-October/062004.html

    (One ideea: use ZeroMQ for communication, but again all this goes
if the previous assertion is true.)

    (Another one: use RabbitMQ, to provide transport for such modules.
You could use direct exchanges for `inet_amqp_dist` and you wouldn't
need anymore the `erl_epmd` resolver.)


> As for the routing mesh issues, they seem to have support for
> websockets etc in the proxy, so presumably it is possible somehow to
> route TCP messages through.

    I have the feeling that Heroku (on their Cedar) stack use the
`mochiweb` Erlang web-server to provide the routing... As such you
could try to find "undocumented" ways to fool mochiweb into routing
requests to the respective dyno.

    Hope this helps,
    Ciprian.



More information about the riak-users mailing list