Issue with clustering Riak nodes on CentOS servers.

Bryan Hughes bryan at go-factory.net
Wed Oct 31 00:20:14 EDT 2012


Hi James,

Gonna move this back to the list so that others can add to the 
discussion.  Definitely sounds like it may be a firewall issue. You 
should get "Firewall is stopped" if it is actually stopped.

$ sudo service iptables status
Firewall is stopped.

 From your output, it is still running.  Here are some instructions on 
how to permanently turn it off.

http://www.cyberciti.biz/faq/disable-linux-firewall-under-centos-rhel-fedora/

Since it sounds like you are safe in your companies intranet, I would 
suggest trying the above and turn off the firewall completely.

In regards to our install - for a production solution, we build from 
sources in a manner that we can actually have multiple installs of both 
Erlang and Riak (we are 100% Erlang) to allow us to roll back quickly.  
This gives us fine grain control over 1) when we want to upgrade to a 
new release, 2) the mount point for our data (we use bitcask), and 3) 
the ability to roll back to a previous release. This is true for both 
Erlang and Riak.  This also assumes that a RIAK upgrade does not 
transform physical data - which it has never yet as far as I am aware.

Our solution is pretty simple and is based on a blog post by Robert Aloi:

http://aloiroberto.wordpress.com/2010/11/24/how-to-manage-multiple-erlang-installations/

Here are the steps that I follow (sorry, havent made a shell script out 
of it).  Would love to hear any feedback from anyone on the list on 
where we could be doing better.  Following this, I can build a new 5 
node cluster from scratch within an hour.

Configuring Erlang
./configure --prefix=/home/riak/erlang/R15B01 --enable-ssl=yes 
--enable-kernel-poll --enable-hipe --enable-smp-support --enable-threads 
--enable-m64-build

FOR THE MAC:
CFLAGS=-O0 ./configure --enable-hipe --enable-smp-support 
--enable-threads --enable-kernel-poll --enable-darwin-64bit 
--prefix=/opt/erlang/R15B01

As root:
  1. useradd -m -G wheel riak
  2. visudo to enable wheel sudoers
  3. passwd riak <whatever>
  4. su - riak

As riak:
[building erlang]
  5. sudo yum install gcc gcc-c++ glibc-devel make ncurses-devel 
openssl-devel autoconf git

NOTE: If yum can not install git, do the following:
      wget 
http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
      rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm
      yum install --enablerepo=webtatic git-all

  6. mkdir erlang
  7. cd erlang
  8. mkdir R15B01
  9. wget http://erlang.org/download/otp_src_R15B01.tar.gz
10. tar zxvf otp_src_R15B01.tar.gz
11. cd otp_src_R15B01
12. ./configure --prefix=/home/riak/erlang/R15B01 --enable-ssl=yes 
--enable-kernel-poll --enable-hipe --enable-smp-support --enable-threads 
--enable-m64-build
13. make
14. make install
15. cd ~
16. mkdir bin
17. vi .bashrc (add)
RIAK_HOME=$HOME/riak-1.2.0/rel/riak
PATH=$PATH:/$HOME/bin:/sbin:/usr/bin:$RIAK_HOME/bin

18. . .bashrc
19. cd bin
20. vi e15 (add)
#!/bin/bash
env PATH=/home/riak/erlang/R15B01/bin:$PATH "$@"


21. chmod +x e15
to test, type the following on the command line:
e15 erl

22. Do the following. Append these in /etc/sysctl.conf then run sysctl 
-p to apply them. No need to reboot, now your kernel should be able to 
handle a lot more open connections, yay.

# General gigabit tuning:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# this gives the kernel more memory for tcp
# which you need with many (100k+) open socket connections

net.ipv4.tcp_mem = 50576   64768   98152
net.core.netdev_max_backlog = 2500

23. Make sure the file max is high enough
sysctl fs.file-max

[building riak]
24. cd /home/riak
25. wget 
http://downloads.basho.com.s3-website-us-east-1.amazonaws.com/riak/CURRENT/riak-1.2.0.tar.gz
26. tar zxvf riak-1.2.0.tar.gz
27. cd riak-1.2.0
28. e15 make rel
29. ifconfig to get ipaddress

[configuring the firewall (if needed)]
30. Add

## Firewall
{ kernel, [
             {inet_dist_listen_min, 6000},
             {inet_dist_listen_max, 7999}
           ]},

to app.config at top level (same as risk.core)

31. sudo vi /etc/sysconfig/iptables (and add the following to 
RH-Firewall-1-INPUT just above the icmp-host-prohibited)

-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4369 
-j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8099 
-j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 
7010:7015 -j ACCEPT

32. sudo service iptables restart


Cheers,
Bryan

On 10/30/12 8:14 PM, SWEENEY, JAMES wrote:
>
> Hey thanks for the reply Bryan,  I did go through the instructions 
> below.   Plus I had checked the iptables right away after I installed, 
> and iptables does not seem to be running on any of my centos servers  
> ie this is the result I get when running service iptables status --
>
> [root at essd-riak-test-server ~]# sudo service iptables status
>
> Table: filter
>
> Chain INPUT (policy ACCEPT)
>
> num target     prot opt source               destination
>
> Chain FORWARD (policy ACCEPT)
>
> num target     prot opt source               destination
>
> Chain OUTPUT (policy ACCEPT)
>
> num target     prot opt source               destination
>
> [root at essd-riak-test-server ~]#
>
> The type of install I have done is installed from the dev package from 
> the riak site,  not installed from source code.  Does that matter for 
> clustering?   Everything about my install seems fine except that it my 
> cluster commands fail and I can't join nodes for a clustered environment.
>
> The nodes are on a cloud server that is controlled by another group in 
> my company.  We have made sure the ports that riak needs are opened 
> for centos to centos communication, and I can telnet from one centos / 
> riak server to another successfully,  but just can join a node.
>
> *From:*riak-users [mailto:riak-users-bounces at lists.basho.com] *On 
> Behalf Of *Bryan Hughes
> *Sent:* Tuesday, October 30, 2012 5:36 PM
> *To:* riak-users at lists.basho.com
> *Subject:* Re: Issue with clustering Riak nodes on CentOS servers.
>
> Hi James,
>
> We have a 5 node cluster running in production with no problems on 
> CentOS.  We have each node isolated on a private lan.  Our application 
> server has two interfaces, one to the outside world with a very 
> restricted firewall, and the second connected to the private lan with 
> the 5 nodes.
>
> Assuming you set the node names accordingly and followed:
>
> http://docs.basho.com/riak/latest/cookbooks/Basic-Cluster-Setup/
>
> One thing to check would your firewall, which is by default on with 
> CentOS.
>
> >sudo service iptables status
>
> If you have your 4 nodes on a private lan, or perhaps well protected 
> behind a firewall as an internal deployment, then it is relatively 
> straight forward.  You can just turn off your firewall on the 4 machines.
>
> If your machines are not on a isolated private lan and are exposed to 
> the internet, or just need to be protected behind a firewall, you will 
> need to do the following.
>
> http://docs.basho.com/riak/1.2.0/cookbooks/Network-Security-and-Firewall-Configurations/
>
> First, in your app.config, you will need to add the following at the 
> top level (same as riak.core).  You can pick whatever min and max 
> range you want - here I arbitrarily chosen 4 ports:
>
> { kernel, [
>             {inet_dist_listen_min, 7010},
>             {inet_dist_listen_max, 7014}
>           ]},
>
> Next, on each or our riak nodes, you will need to edit your iptables.  
> I tend to edit them manually using sudo.
>
> > sudo vi /etc/sysconfig/iptables (and add the following to 
> RH-Firewall-1-INPUT just above the icmp-host-prohibited)
>
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4369 
> -j ACCEPT
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8087 
> -j ACCEPT
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8099 
> -j ACCEPT
> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 
> 7010:7014 -j ACCEPT
>
> > sudo service iptables restart
>
> After doing this on all the nodes, they should all be visible to each 
> other.  If you wanted to lock down your firewall even further, you can 
> specify a range of IP addresses.
>
> Finally, on the machine that your erlang application or Riak client, 
> you will need to do the following:
>
> sudo vi /etc/sysconf/iptable (and add the following to 
> RH-Firewall-1-INPUT just above the icmp-host-prohibited)
>
> -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8098 
> -j ACCEPT
> -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8087 
> -j ACCEPT
>
> > sudo service iptables restart
>
>
> Hope this helps...
> Bryan
>
> On 10/30/12 11:39 AM, SWEENEY, JAMES wrote:
>
>     Hello,  I am attempting to create a 4 node riak installation on 4 separate CentOS servers.   Installation of Riak went well.  We have run simple tests like riak ping, sudo curl \-vhttp://127.0.0.1:8098/riak/test, ect,  and everything seems fine with the riak servers,  they all start up with no errors.  We used the default ports in the app.config files.   Ie:  epmd listener: TCP:4369
>
>     handoff_port listener: TCP:8099
>
>     web_port: TCP:8098
>
>     pb_port: TCP:8087
>
>       
>
>     We have updated all ip addresses in the config files as instructed on the riak site.  Also, we have verified by telnet that the ports are all open and that I can telnet between the riak servers.    Still after all of that,  when I try to do a cluster command I get the following response:
>
>       
>
>       
>
>     [root at essd-riak-test-server ~]# riak-admin cluster joinriak at 10.1.78.9  <mailto:riak at 10.1.78.9>
>
>     Attempting to restart script through sudo -H -u riak
>
>     Noderiak at 10.1.78.9  <mailto:riak at 10.1.78.9>  is not reachable!
>
>     [root at essd-riak-test-server ~]#
>
>       
>
>     I have been stuck on this issue for quite a while and tried everything I found on the riak web site to resolve this.  Any help you could offer would greatly be appreciated.   Thanks in advance.
>
>       
>
>     Sincerely,
>
>       
>
>     James Sweeney
>
>       
>
>       
>
>       
>
>       
>
>     _______________________________________________
>
>     riak-users mailing list
>
>     riak-users at lists.basho.com  <mailto:riak-users at lists.basho.com>
>
>     http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>     -- 
>
>     Bryan Hughes
>     CTO and Founder / *Go Factory*
>     (415) 515-7916
>     http://www.go-factory.net
>
>     /"Art is never finished, only abandoned. - Leonardo da Vinci"/
>

-- 

Bryan Hughes
CTO and Founder / *Go Factory*
(415) 515-7916
http://www.go-factory.net

/"Art is never finished, only abandoned. - Leonardo da Vinci"/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20121030/c3cc8e04/attachment.html>


More information about the riak-users mailing list