Need help getting riak started

Shawn Debnath shawn at debnath.net
Thu Apr 16 12:03:46 EDT 2015


Turns out it was the nodename that was the culprit. The suffix part of the nodename, @cluster.example.com, needs to be resolvable to that specific host for ping to work. So had to switch to node at node.cluster.example.com. Though it looks strange, at least the service boots up properly.

Attempted to get it to work by having cluster.example.com return  a set of IPs for all the riak nodes in the cluster, but that didn’t help either. In the perfect world, this should.

Thanks to Alex, Zeeshan, and Sean for helping out on IRC.

On 4/15/15, 11:28 PM, "Shawn Debnath" <shawn at debnath.net<mailto:shawn at debnath.net>> wrote:

Just to add to this, seems like even after waiting for 15 mins, the start script fails because it is unable to ping the node. Tracing through the scripts, we arrive at the nodetool in /usr/lib/riak/erts-5.10.3/bin  which is calling {net_kernel:hidden_connect_node(TargetNode), net_adm:ping(TargetNode)} to check the status of the node and is getting {_, pang} in return. Results in printing out "Node 'riak01 at riak.prod.pryvy.com<mailto:'riak01 at riak.prod.pryvy.com>' not responding to pings." At this stage riak console is still unable to connect and still nothing in error or crash logs.  Any hints on how to proceed, or ways to collect more debug info appreciated.

On 4/15/15, 7:39 PM, "Shawn Debnath" <shawn at debnath.net<mailto:shawn at debnath.net>> wrote:

System: Ubuntu 14.04.2 LTS.

From a different vm (10.IP.IP.IP2) after exposing 8087 on 10.IP.IP.IP1:
1> {ok, Pid} = riakc_pb_socket:start(“10.IP.IP.IP1", 8087).
{ok,<0.35.0>}
2> riakc_pb_socket:is_connected(Pid).
true
3>


On 4/15/15, 5:53 PM, "Jon Meredith" <jmeredith at basho.com<mailto:jmeredith at basho.com>> wrote:

What distribution/ version of Linux are you running on the VM Shawn?

On Wed, Apr 15, 2015 at 5:11 PM, Shawn Debnath <shawn at debnath.net<mailto:shawn at debnath.net>> wrote:
Hi there,

Building out a new cluster (for the first time) and package cloud pushed down 2.1.0-1. I have gone through and installed all the necessary packages, configured riak through riak.conf and attempting to start the first node in the cluster. Unfortunately, even though the processes are running, riak-admin reports that no nodes are running.

root at riak-01:/etc/riak# riak-admin diag
Node is not running!
root at riak-01:/etc/riak# riak-admin status
Node is not running!

12538 ?        S      0:00 /usr/lib/riak/erts-5.10.3/bin/epmd -daemon
13799 ?        S      0:00 /usr/lib/riak/erts-5.10.3/bin/run_erl -daemon /tmp/riak// /var/log/riak exec /usr/sbin/riak console
13802 pts/2    Ssl+   0:26 /usr/lib/riak/erts-5.10.3/bin/beam.smp -scl false -sfwi 500 -P 256000 -e 256000 -Q 65536 -A 64 -K true -W w -zdbbl 32768 -- -root /usr/lib/riak -progname riak -- -home /var/lib/riak -- -boot /usr/lib/riak/releases/2.1.0/riak -config /var/lib/riak/generated.configs/app.2015.04.15.22.17.00.config -setcookie riak -name riak-01 at riak.prod.pryvy.com<mailto:riak-01 at riak.prod.pryvy.com> -smp enable -vm_args /var/lib/riak/generated.configs/vm.2015.04.15.22.17.00.args -pa /usr/lib/riak/lib/basho-patches -- console
14047 ?        Ss     0:00 sh -s disksup
14050 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.13/priv/bin/memsup
14052 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.13/priv/bin/cpu_sup

The config states that it should be listening on:

listener.http.internal = 127.0.0.1:8098<http://127.0.0.1:8098>
##listener.protobuf.internal = 10.IP.IP.IP:8087
listener.protobuf.internal = 127.0.0.1:8087<http://127.0.0.1:8087>

I initially had it listening to our internal network IP, but as part of testing, switched to localhost to see if it resolves it but alas, that’s not the case.

In the log directory, files crash.log and error.log are empty and the console.log reports things are ok:

[snip]
2015-04-15 22:17:03.386 [info] <0.7.0> Application riak_kv started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.402 [info] <0.7.0> Application merge_index started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.406 [info] <0.7.0> Application riak_search started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.421 [info] <0.7.0> Application ibrowse started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.429 [info] <0.7.0> Application yokozuna started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.434 [info] <0.7.0> Application cluster_info started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.443 [info] <0.192.0>@riak_core_capability:process_capability_changes:555 New capability: {riak_control,member_info_version} = v1
2015-04-15 22:17:03.461 [info] <0.7.0> Application riak_control started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.462 [info] <0.7.0> Application erlydtl started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:03.485 [info] <0.7.0> Application riak_auth_mods started on node 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:19.618 [info] <0.369.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:853 Changing AAE throttle from undefined -> 0 msec/key, based on maximum vnode mailbox size 0 from 'riak-01 at riak.prod.pryvy.com<mailto:'riak-01 at riak.prod.pryvy.com>'
2015-04-15 22:17:19.672 [info] <0.352.0>@riak_core:wait_for_service:483 Wait complete for service riak_kv (16 seconds)

Full log @ http://pastebin.com/8hb43MwP

Any help will be greatly appreciated.

Thanks,
Shawn


_______________________________________________
riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
Jon Meredith
Chief Architect
Basho Technologies, Inc.
jmeredith at basho.com<mailto:jmeredith at basho.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150416/c008bc7e/attachment-0002.html>


More information about the riak-users mailing list