Tuning a Riak cluster.

rkevinburton at charter.net rkevinburton at charter.net
Fri Feb 22 18:05:01 EST 2013


Here is the output from one configuration try:

17:00:08.878 [info] Using http target "172.16.33.138":8098 for worker 2

17:00:08.878 [info] Using pb target "172.16.33.138":8087 for worker 2

17:00:08.993 [debug] Supervisor basho_bench_sup started 
basho_bench_worker:start_link(basho_bench_worker_2, 2) at pid <0.96.0>

17:00:08.999 [info] Using http target "172.16.33.107":8098 for worker 3

17:00:09.001 [info] Using pb target "172.16.33.107":8087 for worker 3

17:00:09.006 [debug] Supervisor basho_bench_sup started 
basho_bench_worker:start_link(basho_bench_worker_3, 3) at pid <0.99.0>

17:00:09.022 [debug] Supervisor net_sup started erl_epmd:start_link() at 
pid <0.104.0>

17:00:09.029 [debug] Supervisor net_sup started auth:start_link() at pid 
<0.105.0>

17:00:09.030 [info] Can't set long node name!
Please check your configuration

17:00:09.030 [error] Failed to start net_kernel for 
basho_bench_measurement_erlangvm: 
{shutdown,{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[basho_bench,longnames]]},permanent,1000,supervisor,[erl_distribution]}}

There is alot more output but this is the tail end of the output.

On Fri, Feb 22, 2013 at 4:34 PM, Richard Shaw wrote:

  Hi Kevin,

Always good practise to reply to the mailing list so others can benefit 
from the content

basho bench is the best place to start, If you have a working cluster or 
even a working single node, follow the benchmarking guide[0] 

Please let me see some of the specific errors you're having and we can 
help resolve them.

Outside of basho bench, I recommend benchmarking the raw performance of 
your VM, you can do this using a variety of tools like iostat, iperf or 
a test suite like Phoronix test suite, a very comprehensive FOSS tool[1] 
look at the `aio` test

I'd also recommend benchmarking each time you make a configuration 
change to really understand how effective that change was

[0] http://docs.basho.com/riak/latest/cookbooks/Benchmarking/ 
<http://docs.basho.com/riak/latest/cookbooks/Benchmarking/> 
<http://docs.basho.com/riak/latest/cookbooks/Benchmarking/>
[1] http://www.phoronix-test-suite.com/ 
<http://www.phoronix-test-suite.com/> 
<http://www.phoronix-test-suite.com/>

  <http://www.phoronix-test-suite.com/>
Kind Regards  <http://www.phoronix-test-suite.com/>

  <http://www.phoronix-test-suite.com/>
Richard  <http://www.phoronix-test-suite.com/>

On Fri, Feb 22, 2013 at 10:14 PM, Kevin Burton < 
rkevinburton at charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')> 
> wrote:
That was my next question. I have tried to get basho_bench to work but 
so far have been unsuccessful. If you have a simple "hello world" sample 
config I would greatly appreciate it. I have tried a bunch of configs in 
the examples directory but get met with one error or another.  
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>
Other than that I have just timed various client driver calls. Mostly 
random reads some writes. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>

On Feb 22, 2013, at 3:48 PM, Richard Shaw < richard at basho.com 
<javascript:parent.wgMail.openComposeWindow('richard at basho.com')> > 
wrote:

  <javascript:parent.wgMail.openComposeWindow('richard at basho.com')>
Hi Kevin,
  <javascript:parent.wgMail.openComposeWindow('richard at basho.com')>
What kind of benchmarking have you done on your VMs and what did you use 
?  <javascript:parent.wgMail.openComposeWindow('richard at basho.com')>

  <javascript:parent.wgMail.openComposeWindow('richard at basho.com')>
Kind Regards 
<javascript:parent.wgMail.openComposeWindow('richard at basho.com')>

  <javascript:parent.wgMail.openComposeWindow('richard at basho.com')>
Richard 
<javascript:parent.wgMail.openComposeWindow('richard at basho.com')>

On Fri, Feb 22, 2013 at 9:49 AM, Kevin Burton < rkevinburton at charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')> 
> wrote:
Ok I suspected as much. This will give me more ammo when requesting more 
resources. Thank you. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>
Now, back to the original question. Given that the physical hardware is 
taken care of. What parameters are most important when tuning a cluster. 
Again for arguments sake assume the same 4 node cluster with a 
read-write ratio of about 75 percent. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>

On Feb 21, 2013, at 8:26 PM, Alexander Sicular < siculars at gmail.com 
<javascript:parent.wgMail.openComposeWindow('siculars at gmail.com')> > 
wrote:

  <javascript:parent.wgMail.openComposeWindow('siculars at gmail.com')>
It can't be said enough times but the number one thing you can do to 
ensure that you are getting true performance (not to mention redundancy) 
is to use different physical hardware for each of your nodes. Under no 
circumstances should you have more than one VM (one logical node in a 
Riak cluster) on the same physical hardware. Also, use multiple 
connections/threads/parallelism/whatever on client side and be sure to 
hit all the nodes in the cluster haproxy roundrobin-esque when writing 
to Riak. Everything else is in the noise.

-Alexander Sicular 
<javascript:parent.wgMail.openComposeWindow('siculars at gmail.com')>

  <javascript:parent.wgMail.openComposeWindow('siculars at gmail.com')>
@siculars 
<javascript:parent.wgMail.openComposeWindow('siculars at gmail.com')>

On Feb 21, 2013, at 9:04 PM, Kevin Burton < rkevinburton at charter.net 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')> 
> wrote: 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>

  
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>
There each has about 20-30GB of disk space.  They each are a VM so I am 
not sure how to specify the CPU. They all seem to be 64 bit Intel 
processors but I could tell you the clock speed. The network is 1Gb 
Ethernet. 
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>
  
<javascript:parent.wgMail.openComposeWindow('rkevinburton at charter.net')>
From:   Sean Carey [mailto: carey@ 
<javascript:parent.wgMail.openComposeWindow('carey@')> basho.com 
<http://basho.com> ]  
Sent:   Thursday, February 21, 2013 7:59 PM
To:   Kevin Burton
Cc:   riak-users at lists.basho.com 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
Subject:   Re: Tuning a Riak cluster. 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
  
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
Kevin, 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
Disk and CPU, and Network? 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
  
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>

Sean Carey
@densone 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
On Thursday, February 21, 2013 at 20:31, Kevin Burton wrote: 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
  
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
I have a cluster of 4 machines (4 Linux VM machines each allocated about 
1 Gb of memory – yea I know it isn’t a lot). I would like to get some 
pointers on getting the fastest query time possible given these meager 
resources. Thank you. 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
_______________________________________________ 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
riak-users mailing list 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
riak-users at lists.basho.com 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')> 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com> 
<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
   <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
_______________________________________________
riak-users mailing list
riak-users at lists.basho.com 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>

_______________________________________________
riak-users mailing list
riak-users at lists.basho.com 
<javascript:parent.wgMail.openComposeWindow('riak-users at lists.basho.com')>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130222/1bfee04a/attachment.html>


More information about the riak-users mailing list