Tuning a Riak cluster.

Kevin Burton rkevinburton at charter.net
Tue Feb 26 09:06:46 EST 2013


Here is the config that I am using:

 

{driver,

    basho_bench_driver_2i}.

 

{operations, [

    {get_pb, 1},

    {{put_pb, 5}, 20},

    {{query_http, 10}, 1},

    {{query_mr,   10}, 1},

    {{query_pb,   10}, 1}

]}.

 

{measurement_driver,

    basho_bench_measurement_erlangvm}.

 

{measurements, [

    {memory, 1000},

    {cpu, 1000},

    {processes, 1000},

    {filehandles, 1000}

]}.

 

%%% LOAD SETTINGS %%%

 

{mode, max}.

{duration, 1}.

{concurrent, 3}.

 

%%% DATA SHAPE %%%

 

{key_generator, {uniform_int, 1000}}.

{value_generator, {fixed_bin, 1000}}.

 

{pb_ips, ["172.16.33.107", "172.16.33.138", "172.16.33.221",
"172.16.34.56"]}.

{pb_replies, 1}.

 

{http_ips, ["172.16.33.107", "172.16.33.138", "172.16.33.221",
"172.16.34.56"]}.

{http_port, 8098}.

 

%%It can also be configured for devrel!

%%{pb_ips, [{"127.0.0.1", 8081},

%%          {"127.0.0.1", 8082},

%%          {"127.0.0.1", 8083},

%%          {"127.0.0.1", 8084}

%%]}.

%%{pb_replies, 1}.

%%

%%{http_ips, [{"127.0.0.1", 8091},

%%            {"127.0.0.1", 8092},

%%            {"127.0.0.1", 8093},

%%            {"127.0.0.1", 8094}

%%]}.

 

{rng_seed, {1, 2, 3}}.

 

%% enforce_keyrange is for use with sequential_int

%% Will error if keys in a range are missing.

%%{enforce_keyrange, 10000}.

 

%%% MEASUREMENT SETTINGS %%%

 

{nodes, ['riak at 172.16.33.107', 

         'riak at 172.16.33.138', 

         'riak at 172.16.33.221',

         'riak at 172.16.34.56']}.

{cookie, riak}.

 

From: Richard Shaw [mailto:richard at basho.com] 
Sent: Tuesday, February 26, 2013 5:46 AM
To: Kevin Burton
Cc: riak-users
Subject: Re: Tuning a Riak cluster.

 

Hi Kevin,

 

Yes I have, please can you let me know which basho bench config you're using
and its contents

 

Thanks

 

Richard




 

Richard Shaw

Client Services Engineer EMEA
Basho Technologies
+44 (0)7590 507683
 <mailto:richard at basho.com> richard at basho.com

 

On Mon, Feb 25, 2013 at 10:10 PM, Kevin Burton <rkevinburton at charter.net>
wrote:

Did you have a chance to look at the basho_bench errors?


On Feb 23, 2013, at 2:35 AM, Richard Shaw <richard at basho.com> wrote:

 

Hi Kevin,

 

I sympathise with R, i'm going to write an installation guide to go on the
site.

 

In regards to the error, please paste in the contents of you vm.args file
from the target node

 

Thanks

 

Richard

 

On Fri, Feb 22, 2013 at 11:05 PM, <rkevinburton at charter.net> wrote:

Here is the output from one configuration try:

 

17:00:08.878 [info] Using http target "172.16.33.138":8098 for worker 2

 

17:00:08.878 [info] Using pb target "172.16.33.138":8087 for worker 2

 

17:00:08.993 [debug] Supervisor basho_bench_sup started
basho_bench_worker:start_link(basho_bench_worker_2, 2) at pid <0.96.0>

 

17:00:08.999 [info] Using http target "172.16.33.107":8098 for worker 3

 

17:00:09.001 [info] Using pb target "172.16.33.107":8087 for worker 3

 

17:00:09.006 [debug] Supervisor basho_bench_sup started
basho_bench_worker:start_link(basho_bench_worker_3, 3) at pid <0.99.0>

 

17:00:09.022 [debug] Supervisor net_sup started erl_epmd:start_link() at pid
<0.104.0>

 

17:00:09.029 [debug] Supervisor net_sup started auth:start_link() at pid
<0.105.0>

 

17:00:09.030 [info] Can't set long node name!

Please check your configuration

 

17:00:09.030 [error] Failed to start net_kernel for
basho_bench_measurement_erlangvm:
{shutdown,{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[[ba
sho_bench,longnames]]},permanent,1000,supervisor,[erl_distribution]}}

 

There is alot more output but this is the tail end of the output.

 

On Fri, Feb 22, 2013 at 4:34 PM, Richard Shaw wrote:

 

 Hi Kevin, 

 

Always good practise to reply to the mailing list so others can benefit from
the content 

 

basho bench is the best place to start, If you have a working cluster or
even a working single node, follow the benchmarking guide[0]  

 

Please let me see some of the specific errors you're having and we can help
resolve them. 

 

Outside of basho bench, I recommend benchmarking the raw performance of your
VM, you can do this using a variety of tools like iostat, iperf or a test
suite like Phoronix test suite, a very comprehensive FOSS tool[1] look at
the `aio` test 

 

I'd also recommend benchmarking each time you make a configuration change to
really understand how effective that change was 

 

[0] http://docs.basho.com/riak/latest/cookbooks/Benchmarking/

[1] http://www.phoronix-test-suite.com/

 

Kind Regards 

 

Richard 

 

On Fri, Feb 22, 2013 at 10:14 PM, Kevin Burton < rkevinburton at charter.net>
wrote: 

That was my next question. I have tried to get basho_bench to work but so
far have been unsuccessful. If you have a simple "hello world" sample config
I would greatly appreciate it. I have tried a bunch of configs in the
examples directory but get met with one error or another.  

Other than that I have just timed various client driver calls. Mostly random
reads some writes. 

 

On Feb 22, 2013, at 3:48 PM, Richard Shaw < richard at basho.com> wrote: 

 

Hi Kevin, 

What kind of benchmarking have you done on your VMs and what did you use ? 

 

Kind Regards 

 

Richard 

 

On Fri, Feb 22, 2013 at 9:49 AM, Kevin Burton < rkevinburton at charter.net>
wrote: 

Ok I suspected as much. This will give me more ammo when requesting more
resources. Thank you. 

Now, back to the original question. Given that the physical hardware is
taken care of. What parameters are most important when tuning a cluster.
Again for arguments sake assume the same 4 node cluster with a read-write
ratio of about 75 percent. 

 

On Feb 21, 2013, at 8:26 PM, Alexander Sicular < siculars at gmail.com> wrote: 

 

It can't be said enough times but the number one thing you can do to ensure
that you are getting true performance (not to mention redundancy) is to use
different physical hardware for each of your nodes. Under no circumstances
should you have more than one VM (one logical node in a Riak cluster) on the
same physical hardware. Also, use multiple
connections/threads/parallelism/whatever on client side and be sure to hit
all the nodes in the cluster haproxy roundrobin-esque when writing to Riak.
Everything else is in the noise. 

 

-Alexander Sicular 

 

@siculars 

 

On Feb 21, 2013, at 9:04 PM, Kevin Burton < rkevinburton at charter.net> wrote:


 

  

There each has about 20-30GB of disk space.  They each are a VM so I am not
sure how to specify the CPU. They all seem to be 64 bit Intel processors but
I could tell you the clock speed. The network is 1Gb Ethernet. 

  

From:   Sean Carey [mailto: carey at basho.com]   

Sent:   Thursday, February 21, 2013 7:59 PM 

To:   Kevin Burton 

Cc:   riak-users at lists.basho.com

Subject:   Re: Tuning a Riak cluster. 

  

Kevin, 

Disk and CPU, and Network? 

  

 

Sean Carey 

@densone 

On Thursday, February 21, 2013 at 20:31, Kevin Burton wrote: 

  

I have a cluster of 4 machines (4 Linux VM machines each allocated about 1
Gb of memory - yea I know it isn't a lot). I would like to get some pointers
on getting the fastest query time possible given these meager resources.
Thank you. 

_______________________________________________ 

riak-users mailing list 

riak-users at lists.basho.com

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

  

_______________________________________________ 

riak-users mailing list 

riak-users at lists.basho.com

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

 

_______________________________________________ 

riak-users mailing list 

riak-users at lists.basho.com

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130226/31e9fe6a/attachment.html>


More information about the riak-users mailing list