50-100 ops/sec on a 5 Node cluster

Yousuf Fauzan yousuffauzan at gmail.com
Fri Jun 29 16:08:23 EDT 2012


Thats a very useful information. Thanks guys.

I figured out the problem was with my data loading script. Plus, instead of
going through ELB, I created a pool of connections for each node.

Now I am getting around 1k writes/sec. I think I could further improve this
my using a more powerful machine for loading data (I am using small
instance of EC2)

Thanks all for your help.

--
Yousuf Fauzan
http://fauzism.com

On Fri, Jun 29, 2012 at 10:25 PM, Mark Rose <markrose at markrose.ca> wrote:

> There are other reasons to avoid using an ELB for this purpose I'd like to
> add.
>
> ELB works by resolving a DNS name to an A record, so if the ELB instance
> goes down or has trouble, you will have trouble as well. This is also the
> source of the scaling up issue as it takes time for the DNS records on the
> client machines to expire and for new A records to be returned.
>
> Also, you have to pay per request, plus the bandwidth used. Not to mention
> the latency of client->elb->server instead of running a proxy on the local
> machine.
>
> Instead, I would recommend setting up HAProxy on every client machine, and
> configure HAProxy to connect to some or all of your nodes. It will react
> much faster to node status, plus you get useful reports and whatnot. It's
> also extremely efficient, using little CPU and memory.
>
> -Mark
>
>
> On Fri, Jun 29, 2012 at 12:28 PM, Sean O'Connor <sean at focuslab.io> wrote:
>
>> There are a few things you probably want to look at in regard to how your
>> using ELB.
>>
>> If your using the straight ELB offering (e.g. not part of a virtual
>> private cloud), that's probably your problem.  This form of ELB was really
>> only designed for routing traffic from outside the AWS network to servers
>> within EC2.  Using it route traffic with the AWS tends to be very
>> slow/throttled as your traffic has to go out to the public network and then
>> back into the private AWS network.
>>
>> Recently (i.e. the last two or three weeks) Amazon announced a new
>> version of ELB that you can use with virtual private cloud to route traffic
>> between VPC networks.  I haven't tried it yet but in theory this should
>> avoid many of the problems mentioned above.
>>
>> One last thing to be aware of with ELB is it can often take a while for
>> an ELB instance to deal with spikes in capacity (e.g. load tests).  Under
>> the covers ELB is basically as series of managed EC2 instances running a
>> load balancer program.  When your capacity exceeds what one instance can
>> handle, they'll spin up more instances but that takes time to happen.
>>  Accordingly when doing load testing with ELB, you often need to "prime"
>> the ELB by running test traffic for 10-15 minutes and then start recording
>> your benchmark data.
>>
>> With the possible exception of the new VPC ELB offering, I would strongly
>> recommend against using ELB to load balance traffic to your riak cluster.
>>
>> --
>> Sean O'Connor
>> CTO/Co-Founder
>> (845)669-0883
>> http://focuslab.io
>>
>> On Thursday, June 28, 2012 at 4:03 PM, Ruslan Bukin wrote:
>>
>> On Tue, Jun 26, 2012 at 04:29:38PM +0530, Yousuf Fauzan wrote:
>>
>> Hello,
>>
>> Here is my setup of Riak 1.1.4
>>
>>  - 5 Nodes
>> - ami-a29943cb (EC2 AMI)
>> - m1.large (7.5GB Ram, 4 EC2 compute instances)
>> - EBS storage
>> - LevelDB backend
>> - Python Client with Protobuf (on a separate machine)
>>
>> While loading data I am getting around 50 ops/sec.
>>
>> I tried running basho benchmark on the setup and it showed 100 ops/sec
>> throughput.
>>
>> After talking to people or Riak IRC, I realized that upwards of 1.5K
>> ops/sec is what I should be getting.
>>
>> Could anyone help me in figuring out what I am doing wrong?
>>
>>
>> are your riak nodes located in the same DC?
>>
>> -Ruslan
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120630/20430010/attachment.html>


More information about the riak-users mailing list