New to Riak, hitting a connection limit

David Jacobs citeon at gmail.com
Wed Aug 8 16:55:18 EDT 2012


Doug,

Using the node_riak and async modules, you can just do async.forEachLimit
with an appropriate batch size. That effectively throttles the simultaneous
requests and prevents overloading the cluster, but allows much higher
throughput. This enables a sustainable load to be generated over an
arbitrary duration of time for benchmarking. You can also run multiple
batches in parallel, with each forEachLimit batched independently. This
makes an easy way to concurrently run different batches of different
operations (just reduce the batch size of each group accordingly).

It is probably always preferably to handle a large sequence of operations
this way, rather than relying on an in-memory connection pool to queue an
unlimited number of items. That would quickly become a problem, crashing
Node.js from unbounded memory growth. Better to have lazier iterators that
are throttled based on that actual performance at that time - if a batch
takes longer than normal, the next batch will wait longer than normal. This
is typically what you'd want.

-David
@MetaThis



> Date: Fri, 3 Aug 2012 09:53:13 -0700
> From: Mark Phillips <mark at basho.com>
> To: Douglas Muth <doug.muth at gmail.com>
> Cc: riak-users at lists.basho.com
> Subject: Re: New to Riak, hitting a connection limit
> Message-ID:
>         <
> CAE8zJQGkz4W_mTSoNYh7G0fbhaUtuM5SUwX1rv1cTmf9WWUSww at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Doug,
>
> In line...
>
> On Fri, Aug 3, 2012 at 7:48 AM, Douglas Muth <doug.muth at gmail.com> wrote:
> > Hi Mark,
> >
> > On Thu, Aug 2, 2012 at 2:02 AM, Mark Phillips <mark at basho.com> wrote:
> >>
> >> I think you're problem here is that you're front-loading requests.
> >> You're better off to issue them serially and wait for one to finish
> >> before issuing the next. Rumor has there's an example of how one might
> >> do this somewhere in Riaktant [1], a somewhat dusty but useful sample
> >> app. I'll dig through it tomorrow if I remember.
> >>
> >
> > Your comment gave me some ideas, so I went and started using the
> > node-pool module (https://github.com/coopernurse/node-pool) in my test
> > script.  That artificially limited the number of concurrent
> > connections to whatever number I specified, and solved the problem.
> >
> > Unfortunately found another issue in node_riak which makes it
> > unsuitable for production use: unbounded use of network sockets.  It
> > seems that socks aren't being closed/reused properly.  At ~250
> > queries/second, I could start getting EADDRNOTAVAIL errors after about
> > 15k total queries. :-(
> >
>
> Hmm. I would say you should ping Matt Ranney and Voxer about this but
> it looks like you've already dug into that. [1]
>
> > Since my workplace is a node.js shop, and I really want to use Riak,
> > I'm going to take a whack at writing my own node.js driver for Riak.
> > Would anyone find it useful if I could talk my employer into releasing
> > it under the GPL?
> >
>
> So, as much as I love to see more code written (especially from the
> community), you might want to hold off. To date, the most heavily-used
> node.js code with Riak has been riak-js (which used to live at
> riakjs.org). Francisco Treacy was leading the charge on this but had
> to put it down on due to some other opportunities. Sadly, it hasn't
> seen too much love over the past few months. However, I have it from a
> few very solid authorities that there will soon be consistent work
> being done on Riak-js. There's a fork of it here (but there should be
> a different location for the new work that's happening) :
>
> https://github.com/mostlyserious/riak-js
>
> So, I would say hold off on writing your own and ping me offline if
> you're interested in lending a hand on riak-js. Or sit tight for a few
> days until they make their plans known. :)
>
>
> Mark
>
> [1] https://github.com/mranney/node_riak/issues/5
>
> > -- Doug
> >
> > --
> > http://www.dmuth.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120808/60b15021/attachment.html>


More information about the riak-users mailing list