java client: overload leads to BlockingOperationException

Luke Bakken lbakken at basho.com
Thu Jul 14 08:42:05 EDT 2016


Hi Henning,

The best place to continue discussion would be to file an issue in
GitHub. This sounds like a bug or at least a place for improvement.

https://github.com/basho/riak-java-client/issues

> How many active, busy connections does Riak KV support?

You're correct that "it depends" is the right answer. In doing some
benchmarks with the .NET client, I found that there was little benefit
to the maximum number of connections exceeding the ring size in the
cluster. This is probably specific to the benchmarks I was doing at
the time, too. The best option is always to simulate your workload,
tweak settings, and benchmark.

--
Luke Bakken
Engineer
lbakken at basho.com


On Wed, Jul 13, 2016 at 7:03 AM, Henning Verbeek <hankipanky at gmail.com> wrote:
> I'm still struggling with a BlockingOperationException thrown by
> riak-java-client 2.0.6, which occurs when I put heavy load on Riak KV.
> Since https://github.com/basho/riak-java-client/issues/523 is fixed,
> this happens only in - what I assume is - an overload-scenario.
>
> The exception:
>
> 2016-07-13 14:41:12.789  localhost [nioEventLoopGroup-2-2] ERROR
> com.basho.riak.client.core.RiakNode - Operation onException() channel:
> id:237445453 localhost:8087 {}
> io.netty.util.concurrent.BlockingOperationException:
> DefaultChannelPromise at 77ccd827(incomplete)
>         at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)
>         at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
>         at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)
>         at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
>         at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
>         at com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:697)
>         at com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:656)
>         at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:587)
>         at com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:91)
>         at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:322)
>         at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:240)
>         at com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117)
>         at com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182)
>         at com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78)
>         at com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120)
>         at com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:176)
>         at com.basho.riak.client.core.FutureOperation.setComplete(FutureOperation.java:224)
>         at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:878)
>         at com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:30)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
>         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
>         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
>         at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
>         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
>         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
>         at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>         at java.lang.Thread.run(Thread.java:745)
>
> Then shortly thereafter:
>
> 2016-07-13 14:41:12.820  localhost [nioEventLoopGroup-2-2] ERROR
> com.basho.riak.client.core.RiakNode - Write failed on RiakNode
> localhost:8087 id: 237445453; cause: {}
> java.nio.channels.ClosedChannelException: null
> 2016-07-13 14:41:12.843  localhost [nioEventLoopGroup-2-2] ERROR
> com.basho.riak.client.core.RiakNode - Write failed on RiakNode
> localhost:8087 id: 237445453; cause: {}
> java.nio.channels.ClosedChannelException: null
>
> The result is a hanging thread, never returning from the call to
> `RiakClient.execute()`. It would be great, if in such a case, the
> client threw a RiakException (as javadoc says).
>
> But in the meantime, I'm trying to avoid getting into this scenario in
> the first place, by sizing the maxConnections of RiakConnector and my
> applications frontend threadpools appropriately. My current
> observation is this: If I set RiakConnector.withMaxConnections() too
> small for the given load (let's say 100), I get the
> BlockingOperationException, although according to my (potentially
> unprecise connection monitoring), the threshold is never reached.
> Setting maxConnections to a higher value (let's say 160) lets the
> loadtest run through fine, with around 110 active connections to Riak.
>
> It seems that in order to avoid the BlockingOperationException, I must
> set the maxConnections limit higher than what I'll need. Maybe I
> should skip maxConnections altogether then, and leave it at 0? But
> then, how do I protect against an overload scenario? How many active,
> busy connections does Riak KV support? I'm pretty sure the answer to
> that is "it depends...".
>
> Thanks for your input.
> Cheers,
> Henning
> --
> My other signature is a regular expression.
> http://www.pray4snow.de
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




More information about the riak-users mailing list