java client: overload leads to BlockingOperationException

Henning Verbeek hankipanky at gmail.com
Wed Jul 13 10:03:15 EDT 2016


I'm still struggling with a BlockingOperationException thrown by
riak-java-client 2.0.6, which occurs when I put heavy load on Riak KV.
Since https://github.com/basho/riak-java-client/issues/523 is fixed,
this happens only in - what I assume is - an overload-scenario.

The exception:

2016-07-13 14:41:12.789  localhost [nioEventLoopGroup-2-2] ERROR
com.basho.riak.client.core.RiakNode - Operation onException() channel:
id:237445453 localhost:8087 {}
io.netty.util.concurrent.BlockingOperationException:
DefaultChannelPromise at 77ccd827(incomplete)
        at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)
        at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
        at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)
        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
        at com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:697)
        at com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:656)
        at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:587)
        at com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:91)
        at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:322)
        at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:240)
        at com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117)
        at com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182)
        at com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78)
        at com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120)
        at com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:176)
        at com.basho.riak.client.core.FutureOperation.setComplete(FutureOperation.java:224)
        at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:878)
        at com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:30)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
        at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

Then shortly thereafter:

2016-07-13 14:41:12.820  localhost [nioEventLoopGroup-2-2] ERROR
com.basho.riak.client.core.RiakNode - Write failed on RiakNode
localhost:8087 id: 237445453; cause: {}
java.nio.channels.ClosedChannelException: null
2016-07-13 14:41:12.843  localhost [nioEventLoopGroup-2-2] ERROR
com.basho.riak.client.core.RiakNode - Write failed on RiakNode
localhost:8087 id: 237445453; cause: {}
java.nio.channels.ClosedChannelException: null

The result is a hanging thread, never returning from the call to
`RiakClient.execute()`. It would be great, if in such a case, the
client threw a RiakException (as javadoc says).

But in the meantime, I'm trying to avoid getting into this scenario in
the first place, by sizing the maxConnections of RiakConnector and my
applications frontend threadpools appropriately. My current
observation is this: If I set RiakConnector.withMaxConnections() too
small for the given load (let's say 100), I get the
BlockingOperationException, although according to my (potentially
unprecise connection monitoring), the threshold is never reached.
Setting maxConnections to a higher value (let's say 160) lets the
loadtest run through fine, with around 110 active connections to Riak.

It seems that in order to avoid the BlockingOperationException, I must
set the maxConnections limit higher than what I'll need. Maybe I
should skip maxConnections altogether then, and leave it at 0? But
then, how do I protect against an overload scenario? How many active,
busy connections does Riak KV support? I'm pretty sure the answer to
that is "it depends...".

Thanks for your input.
Cheers,
Henning
-- 
My other signature is a regular expression.
http://www.pray4snow.de




More information about the riak-users mailing list