java client: overload leads to BlockingOperationException

Henning Verbeek hankipanky at gmail.com
Fri Jul 15 06:32:30 EDT 2016


Thanks, Luke!
https://github.com/basho/riak-java-client/issues/638

On Thu, Jul 14, 2016 at 2:42 PM, Luke Bakken <lbakken at basho.com> wrote:
> Hi Henning,
>
> The best place to continue discussion would be to file an issue in
> GitHub. This sounds like a bug or at least a place for improvement.
>
> https://github.com/basho/riak-java-client/issues
>
>> How many active, busy connections does Riak KV support?
>
> You're correct that "it depends" is the right answer. In doing some
> benchmarks with the .NET client, I found that there was little benefit
> to the maximum number of connections exceeding the ring size in the
> cluster. This is probably specific to the benchmarks I was doing at
> the time, too. The best option is always to simulate your workload,
> tweak settings, and benchmark.
>
> --
> Luke Bakken
> Engineer
> lbakken at basho.com
>
>
> On Wed, Jul 13, 2016 at 7:03 AM, Henning Verbeek <hankipanky at gmail.com> wrote:
>> I'm still struggling with a BlockingOperationException thrown by
>> riak-java-client 2.0.6, which occurs when I put heavy load on Riak KV.
>> Since https://github.com/basho/riak-java-client/issues/523 is fixed,
>> this happens only in - what I assume is - an overload-scenario.
>>
>> The exception:
>>
>> 2016-07-13 14:41:12.789  localhost [nioEventLoopGroup-2-2] ERROR
>> com.basho.riak.client.core.RiakNode - Operation onException() channel:
>> id:237445453 localhost:8087 {}
>> io.netty.util.concurrent.BlockingOperationException:
>> DefaultChannelPromise at 77ccd827(incomplete)
>>         at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)
>>         at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
>>         at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)
>>         at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
>>         at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
>>         at com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:697)
>>         at com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:656)
>>         at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:587)
>>         at com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:91)
>>         at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:322)
>>         at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:240)
>>         at com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117)
>>         at com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182)
>>         at com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78)
>>         at com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120)
>>         at com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:176)
>>         at com.basho.riak.client.core.FutureOperation.setComplete(FutureOperation.java:224)
>>         at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:878)
>>         at com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:30)
>>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
>>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
>>         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
>>         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
>>         at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
>>         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
>>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
>>         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
>>         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>>         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>>         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>>         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
>>         at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> Then shortly thereafter:
>>
>> 2016-07-13 14:41:12.820  localhost [nioEventLoopGroup-2-2] ERROR
>> com.basho.riak.client.core.RiakNode - Write failed on RiakNode
>> localhost:8087 id: 237445453; cause: {}
>> java.nio.channels.ClosedChannelException: null
>> 2016-07-13 14:41:12.843  localhost [nioEventLoopGroup-2-2] ERROR
>> com.basho.riak.client.core.RiakNode - Write failed on RiakNode
>> localhost:8087 id: 237445453; cause: {}
>> java.nio.channels.ClosedChannelException: null
>>
>> The result is a hanging thread, never returning from the call to
>> `RiakClient.execute()`. It would be great, if in such a case, the
>> client threw a RiakException (as javadoc says).
>>
>> But in the meantime, I'm trying to avoid getting into this scenario in
>> the first place, by sizing the maxConnections of RiakConnector and my
>> applications frontend threadpools appropriately. My current
>> observation is this: If I set RiakConnector.withMaxConnections() too
>> small for the given load (let's say 100), I get the
>> BlockingOperationException, although according to my (potentially
>> unprecise connection monitoring), the threshold is never reached.
>> Setting maxConnections to a higher value (let's say 160) lets the
>> loadtest run through fine, with around 110 active connections to Riak.
>>
>> It seems that in order to avoid the BlockingOperationException, I must
>> set the maxConnections limit higher than what I'll need. Maybe I
>> should skip maxConnections altogether then, and leave it at 0? But
>> then, how do I protect against an overload scenario? How many active,
>> busy connections does Riak KV support? I'm pretty sure the answer to
>> that is "it depends...".
>>
>> Thanks for your input.
>> Cheers,
>> Henning
>> --
>> My other signature is a regular expression.
>> http://www.pray4snow.de
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
My other signature is a regular expression.
http://www.pray4snow.de




More information about the riak-users mailing list