riak java client causing OOMs on high latency [resolution]

Dietrich Featherston d at d2fn.com
Thu Jan 10 17:41:52 EST 2013


We now believe this to be fixed in the following commit to rjc. Socket
timeouts were not resulting in permits being released -
https://github.com/basho/riak-java-client/commit/b8a4e7b872ae26e2c38e0ff4dd3e2d617a2c8344

The content of the byte[]s indicated that multiple threads were
reading from the same socket, and hence reading "length headers" out
of the middle of messages. We went digging for culprits, found the
missing finally block, then noticed this change downstream.

Will post more info if this doesn't do the trick.


On Thu, Jan 10, 2013 at 11:44 AM, Dietrich Featherston <d at d2fn.com> wrote:
> Have confirmed that this is likely a protobuf corruption issue and not
> related to the actual data we are storing and fetching. We never see
> object sizes as reported in /stats higher than about 600-700KB.
> However, we forked the riak driver to instrument the byte[] allocation
> sizes in this block of code. In some cases, it is attempting to create
> ~2GB byte[] arrays and in other cases it is actually returning a
> negative size header. See attached screenshot for the range of lengths
> we are seeing returned.
>
>
> On Tue, Jan 8, 2013 at 6:50 AM, Brian Roach <roach at basho.com> wrote:
>> The code you cite is reading a size (32 bit int, network byte order)
>> and a message code (8bit int)  from the socket. It then creates a
>> byte[] of the size required for the amount of data that has been
>> requested and then sent back by Riak to the client in a response. (See
>> the docs here: http://docs.basho.com/riak/latest/references/apis/protocol-buffers/
>>  that show this format )
>>
>> That byte[] is then passed into the Google protocol buffers generated
>> code where the appropriate protocol buffers object(s) are deserialized
>> from those bytes and the information contained therein is extracted
>> from them into our own objects which are returned to the caller as a
>> response.
>>
>> From the client's perspective, if that's how much data you're getting,
>> that's how much data you've requested, and how much Riak has sent you.
>>
>> Thanks,
>> Brian Roach
>>
>> On Mon, Jan 7, 2013 at 3:26 PM, Dietrich Featherston <d at d2fn.com> wrote:
>>> We're seeing instances of a JVM app which talks to riak run out of
>>> memory when riak operations rise in latency or riak becomes otherwise
>>> unresponsive. A heap dump of the JVM at the time of the OOM show that
>>> 91% of the 1G (active) heap is consumed by large byte[] instances. In
>>> our case 3 of those byte[]s are in the 200MB range with size dropping
>>> off after that. The byte[] instances cannot be traced back to a
>>> specific variable as their references appear to be stack-allocated
>>> local method variables. But, based on the name of the thread, we can
>>> tell that the thread is doing a store operation against
>>> riak at localhost.
>>>
>>> Inspection of the data in one of these byte[]s shows what looks like
>>> an r_object response with headers and footer boilerplate around our
>>> object payload. This 200+MB byte[] is filled with 0s after the 338th
>>> element which is really confusing and indicates that far too much
>>> space is being allocated to read the protobuf payload. Here's a dump
>>> of one of these instances:
>>> https://gist.github.com/40ef9b2ff561e973a72c
>>>
>>> It's also worth mentioning that, according to /stats,
>>> get_fsm_objsize_100 is consistently under 1MB so there is no reason to
>>> think that our objects are actually this large.
>>>
>>> At this point I'm suspicious of the following code creating too large
>>> a byte[] from possibly too large a return from dis.readInt()
>>>
>>> https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/pbc/RiakConnection.java#L110
>>>
>>> Unsure if that indicates a problem in the driver or the server-side
>>> erlang protobuf server.
>>>
>>> Suspicious that requests pile up and many of these byte[]s are hanging
>>> out--enough to cause an OOM. It's possible that they are always very
>>> large, but are short-lived enough as to not cause a problem until
>>> latencies rise increasing their numbers briefly.
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> D
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




More information about the riak-users mailing list