Strange Exception in java-client

Brian Roach roach at basho.com
Sat Feb 2 13:43:28 EST 2013


Ingo -

As an FYI, once I started looking into this I found you had stumbled
into quite the can of worms. I just finished a pretty comprehensive
set of changes that will now allow proper handling of tombstones in
the Java client - https://github.com/basho/riak-java-client/pull/195

Sorry you had to be the one, but thanks for letting us know about this!
- Roach

On Thu, Jan 31, 2013 at 10:36 AM, Ingo Rockel
<ingo.rockel at bluelionmobile.com> wrote:
> Hi Brian,
>
> thanks for the suggestion but I already chose a different solution for now,
> if these messages get deleted I just delete the links to the message and
> mark the message as "abandoned" and available for reuse. So I don't run into
> the conflict if I need to store the message again.
>
> I just started the replay again and let it run for a while to see if this
> works for me.
>
> Thanks!
>
>         Ingo
>
> Am 31.01.2013 16:54, schrieb Brian Roach:
>
>> Ingo -
>>
>> Unfortunately once you've got a sibling tombstone, things get a bit
>> tricky. It's not going away until you resolve them which when using
>> the JSONConverter in the Java client, you can't. Oddly enough, this is
>> the first time anyone has hit this.
>>
>> I've got a couple ideas on how to address this properly but I need to
>> look at some things first.
>>
>> In the meantime, what I'd suggest as a workaround is to copy and paste
>> the source for the JSONConverter into your own Converter<T> that
>> you'll pass to the StoreObject and modify it to return null:
>>
>>
>> https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/client/convert/JSONConverter.java#L141
>>
>> Have it check to see if riakObject.getValue() returns null and if it
>> does ... return null. You'll also need to modify your ConflictResolver
>> to check for null as it iterates through the list of your POJOs that
>> gets passed to it and act accordingly. If there's only a tombstone,
>> just return null ... which means you will also need to modify your
>> Mutation<T> to handle a null being passed to it in the case of there
>> only being a tombstone.
>>
>> In the end this may well be what I do but I think I have a slightly
>> more elegant solution that I want to look into.
>>
>> I've got an errand I need to run this morning, but I'll get to work on
>> this as soon as I get back.
>>
>> Thanks, and sorry for the trouble.
>> - Brian Roach
>>
>>
>> On Thu, Jan 31, 2013 at 3:56 AM, Ingo Rockel
>> <ingo.rockel at bluelionmobile.com> wrote:
>>>
>>> Hi Brian,
>>>
>>> thanks for the detailed explaination!
>>>
>>> I had a look at an object which constantly fails to load even if
>>> retrying:
>>>
>>> lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081"
>>> ---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091
>>> ---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1
>>> ---> Host: 172.22.3.14:8091
>>> ---> User-Agent: lftp/4.3.3
>>> ---> Accept: */*
>>> ---> Connection: keep-alive
>>> --->
>>> <--- HTTP/1.1 300 Multiple Choices
>>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>>> <--- Vary: Accept, Accept-Encoding
>>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
>>> <--- ETag: "6PSreYIOL25KOpNyG0XPe7"
>>> <--- Date: Thu, 31 Jan 2013 10:42:41 GMT
>>> <--- Content-Type: text/plain
>>> <--- Content-Length: 56
>>> <---
>>> <--* Siblings:
>>> <--* 50Uz9nvQWwOUBE6USi2gki
>>> <--* 1JsgLs3CE3k2mWsaCEiPp4
>>> cat: Zugriff nicht möglich: 300 Multiple Choices
>>> (/riak/m/Um|18498012|4|0|18298081)
>>> lftp :~> cat
>>>
>>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki"
>>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki
>>> HTTP/1.1
>>> ---> Host: 172.22.3.14:8091
>>> ---> User-Agent: lftp/4.3.3
>>> ---> Accept: */*
>>> ---> Connection: keep-alive
>>> --->
>>> <--- HTTP/1.1 200 OK
>>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>>> <--- Vary: Accept-Encoding
>>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>>> <--- Link: </riak/m>; rel="up"
>>> <--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT
>>> <--- ETag: "50Uz9nvQWwOUBE6USi2gki"
>>> <--- Date: Thu, 31 Jan 2013 10:42:49 GMT
>>> <--- Content-Type: application/octet-stream
>>> <--- Content-Length: 0
>>> <---
>>> lftp :~> cat
>>>
>>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4"
>>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4
>>> HTTP/1.1
>>> ---> Host: 172.22.3.14:8091
>>> ---> User-Agent: lftp/4.3.3
>>> ---> Accept: */*
>>> ---> Connection: keep-alive
>>> --->
>>> <--- HTTP/1.1 200 OK
>>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>>> <--- Vary: Accept-Encoding
>>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>>> <--- Link: </riak/m>; rel="up"
>>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
>>> <--- ETag: "1JsgLs3CE3k2mWsaCEiPp4"
>>> <--- Date: Thu, 31 Jan 2013 10:43:01 GMT
>>> <--- Content-Type: application/json; charset=UTF-8
>>> <--- Content-Length: 114
>>> <---
>>>
>>> {"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0}
>>>
>>> the object has two siblings, one the deleted empty "tombstone" and one
>>> with
>>> the new data. And there's a gap auf 2:30min between both siblings.
>>> There's
>>> no immediate write after the deletion. I logged the write operations and
>>> this gap is there also. And the java client constantly fails to load this
>>> object.
>>>
>>> This objects are user notifications which are identified by their type
>>> (the
>>> key) and can be updated frequently but the user also is allowed to delete
>>> a
>>> single notification (which he did in this case).
>>>
>>> Is there anything I can do to resolve this? A last write wins just for
>>> this
>>> case? If I write a new notification to update the old one I don't care if
>>> there's any deletion tombstone...
>>>
>>> Ingo
>>>
>>> Am 30.01.2013 22:42, schrieb Brian Roach:
>>>
>>>> Ingo -
>>>>
>>>> Riak is returning an object with no contents (which ends up being an
>>>> empty String passed to Jackson).
>>>>
>>>> Unless you've somehow mangled the data yourself (which sounds unlikely
>>>> given the bit about the 404 from the command line; more on that in a
>>>> bit) what's happening is that you're encountering a tombstone; an
>>>> object that has been deleted via a delete operation but hasn't been
>>>> removed yet. This causes an "empty" object to be returned (the
>>>> tombstone) and causes Jackson to puke (HTTP will actually return this
>>>> as a 404 but if you look there's still a X-Riak-Vclock: header with a
>>>> vclock).
>>>>
>>>> Probably the best description of how this works in Riak is a post by
>>>> Jon Meredith which can be found here:
>>>>
>>>>
>>>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html
>>>>
>>>> Unfortunately this is something the Java client doesn't know what to
>>>> do with when using the default JSONConverter and your own POJOs. And
>>>> it's not as simple as "just return null" because of a case where a
>>>> tombstone could actually be a sibling and the client then needs to
>>>> resolve the conflict which is the next step in the process. It's
>>>> something I'm going to have to think about.
>>>>
>>>> As you've discovered, when the tombstone isn't a sibling simply
>>>> retrying will often work because by then the delete has fully
>>>> completed and the tombstones have been removed from the Riak nodes.
>>>>
>>>> Is there a reason you're rapidly doing a delete then a store (which
>>>> triggers that fetch)?
>>>>
>>>> Thanks,
>>>> Brian Roach
>>>>
>>>> On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel
>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>
>>>>>
>>>>> Hi Dmitri,
>>>>>
>>>>> it doesn't happen in my code and it does happen while the riak-client
>>>>> tries
>>>>> to deserialize a fetched object from riak in a "fetch-before-store"
>>>>> (see
>>>>> the
>>>>> stack), I also get this error randomly while trying just to fetch an
>>>>> object
>>>>> from the database.
>>>>>
>>>>> And if I try to fetch the object from the cmdline I just get a 404. So
>>>>> I
>>>>> would expect the java-client just returns a null-result for this fetch
>>>>> and
>>>>> not to throw an exception.
>>>>>
>>>>> All my objects are stored using the riak-java-client and the
>>>>> json-serializer.
>>>>>
>>>>> Ahh, just tested: if I retry it sometimes works, although most of the
>>>>> time
>>>>> still fails (haven't tried with a sleep so far).
>>>>>
>>>>> Ingo
>>>>>
>>>>> Am 30.01.2013 16:57, schrieb Dmitri Zagidulin:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Ingo.
>>>>>>
>>>>>> It's difficult to diagnose the exact reason without looking at your
>>>>>> code.
>>>>>> But that error is a JSON parser error. It gets thrown whenever the
>>>>>> code
>>>>>> tries to parse an empty string as a json object.
>>>>>> The general-case solution is to validate your strings or input streams
>>>>>> that you're turning into JSON objects, or to catch an exception when
>>>>>> creating that object and deal with it accordingly.
>>>>>>
>>>>>> But again, it's hard to say why it's happening exactly, in your case
>>>>>> --
>>>>>> try to determine where in your code that's happening and think of ways
>>>>>> some input or result is empty, and check for that.
>>>>>>
>>>>>> Dmitri
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel
>>>>>> <ingo.rockel at bluelionmobile.com
>>>>>> <mailto:ingo.rockel at bluelionmobile.com>>
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>       Hi,
>>>>>>
>>>>>>       I wrote a java tool to convert part of our data from a
>>>>>>       mysql-database into riak. As this tool is running while our
>>>>>> system
>>>>>>       is still up, it needs to replay all modifications done in the
>>>>>> mysql
>>>>>>       database, during these modifications I sometimes get this
>>>>>> exception
>>>>>>       from the riak client:
>>>>>>
>>>>>>       com.basho.riak.client.convert.__ConversionException:
>>>>>>
>>>>>>       java.io.EOFException: No content to map to Object due to end of
>>>>>> input
>>>>>>       com.basho.riak.client.convert.__ConversionException:
>>>>>>
>>>>>>       java.io.EOFException: No content to map to Object due to end of
>>>>>> input
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243)
>>>>>>                at java.lang.Thread.run(Thread.__java:722)
>>>>>>
>>>>>>       Caused by: java.io.EOFException: No content to map to Object due
>>>>>> to
>>>>>>       end of input
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863)
>>>>>>                at
>>>>>>
>>>>>>
>>>>>>
>>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156)
>>>>>>
>>>>>>                ... 7 more
>>>>>>
>>>>>>       it only happens once every few thousand updates and if I check
>>>>>> the
>>>>>>       object from the cmdline I only get a 404.
>>>>>>
>>>>>>       Any ideas what might cause this and how to fix/workaround it?
>>>>>>
>>>>>>       Ingo
>>>>>>
>>>>>>
>>>>>>       _________________________________________________
>>>>>>       riak-users mailing list
>>>>>>       riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>>>>>
>>>>>> http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com
>>>>>>
>>>>>> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> riak-users mailing list
>>>>>> riak-users at lists.basho.com
>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Software Architect
>>>>>
>>>>> Blue Lion mobile GmbH
>>>>> Tel. +49 (0) 221 788 797 14
>>>>> Fax. +49 (0) 221 788 797 19
>>>>> Mob. +49 (0) 176 24 87 30 89
>>>>>
>>>>> ingo.rockel at bluelionmobile.com
>>>>>>>>
>>>>>>>>
>>>>>>>> qeep: Hefferwolf
>>>>>
>>>>>
>>>>>
>>>>> www.bluelionmobile.com
>>>>> www.qeep.net
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>>
>>>
>>> --
>>> Software Architect
>>>
>>> Blue Lion mobile GmbH
>>> Tel. +49 (0) 221 788 797 14
>>> Fax. +49 (0) 221 788 797 19
>>> Mob. +49 (0) 176 24 87 30 89
>>>
>>> ingo.rockel at bluelionmobile.com
>>>>>>
>>>>>> qeep: Hefferwolf
>>>
>>>
>>> www.bluelionmobile.com
>>> www.qeep.net
>
>
>
> --
> Software Architect
>
> Blue Lion mobile GmbH
> Tel. +49 (0) 221 788 797 14
> Fax. +49 (0) 221 788 797 19
> Mob. +49 (0) 176 24 87 30 89
>
> ingo.rockel at bluelionmobile.com
>>>> qeep: Hefferwolf
>
> www.bluelionmobile.com
> www.qeep.net




More information about the riak-users mailing list