Strange Exception in java-client

Ingo Rockel ingo.rockel at bluelionmobile.com
Tue Feb 5 09:21:33 EST 2013


Hi Brian,

thanks for fixing this so fast!

Ingo

Am 02.02.2013 19:43, schrieb Brian Roach:
> Ingo -
>
> As an FYI, once I started looking into this I found you had stumbled
> into quite the can of worms. I just finished a pretty comprehensive
> set of changes that will now allow proper handling of tombstones in
> the Java client - https://github.com/basho/riak-java-client/pull/195
>
> Sorry you had to be the one, but thanks for letting us know about this!
> - Roach
>
> On Thu, Jan 31, 2013 at 10:36 AM, Ingo Rockel
> <ingo.rockel at bluelionmobile.com> wrote:
>> Hi Brian,
>>
>> thanks for the suggestion but I already chose a different solution for now,
>> if these messages get deleted I just delete the links to the message and
>> mark the message as "abandoned" and available for reuse. So I don't run into
>> the conflict if I need to store the message again.
>>
>> I just started the replay again and let it run for a while to see if this
>> works for me.
>>
>> Thanks!
>>
>>          Ingo
>>
>> Am 31.01.2013 16:54, schrieb Brian Roach:
>>
>>> Ingo -
>>>
>>> Unfortunately once you've got a sibling tombstone, things get a bit
>>> tricky. It's not going away until you resolve them which when using
>>> the JSONConverter in the Java client, you can't. Oddly enough, this is
>>> the first time anyone has hit this.
>>>
>>> I've got a couple ideas on how to address this properly but I need to
>>> look at some things first.
>>>
>>> In the meantime, what I'd suggest as a workaround is to copy and paste
>>> the source for the JSONConverter into your own Converter<T> that
>>> you'll pass to the StoreObject and modify it to return null:
>>>
>>>
>>> https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/client/convert/JSONConverter.java#L141
>>>
>>> Have it check to see if riakObject.getValue() returns null and if it
>>> does ... return null. You'll also need to modify your ConflictResolver
>>> to check for null as it iterates through the list of your POJOs that
>>> gets passed to it and act accordingly. If there's only a tombstone,
>>> just return null ... which means you will also need to modify your
>>> Mutation<T> to handle a null being passed to it in the case of there
>>> only being a tombstone.
>>>
>>> In the end this may well be what I do but I think I have a slightly
>>> more elegant solution that I want to look into.
>>>
>>> I've got an errand I need to run this morning, but I'll get to work on
>>> this as soon as I get back.
>>>
>>> Thanks, and sorry for the trouble.
>>> - Brian Roach
>>>
>>>
>>> On Thu, Jan 31, 2013 at 3:56 AM, Ingo Rockel
>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>
>>>> Hi Brian,
>>>>
>>>> thanks for the detailed explaination!
>>>>
>>>> I had a look at an object which constantly fails to load even if
>>>> retrying:
>>>>
>>>> lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081"
>>>> ---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091
>>>> ---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1
>>>> ---> Host: 172.22.3.14:8091
>>>> ---> User-Agent: lftp/4.3.3
>>>> ---> Accept: */*
>>>> ---> Connection: keep-alive
>>>> --->
>>>> <--- HTTP/1.1 300 Multiple Choices
>>>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>>>> <--- Vary: Accept, Accept-Encoding
>>>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>>>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
>>>> <--- ETag: "6PSreYIOL25KOpNyG0XPe7"
>>>> <--- Date: Thu, 31 Jan 2013 10:42:41 GMT
>>>> <--- Content-Type: text/plain
>>>> <--- Content-Length: 56
>>>> <---
>>>> <--* Siblings:
>>>> <--* 50Uz9nvQWwOUBE6USi2gki
>>>> <--* 1JsgLs3CE3k2mWsaCEiPp4
>>>> cat: Zugriff nicht möglich: 300 Multiple Choices
>>>> (/riak/m/Um|18498012|4|0|18298081)
>>>> lftp :~> cat
>>>>
>>>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki"
>>>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki
>>>> HTTP/1.1
>>>> ---> Host: 172.22.3.14:8091
>>>> ---> User-Agent: lftp/4.3.3
>>>> ---> Accept: */*
>>>> ---> Connection: keep-alive
>>>> --->
>>>> <--- HTTP/1.1 200 OK
>>>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>>>> <--- Vary: Accept-Encoding
>>>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>>>> <--- Link: </riak/m>; rel="up"
>>>> <--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT
>>>> <--- ETag: "50Uz9nvQWwOUBE6USi2gki"
>>>> <--- Date: Thu, 31 Jan 2013 10:42:49 GMT
>>>> <--- Content-Type: application/octet-stream
>>>> <--- Content-Length: 0
>>>> <---
>>>> lftp :~> cat
>>>>
>>>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4"
>>>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4
>>>> HTTP/1.1
>>>> ---> Host: 172.22.3.14:8091
>>>> ---> User-Agent: lftp/4.3.3
>>>> ---> Accept: */*
>>>> ---> Connection: keep-alive
>>>> --->
>>>> <--- HTTP/1.1 200 OK
>>>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>>>> <--- Vary: Accept-Encoding
>>>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>>>> <--- Link: </riak/m>; rel="up"
>>>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
>>>> <--- ETag: "1JsgLs3CE3k2mWsaCEiPp4"
>>>> <--- Date: Thu, 31 Jan 2013 10:43:01 GMT
>>>> <--- Content-Type: application/json; charset=UTF-8
>>>> <--- Content-Length: 114
>>>> <---
>>>>
>>>> {"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0}
>>>>
>>>> the object has two siblings, one the deleted empty "tombstone" and one
>>>> with
>>>> the new data. And there's a gap auf 2:30min between both siblings.
>>>> There's
>>>> no immediate write after the deletion. I logged the write operations and
>>>> this gap is there also. And the java client constantly fails to load this
>>>> object.
>>>>
>>>> This objects are user notifications which are identified by their type
>>>> (the
>>>> key) and can be updated frequently but the user also is allowed to delete
>>>> a
>>>> single notification (which he did in this case).
>>>>
>>>> Is there anything I can do to resolve this? A last write wins just for
>>>> this
>>>> case? If I write a new notification to update the old one I don't care if
>>>> there's any deletion tombstone...
>>>>
>>>> Ingo
>>>>
>>>> Am 30.01.2013 22:42, schrieb Brian Roach:
>>>>
>>>>> Ingo -
>>>>>
>>>>> Riak is returning an object with no contents (which ends up being an
>>>>> empty String passed to Jackson).
>>>>>
>>>>> Unless you've somehow mangled the data yourself (which sounds unlikely
>>>>> given the bit about the 404 from the command line; more on that in a
>>>>> bit) what's happening is that you're encountering a tombstone; an
>>>>> object that has been deleted via a delete operation but hasn't been
>>>>> removed yet. This causes an "empty" object to be returned (the
>>>>> tombstone) and causes Jackson to puke (HTTP will actually return this
>>>>> as a 404 but if you look there's still a X-Riak-Vclock: header with a
>>>>> vclock).
>>>>>
>>>>> Probably the best description of how this works in Riak is a post by
>>>>> Jon Meredith which can be found here:
>>>>>
>>>>>
>>>>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html
>>>>>
>>>>> Unfortunately this is something the Java client doesn't know what to
>>>>> do with when using the default JSONConverter and your own POJOs. And
>>>>> it's not as simple as "just return null" because of a case where a
>>>>> tombstone could actually be a sibling and the client then needs to
>>>>> resolve the conflict which is the next step in the process. It's
>>>>> something I'm going to have to think about.
>>>>>
>>>>> As you've discovered, when the tombstone isn't a sibling simply
>>>>> retrying will often work because by then the delete has fully
>>>>> completed and the tombstones have been removed from the Riak nodes.
>>>>>
>>>>> Is there a reason you're rapidly doing a delete then a store (which
>>>>> triggers that fetch)?
>>>>>
>>>>> Thanks,
>>>>> Brian Roach
>>>>>
>>>>> On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel
>>>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>>>
>>>>>>
>>>>>> Hi Dmitri,
>>>>>>
>>>>>> it doesn't happen in my code and it does happen while the riak-client
>>>>>> tries
>>>>>> to deserialize a fetched object from riak in a "fetch-before-store"
>>>>>> (see
>>>>>> the
>>>>>> stack), I also get this error randomly while trying just to fetch an
>>>>>> object
>>>>>> from the database.
>>>>>>
>>>>>> And if I try to fetch the object from the cmdline I just get a 404. So
>>>>>> I
>>>>>> would expect the java-client just returns a null-result for this fetch
>>>>>> and
>>>>>> not to throw an exception.
>>>>>>
>>>>>> All my objects are stored using the riak-java-client and the
>>>>>> json-serializer.
>>>>>>
>>>>>> Ahh, just tested: if I retry it sometimes works, although most of the
>>>>>> time
>>>>>> still fails (haven't tried with a sleep so far).
>>>>>>
>>>>>> Ingo
>>>>>>
>>>>>> Am 30.01.2013 16:57, schrieb Dmitri Zagidulin:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Ingo.
>>>>>>>
>>>>>>> It's difficult to diagnose the exact reason without looking at your
>>>>>>> code.
>>>>>>> But that error is a JSON parser error. It gets thrown whenever the
>>>>>>> code
>>>>>>> tries to parse an empty string as a json object.
>>>>>>> The general-case solution is to validate your strings or input streams
>>>>>>> that you're turning into JSON objects, or to catch an exception when
>>>>>>> creating that object and deal with it accordingly.
>>>>>>>
>>>>>>> But again, it's hard to say why it's happening exactly, in your case
>>>>>>> --
>>>>>>> try to determine where in your code that's happening and think of ways
>>>>>>> some input or result is empty, and check for that.
>>>>>>>
>>>>>>> Dmitri
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel
>>>>>>> <ingo.rockel at bluelionmobile.com
>>>>>>> <mailto:ingo.rockel at bluelionmobile.com>>
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>        Hi,
>>>>>>>
>>>>>>>        I wrote a java tool to convert part of our data from a
>>>>>>>        mysql-database into riak. As this tool is running while our
>>>>>>> system
>>>>>>>        is still up, it needs to replay all modifications done in the
>>>>>>> mysql
>>>>>>>        database, during these modifications I sometimes get this
>>>>>>> exception
>>>>>>>        from the riak client:
>>>>>>>
>>>>>>>        com.basho.riak.client.convert.__ConversionException:
>>>>>>>
>>>>>>>        java.io.EOFException: No content to map to Object due to end of
>>>>>>> input
>>>>>>>        com.basho.riak.client.convert.__ConversionException:
>>>>>>>
>>>>>>>        java.io.EOFException: No content to map to Object due to end of
>>>>>>> input
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243)
>>>>>>>                 at java.lang.Thread.run(Thread.__java:722)
>>>>>>>
>>>>>>>        Caused by: java.io.EOFException: No content to map to Object due
>>>>>>> to
>>>>>>>        end of input
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863)
>>>>>>>                 at
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156)
>>>>>>>
>>>>>>>                 ... 7 more
>>>>>>>
>>>>>>>        it only happens once every few thousand updates and if I check
>>>>>>> the
>>>>>>>        object from the cmdline I only get a 404.
>>>>>>>
>>>>>>>        Any ideas what might cause this and how to fix/workaround it?
>>>>>>>
>>>>>>>        Ingo
>>>>>>>
>>>>>>>
>>>>>>>        _________________________________________________
>>>>>>>        riak-users mailing list
>>>>>>>        riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>>>>>>
>>>>>>> http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com
>>>>>>>
>>>>>>> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> riak-users mailing list
>>>>>>> riak-users at lists.basho.com
>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Software Architect
>>>>>>
>>>>>> Blue Lion mobile GmbH
>>>>>> Tel. +49 (0) 221 788 797 14
>>>>>> Fax. +49 (0) 221 788 797 19
>>>>>> Mob. +49 (0) 176 24 87 30 89
>>>>>>
>>>>>> ingo.rockel at bluelionmobile.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> qeep: Hefferwolf
>>>>>>
>>>>>>
>>>>>>
>>>>>> www.bluelionmobile.com
>>>>>> www.qeep.net
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> riak-users mailing list
>>>>>> riak-users at lists.basho.com
>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Software Architect
>>>>
>>>> Blue Lion mobile GmbH
>>>> Tel. +49 (0) 221 788 797 14
>>>> Fax. +49 (0) 221 788 797 19
>>>> Mob. +49 (0) 176 24 87 30 89
>>>>
>>>> ingo.rockel at bluelionmobile.com
>>>>>>>
>>>>>>> qeep: Hefferwolf
>>>>
>>>>
>>>> www.bluelionmobile.com
>>>> www.qeep.net
>>
>>
>>
>> --
>> Software Architect
>>
>> Blue Lion mobile GmbH
>> Tel. +49 (0) 221 788 797 14
>> Fax. +49 (0) 221 788 797 19
>> Mob. +49 (0) 176 24 87 30 89
>>
>> ingo.rockel at bluelionmobile.com
>>>>> qeep: Hefferwolf
>>
>> www.bluelionmobile.com
>> www.qeep.net


-- 
Software Architect

Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89

ingo.rockel at bluelionmobile.com
 >>> qeep: Hefferwolf

www.bluelionmobile.com
www.qeep.net




More information about the riak-users mailing list