Strange Exception in java-client

Ingo Rockel ingo.rockel at bluelionmobile.com
Thu Jan 31 12:36:10 EST 2013


Hi Brian,

thanks for the suggestion but I already chose a different solution for 
now, if these messages get deleted I just delete the links to the 
message and mark the message as "abandoned" and available for reuse. So 
I don't run into the conflict if I need to store the message again.

I just started the replay again and let it run for a while to see if 
this works for me.

Thanks!

	Ingo

Am 31.01.2013 16:54, schrieb Brian Roach:
> Ingo -
>
> Unfortunately once you've got a sibling tombstone, things get a bit
> tricky. It's not going away until you resolve them which when using
> the JSONConverter in the Java client, you can't. Oddly enough, this is
> the first time anyone has hit this.
>
> I've got a couple ideas on how to address this properly but I need to
> look at some things first.
>
> In the meantime, what I'd suggest as a workaround is to copy and paste
> the source for the JSONConverter into your own Converter<T> that
> you'll pass to the StoreObject and modify it to return null:
>
> https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/client/convert/JSONConverter.java#L141
>
> Have it check to see if riakObject.getValue() returns null and if it
> does ... return null. You'll also need to modify your ConflictResolver
> to check for null as it iterates through the list of your POJOs that
> gets passed to it and act accordingly. If there's only a tombstone,
> just return null ... which means you will also need to modify your
> Mutation<T> to handle a null being passed to it in the case of there
> only being a tombstone.
>
> In the end this may well be what I do but I think I have a slightly
> more elegant solution that I want to look into.
>
> I've got an errand I need to run this morning, but I'll get to work on
> this as soon as I get back.
>
> Thanks, and sorry for the trouble.
> - Brian Roach
>
>
> On Thu, Jan 31, 2013 at 3:56 AM, Ingo Rockel
> <ingo.rockel at bluelionmobile.com> wrote:
>> Hi Brian,
>>
>> thanks for the detailed explaination!
>>
>> I had a look at an object which constantly fails to load even if retrying:
>>
>> lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081"
>> ---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091
>> ---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1
>> ---> Host: 172.22.3.14:8091
>> ---> User-Agent: lftp/4.3.3
>> ---> Accept: */*
>> ---> Connection: keep-alive
>> --->
>> <--- HTTP/1.1 300 Multiple Choices
>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>> <--- Vary: Accept, Accept-Encoding
>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
>> <--- ETag: "6PSreYIOL25KOpNyG0XPe7"
>> <--- Date: Thu, 31 Jan 2013 10:42:41 GMT
>> <--- Content-Type: text/plain
>> <--- Content-Length: 56
>> <---
>> <--* Siblings:
>> <--* 50Uz9nvQWwOUBE6USi2gki
>> <--* 1JsgLs3CE3k2mWsaCEiPp4
>> cat: Zugriff nicht möglich: 300 Multiple Choices
>> (/riak/m/Um|18498012|4|0|18298081)
>> lftp :~> cat
>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki"
>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki
>> HTTP/1.1
>> ---> Host: 172.22.3.14:8091
>> ---> User-Agent: lftp/4.3.3
>> ---> Accept: */*
>> ---> Connection: keep-alive
>> --->
>> <--- HTTP/1.1 200 OK
>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>> <--- Vary: Accept-Encoding
>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>> <--- Link: </riak/m>; rel="up"
>> <--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT
>> <--- ETag: "50Uz9nvQWwOUBE6USi2gki"
>> <--- Date: Thu, 31 Jan 2013 10:42:49 GMT
>> <--- Content-Type: application/octet-stream
>> <--- Content-Length: 0
>> <---
>> lftp :~> cat
>> "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4"
>> ---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4
>> HTTP/1.1
>> ---> Host: 172.22.3.14:8091
>> ---> User-Agent: lftp/4.3.3
>> ---> Accept: */*
>> ---> Connection: keep-alive
>> --->
>> <--- HTTP/1.1 200 OK
>> <--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
>> <--- Vary: Accept-Encoding
>> <--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>> <--- Link: </riak/m>; rel="up"
>> <--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
>> <--- ETag: "1JsgLs3CE3k2mWsaCEiPp4"
>> <--- Date: Thu, 31 Jan 2013 10:43:01 GMT
>> <--- Content-Type: application/json; charset=UTF-8
>> <--- Content-Length: 114
>> <---
>> {"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0}
>>
>> the object has two siblings, one the deleted empty "tombstone" and one with
>> the new data. And there's a gap auf 2:30min between both siblings. There's
>> no immediate write after the deletion. I logged the write operations and
>> this gap is there also. And the java client constantly fails to load this
>> object.
>>
>> This objects are user notifications which are identified by their type (the
>> key) and can be updated frequently but the user also is allowed to delete a
>> single notification (which he did in this case).
>>
>> Is there anything I can do to resolve this? A last write wins just for this
>> case? If I write a new notification to update the old one I don't care if
>> there's any deletion tombstone...
>>
>> Ingo
>>
>> Am 30.01.2013 22:42, schrieb Brian Roach:
>>
>>> Ingo -
>>>
>>> Riak is returning an object with no contents (which ends up being an
>>> empty String passed to Jackson).
>>>
>>> Unless you've somehow mangled the data yourself (which sounds unlikely
>>> given the bit about the 404 from the command line; more on that in a
>>> bit) what's happening is that you're encountering a tombstone; an
>>> object that has been deleted via a delete operation but hasn't been
>>> removed yet. This causes an "empty" object to be returned (the
>>> tombstone) and causes Jackson to puke (HTTP will actually return this
>>> as a 404 but if you look there's still a X-Riak-Vclock: header with a
>>> vclock).
>>>
>>> Probably the best description of how this works in Riak is a post by
>>> Jon Meredith which can be found here:
>>>
>>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html
>>>
>>> Unfortunately this is something the Java client doesn't know what to
>>> do with when using the default JSONConverter and your own POJOs. And
>>> it's not as simple as "just return null" because of a case where a
>>> tombstone could actually be a sibling and the client then needs to
>>> resolve the conflict which is the next step in the process. It's
>>> something I'm going to have to think about.
>>>
>>> As you've discovered, when the tombstone isn't a sibling simply
>>> retrying will often work because by then the delete has fully
>>> completed and the tombstones have been removed from the Riak nodes.
>>>
>>> Is there a reason you're rapidly doing a delete then a store (which
>>> triggers that fetch)?
>>>
>>> Thanks,
>>> Brian Roach
>>>
>>> On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel
>>> <ingo.rockel at bluelionmobile.com> wrote:
>>>>
>>>> Hi Dmitri,
>>>>
>>>> it doesn't happen in my code and it does happen while the riak-client
>>>> tries
>>>> to deserialize a fetched object from riak in a "fetch-before-store" (see
>>>> the
>>>> stack), I also get this error randomly while trying just to fetch an
>>>> object
>>>> from the database.
>>>>
>>>> And if I try to fetch the object from the cmdline I just get a 404. So I
>>>> would expect the java-client just returns a null-result for this fetch
>>>> and
>>>> not to throw an exception.
>>>>
>>>> All my objects are stored using the riak-java-client and the
>>>> json-serializer.
>>>>
>>>> Ahh, just tested: if I retry it sometimes works, although most of the
>>>> time
>>>> still fails (haven't tried with a sleep so far).
>>>>
>>>> Ingo
>>>>
>>>> Am 30.01.2013 16:57, schrieb Dmitri Zagidulin:
>>>>>
>>>>>
>>>>> Hi Ingo.
>>>>>
>>>>> It's difficult to diagnose the exact reason without looking at your
>>>>> code.
>>>>> But that error is a JSON parser error. It gets thrown whenever the code
>>>>> tries to parse an empty string as a json object.
>>>>> The general-case solution is to validate your strings or input streams
>>>>> that you're turning into JSON objects, or to catch an exception when
>>>>> creating that object and deal with it accordingly.
>>>>>
>>>>> But again, it's hard to say why it's happening exactly, in your case --
>>>>> try to determine where in your code that's happening and think of ways
>>>>> some input or result is empty, and check for that.
>>>>>
>>>>> Dmitri
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel
>>>>> <ingo.rockel at bluelionmobile.com <mailto:ingo.rockel at bluelionmobile.com>>
>>>>>
>>>>> wrote:
>>>>>
>>>>>       Hi,
>>>>>
>>>>>       I wrote a java tool to convert part of our data from a
>>>>>       mysql-database into riak. As this tool is running while our system
>>>>>       is still up, it needs to replay all modifications done in the mysql
>>>>>       database, during these modifications I sometimes get this exception
>>>>>       from the riak client:
>>>>>
>>>>>       com.basho.riak.client.convert.__ConversionException:
>>>>>
>>>>>       java.io.EOFException: No content to map to Object due to end of
>>>>> input
>>>>>       com.basho.riak.client.convert.__ConversionException:
>>>>>
>>>>>       java.io.EOFException: No content to map to Object due to end of
>>>>> input
>>>>>                at
>>>>>
>>>>>
>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167)
>>>>>                at
>>>>>
>>>>>
>>>>> com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110)
>>>>>                at
>>>>>
>>>>>
>>>>> com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112)
>>>>>                at
>>>>>
>>>>>
>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264)
>>>>>                at
>>>>>
>>>>>
>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138)
>>>>>                at
>>>>>
>>>>>
>>>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205)
>>>>>                at
>>>>>
>>>>>
>>>>> com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243)
>>>>>                at java.lang.Thread.run(Thread.__java:722)
>>>>>
>>>>>       Caused by: java.io.EOFException: No content to map to Object due to
>>>>>       end of input
>>>>>                at
>>>>>
>>>>>
>>>>> org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775)
>>>>>                at
>>>>>
>>>>>
>>>>> org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718)
>>>>>                at
>>>>>
>>>>>
>>>>> org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863)
>>>>>                at
>>>>>
>>>>>
>>>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156)
>>>>>
>>>>>                ... 7 more
>>>>>
>>>>>       it only happens once every few thousand updates and if I check the
>>>>>       object from the cmdline I only get a 404.
>>>>>
>>>>>       Any ideas what might cause this and how to fix/workaround it?
>>>>>
>>>>>       Ingo
>>>>>
>>>>>
>>>>>       _________________________________________________
>>>>>       riak-users mailing list
>>>>>       riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>>>>
>>>>> http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com
>>>>>
>>>>> <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>
>>>>
>>>> --
>>>> Software Architect
>>>>
>>>> Blue Lion mobile GmbH
>>>> Tel. +49 (0) 221 788 797 14
>>>> Fax. +49 (0) 221 788 797 19
>>>> Mob. +49 (0) 176 24 87 30 89
>>>>
>>>> ingo.rockel at bluelionmobile.com
>>>>>>>
>>>>>>> qeep: Hefferwolf
>>>>
>>>>
>>>> www.bluelionmobile.com
>>>> www.qeep.net
>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>> --
>> Software Architect
>>
>> Blue Lion mobile GmbH
>> Tel. +49 (0) 221 788 797 14
>> Fax. +49 (0) 221 788 797 19
>> Mob. +49 (0) 176 24 87 30 89
>>
>> ingo.rockel at bluelionmobile.com
>>>>> qeep: Hefferwolf
>>
>> www.bluelionmobile.com
>> www.qeep.net


-- 
Software Architect

Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89

ingo.rockel at bluelionmobile.com
 >>> qeep: Hefferwolf

www.bluelionmobile.com
www.qeep.net




More information about the riak-users mailing list