Strange Exception in java-client

Ingo Rockel ingo.rockel at bluelionmobile.com
Thu Jan 31 05:56:02 EST 2013


Hi Brian,

thanks for the detailed explaination!

I had a look at an object which constantly fails to load even if retrying:

lftp :~> cat "http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081"
---- Verbinde mit 172.22.3.14 (172.22.3.14) Port 8091
---> GET /riak/m/Um|18498012|4|0|18298081 HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 300 Multiple Choices
<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept, Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
<--- ETag: "6PSreYIOL25KOpNyG0XPe7"
<--- Date: Thu, 31 Jan 2013 10:42:41 GMT
<--- Content-Type: text/plain
<--- Content-Length: 56
<---
<--* Siblings:
<--* 50Uz9nvQWwOUBE6USi2gki
<--* 1JsgLs3CE3k2mWsaCEiPp4
cat: Zugriff nicht möglich: 300 Multiple Choices 
(/riak/m/Um|18498012|4|0|18298081)
lftp :~> cat 
"http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki"
---> GET /riak/m/Um|18498012|4|0|18298081?vtag=50Uz9nvQWwOUBE6USi2gki 
HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 200 OK 

<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Link: </riak/m>; rel="up"
<--- Last-Modified: Thu, 31 Jan 2013 09:57:38 GMT
<--- ETag: "50Uz9nvQWwOUBE6USi2gki"
<--- Date: Thu, 31 Jan 2013 10:42:49 GMT
<--- Content-Type: application/octet-stream
<--- Content-Length: 0
<---
lftp :~> cat 
"http://172.22.3.14:8091/riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4"
---> GET /riak/m/Um|18498012|4|0|18298081?vtag=1JsgLs3CE3k2mWsaCEiPp4 
HTTP/1.1
---> Host: 172.22.3.14:8091
---> User-Agent: lftp/4.3.3
---> Accept: */*
---> Connection: keep-alive
--->
<--- HTTP/1.1 200 OK 

<--- X-Riak-Vclock: a85hYGBgzGDKBVIcaZPWMQZyWttkMCWy5rEyXNhTd4ovCwA=
<--- Vary: Accept-Encoding
<--- Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
<--- Link: </riak/m>; rel="up"
<--- Last-Modified: Thu, 31 Jan 2013 10:00:48 GMT
<--- ETag: "1JsgLs3CE3k2mWsaCEiPp4"
<--- Date: Thu, 31 Jan 2013 10:43:01 GMT
<--- Content-Type: application/json; charset=UTF-8
<--- Content-Length: 114
<---
{"sortKey":1359626448000106,"st":2,"t":4,"r":18498012,"s":18298081,"ct":1359626448000,"rv":21215685,"cv":1,"su":0}

the object has two siblings, one the deleted empty "tombstone" and one 
with the new data. And there's a gap auf 2:30min between both siblings. 
There's no immediate write after the deletion. I logged the write 
operations and this gap is there also. And the java client constantly 
fails to load this object.

This objects are user notifications which are identified by their type 
(the key) and can be updated frequently but the user also is allowed to 
delete a single notification (which he did in this case).

Is there anything I can do to resolve this? A last write wins just for 
this case? If I write a new notification to update the old one I don't 
care if there's any deletion tombstone...

Ingo

Am 30.01.2013 22:42, schrieb Brian Roach:
> Ingo -
>
> Riak is returning an object with no contents (which ends up being an
> empty String passed to Jackson).
>
> Unless you've somehow mangled the data yourself (which sounds unlikely
> given the bit about the 404 from the command line; more on that in a
> bit) what's happening is that you're encountering a tombstone; an
> object that has been deleted via a delete operation but hasn't been
> removed yet. This causes an "empty" object to be returned (the
> tombstone) and causes Jackson to puke (HTTP will actually return this
> as a 404 but if you look there's still a X-Riak-Vclock: header with a
> vclock).
>
> Probably the best description of how this works in Riak is a post by
> Jon Meredith which can be found here:
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html
>
> Unfortunately this is something the Java client doesn't know what to
> do with when using the default JSONConverter and your own POJOs. And
> it's not as simple as "just return null" because of a case where a
> tombstone could actually be a sibling and the client then needs to
> resolve the conflict which is the next step in the process. It's
> something I'm going to have to think about.
>
> As you've discovered, when the tombstone isn't a sibling simply
> retrying will often work because by then the delete has fully
> completed and the tombstones have been removed from the Riak nodes.
>
> Is there a reason you're rapidly doing a delete then a store (which
> triggers that fetch)?
>
> Thanks,
> Brian Roach
>
> On Wed, Jan 30, 2013 at 9:06 AM, Ingo Rockel
> <ingo.rockel at bluelionmobile.com> wrote:
>> Hi Dmitri,
>>
>> it doesn't happen in my code and it does happen while the riak-client tries
>> to deserialize a fetched object from riak in a "fetch-before-store" (see the
>> stack), I also get this error randomly while trying just to fetch an object
>> from the database.
>>
>> And if I try to fetch the object from the cmdline I just get a 404. So I
>> would expect the java-client just returns a null-result for this fetch and
>> not to throw an exception.
>>
>> All my objects are stored using the riak-java-client and the
>> json-serializer.
>>
>> Ahh, just tested: if I retry it sometimes works, although most of the time
>> still fails (haven't tried with a sleep so far).
>>
>> Ingo
>>
>> Am 30.01.2013 16:57, schrieb Dmitri Zagidulin:
>>>
>>> Hi Ingo.
>>>
>>> It's difficult to diagnose the exact reason without looking at your code.
>>> But that error is a JSON parser error. It gets thrown whenever the code
>>> tries to parse an empty string as a json object.
>>> The general-case solution is to validate your strings or input streams
>>> that you're turning into JSON objects, or to catch an exception when
>>> creating that object and deal with it accordingly.
>>>
>>> But again, it's hard to say why it's happening exactly, in your case --
>>> try to determine where in your code that's happening and think of ways
>>> some input or result is empty, and check for that.
>>>
>>> Dmitri
>>>
>>>
>>>
>>> On Wed, Jan 30, 2013 at 10:44 AM, Ingo Rockel
>>> <ingo.rockel at bluelionmobile.com <mailto:ingo.rockel at bluelionmobile.com>>
>>>
>>> wrote:
>>>
>>>      Hi,
>>>
>>>      I wrote a java tool to convert part of our data from a
>>>      mysql-database into riak. As this tool is running while our system
>>>      is still up, it needs to replay all modifications done in the mysql
>>>      database, during these modifications I sometimes get this exception
>>>      from the riak client:
>>>
>>>      com.basho.riak.client.convert.__ConversionException:
>>>
>>>      java.io.EOFException: No content to map to Object due to end of input
>>>      com.basho.riak.client.convert.__ConversionException:
>>>
>>>      java.io.EOFException: No content to map to Object due to end of input
>>>               at
>>>
>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:167)
>>>               at
>>>
>>> com.basho.riak.client.__operations.FetchObject.__execute(FetchObject.java:110)
>>>               at
>>>
>>> com.basho.riak.client.__operations.StoreObject.__execute(StoreObject.java:112)
>>>               at
>>>
>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__storeUniqueMessageDto(__MessageKVImpl.java:264)
>>>               at
>>>
>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__createDataFromDTO(__MessageKVImpl.java:138)
>>>               at
>>>
>>> com.bluelionmobile.qeep.__messaging.db.impl.__MessageKVImpl.__updateDataFromDTO(__MessageKVImpl.java:205)
>>>               at
>>>
>>> com.bluelionmobile.qeep.__messaging.db.utils.Replay$__ReplayRunner.run(Replay.java:__243)
>>>               at java.lang.Thread.run(Thread.__java:722)
>>>
>>>      Caused by: java.io.EOFException: No content to map to Object due to
>>>      end of input
>>>               at
>>>
>>> org.codehaus.jackson.map.__ObjectMapper._initForReading(__ObjectMapper.java:2775)
>>>               at
>>>
>>> org.codehaus.jackson.map.__ObjectMapper._readMapAndClose(__ObjectMapper.java:2718)
>>>               at
>>>
>>> org.codehaus.jackson.map.__ObjectMapper.readValue(__ObjectMapper.java:1863)
>>>               at
>>>
>>> com.basho.riak.client.convert.__JSONConverter.toDomain(__JSONConverter.java:156)
>>>
>>>               ... 7 more
>>>
>>>      it only happens once every few thousand updates and if I check the
>>>      object from the cmdline I only get a 404.
>>>
>>>      Any ideas what might cause this and how to fix/workaround it?
>>>
>>>      Ingo
>>>
>>>
>>>      _________________________________________________
>>>      riak-users mailing list
>>>      riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>>      http://lists.basho.com/__mailman/listinfo/riak-users___lists.basho.com
>>>      <http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>>
>> --
>> Software Architect
>>
>> Blue Lion mobile GmbH
>> Tel. +49 (0) 221 788 797 14
>> Fax. +49 (0) 221 788 797 19
>> Mob. +49 (0) 176 24 87 30 89
>>
>> ingo.rockel at bluelionmobile.com
>>>>> qeep: Hefferwolf
>>
>> www.bluelionmobile.com
>> www.qeep.net
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


-- 
Software Architect

Blue Lion mobile GmbH
Tel. +49 (0) 221 788 797 14
Fax. +49 (0) 221 788 797 19
Mob. +49 (0) 176 24 87 30 89

ingo.rockel at bluelionmobile.com
 >>> qeep: Hefferwolf

www.bluelionmobile.com
www.qeep.net




More information about the riak-users mailing list