Riak Java client not returning deleted sibling

Daniel Iwan iwan.daniel at gmail.com
Tue Oct 8 09:47:55 EDT 2013


I still did not figure it out what's happening.
I wrote a simple test that uses 3 Riak Java clients running on single
machine. Each client connects to a different Riak node and uses the same
converters, mutators etc
as original code. I'm explicitly setting clientId and started updating
single key.

I can see that siblings are generated and resolved and I always get full
set of meta that is merged correctly during resolve

At the end of test I can see there are still 4-6 siblings that are resolved
but not resolved value not stored (that's what test does)
So in the end every client can see all updates and I cannot reproduce
problem I see when clients are running directly on Riak nodes.

For me it looks like data loss as after store each client can see only it's
local changes, and curl shows there are no siblings.
Next fetch using Java client shows the same value as curl.

I'm using N=3, storing and reading with quorum on r, w and rw, bacisQuorum
is false.
I would think with those settings clients should see changes from all
clients.

Any idea why it's not happening and also why I cannot reproduce that with
unit test?

Thanks
Daniel
















I see that




On 7 October 2013 21:21, Daniel Iwan <iwan.daniel at gmail.com> wrote:

> I tested that with curl. Should've mentioned that.
> The output shows there is no siblings for the key and returned header
> looks like this:
>
> < HTTP/1.1 200 OK
> < X-Riak-Vclock:
> a85hYGBgymDKBVIc84WrPgUFPS3IYEpkymNlWJZ4/DQfVIpNStsDKsUIlFoOksoCAA==
> < x-riak-index-oid_bin: 2a9691be-2f70-11e3-b746-c91c38008704-0
> < x-riak-index-bucket_bin: 2a9691be-2f70-11e3-b746-c91c38008704
> < Vary: Accept-Encoding
> < Server: MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue)
> < my_meta: 5c3ff653-21cb-11e3-96a0-9a712d0ac54a
> < Link: </riak/TX>; rel="up"
> < Last-Modified: Mon, 07 Oct 2013 16:47:35 GMT
> < ETag: "aNfE9KrKGTR7qCW2IDPuf"
> < Date: Mon, 07 Oct 2013 17:17:55 GMT
> < Content-Type: application/json
> < Content-Length: 2223
>
> I was thinking that my mutator ConditionalStoreMutation returns false in
> hasMutated()
> but then I should not see new vclock in result of Store right?
>
>
>  D.
>
>
>
>
>
>
> On 7 October 2013 20:12, Brian Roach <roach at basho.com> wrote:
>
>> Daniel -
>>
>> Unfortunately returning the body from a store operation may not
>> reflect all the replicas (and in the case of a concurrent write on two
>> different nodes "may not" really means  "probably doesn't").
>>
>> If you do a subsequent fetch after sending both your writes you'll get
>> back a single vclock with siblings.
>>
>> Thanks,
>> - Roach
>>
>> On Mon, Oct 7, 2013 at 12:37 PM, Daniel Iwan <iwan.daniel at gmail.com>
>> wrote:
>> > Hi Brian
>> >
>> > Thanks for update.
>> > I'm using 1.1.3 now and still have some issues sibling related
>> >
>> > Two clients are updating the same key.
>> > Updated is my custom meta field, which should be merged to contain
>> values
>> > from both clients (set)
>> > I see both client are doing fetch, resolving sibling (only 1 i.e. no
>> > siblings), apply mutation (their own values for meta field). After that
>> > object is converted using fromDomain() in my converter using vclock
>> provided
>> > Both nodes use vclock
>> > 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64cc6365589678fc345f1600
>> >
>> > So far so god.
>> > But the next step is toDomain (which is pare of Store I think since I'm
>> > using withBody) and looks like each node contains info only about
>> > it own changes.
>> > Client one sees vclock
>> > 6bce61606060cc60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc345f1600
>> > Client 2 sees vclock
>> >
>> 6bce61606060ca60ca05521cf385ab3e05053d2dc8604a64ca6365589678fc341f548a4d4adb032ac508945a0e92ca0200
>> >
>> > Both of vclocks are different than original vclock given during store,
>> which
>> > I assume means RIak accepted write.
>> > Resolve is called on both machines but there is only one sibling.
>> >
>> > I guess the fact that I'm changing only meta field should not matter
>> here
>> > and I should see 2 siblings?
>> > allow_multi is of course true and lastWriteWins is false on that bucket
>> >
>> > Any help much appreciated
>> >
>> >
>> > Regards
>> > Daniel
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On 4 October 2013 21:41, Brian Roach <roach at basho.com> wrote:
>> >>
>> >> Hey -
>> >>
>> >> I'm releasing 1.1.3 and 1.4.2 but it'll take a while for them to get
>> >> staged at maven central so I can post an "official" release to the
>> >> mailing list.
>> >>
>> >> I've gone ahead and uploaded the jar-with-dependencies to the usual
>> >> place for you-
>> >>
>> >>
>> >>
>> http://riak-java-client.s3.amazonaws.com/riak-client-1.1.3-jar-with-dependencies.jar
>> >>
>> >> It fixes up the DomainBucket stuff and the JSONConverter.
>> >>
>> >> Thanks,
>> >> - Roach
>> >>
>> >> On Fri, Oct 4, 2013 at 2:58 AM, Daniel Iwan <iwan.daniel at gmail.com>
>> wrote:
>> >> > Thanks Brian for putting fix together so quickly.
>> >> >
>> >> > I think I found something else though.
>> >> > In JSONConverter I don't see vclock being set in toDomain() when
>> >> > converting
>> >> > deleted sibling?
>> >> > That vclock should be used for following delete if I understood it
>> >> > correctly?
>> >> >
>> >> > Also where can I download latest build? I tried
>> >> >
>> >> >
>> http://riak-java-client.s3.amazonaws.com/riak-client-1.1.3-jar-with-dependencies.jar
>> >> > but access is denied
>> >> >
>> >> > Cheers
>> >> > Daniel
>> >> >
>> >> >
>> >> > On 3 October 2013 19:36, Brian Roach <roach at basho.com> wrote:
>> >> >>
>> >> >> On Thu, Oct 3, 2013 at 10:32 AM, Daniel Iwan <iwan.daniel at gmail.com
>> >
>> >> >> wrote:
>> >> >> > Thanks Brian for quick response.
>> >> >> >
>> >> >> > As a side question, what is the best way to delete such an object
>> >> >> > i.e.
>> >> >> > once
>> >> >> > I know one of the siblings has 'deleted' flag true because I
>> fetched
>> >> >> > it?
>> >> >> > Should I just use DomainBucket.delete(key) without providing any
>> >> >> > vclock?
>> >> >> > Would it wipe it from Riak or create yet another sibling?
>> >> >>
>> >> >> You should always use vclocks when possible, which in the case it
>> is.
>> >> >> There are additional issues around doing the delete without a vclock
>> >> >> and if there's concurrently a store operation occurring.
>> >> >>
>> >> >> Ideally you should look at why you're getting that tombstone
>> sibling.
>> >> >> If it's simply a case of high write concurrency and you're using
>> >> >> vclocks with your writes, then there's not much you can do except
>> >> >> resolve it later (without changing how you're using the DB)... but
>> >> >> usually these things are caused by writes without a vclock.
>> >> >>
>> >> >> Thanks,
>> >> >> - Roach
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> >
>> >> >> > Regards
>> >> >> > Daniel
>> >> >> >
>> >> >> >
>> >> >> > On 3 October 2013 17:20, Brian Roach <roach at basho.com> wrote:
>> >> >> >>
>> >> >> >> Daniel -
>> >> >> >>
>> >> >> >> Yeah, that is the case. When the ability to pass
>> fetch/store/delete
>> >> >> >> meta was added to DomainBucket way back when it appears that was
>> >> >> >> missed.
>> >> >> >>
>> >> >> >> I'll add it and forward-port to 1.4.x as well and cut new jars.
>> >> >> >> Should
>> >> >> >> be avail by tomorrow morning at the latest.
>> >> >> >>
>> >> >> >> Thanks!
>> >> >> >> - Roach
>> >> >> >>
>> >> >> >> On Thu, Oct 3, 2013 at 9:38 AM, Daniel Iwan <
>> iwan.daniel at gmail.com>
>> >> >> >> wrote:
>> >> >> >> > Hi I'm using Riak 1.3.1 and Java client 1.1.2
>> >> >> >> >
>> >> >> >> > Using http and curl I see 4 siblings for an object one of which
>> >> >> >> > has
>> >> >> >> > X-Riak-Deleted: true
>> >> >> >> > but when I'm using Java client with DomainBucket my Converter's
>> >> >> >> > method
>> >> >> >> > toDomain is called only 3 times.
>> >> >> >> >
>> >> >> >> > I have set the property
>> >> >> >> >
>> >> >> >> > builder.returnDeletedVClock(true);
>> >> >> >> >
>> >> >> >> > on my DomainBuilder which I keep reusing for all queries and
>> store
>> >> >> >> > operations (I guess that's good practice btw.?)
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > I run that under debugger and it seems raw client sees 4
>> siblings
>> >> >> >> > but
>> >> >> >> > passes
>> >> >> >> > over only 3 due to bug (I think) in DomainBucket.fetch() method
>> >> >> >> > which
>> >> >> >> > should
>> >> >> >> > have
>> >> >> >> >
>> >> >> >> >         if (fetchMeta.hasReturnDeletedVClock()) {
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > so.returnDeletedVClock(fetchMeta.getReturnDeletedVClock());
>> >> >> >> >
>> >> >> >> >         }
>> >> >> >> >
>> >> >> >> > at the end, as store() method has.
>> >> >> >> >
>> >> >> >> > Could you confirm or I'm I completely wrong?
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Regards
>> >> >> >> >
>> >> >> >> > Daniel
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > _______________________________________________
>> >> >> >> > riak-users mailing list
>> >> >> >> > riak-users at lists.basho.com
>> >> >> >> >
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131008/e317e72f/attachment.html>


More information about the riak-users mailing list