full-writes vs. merges
bryan at basho.com
Mon Oct 26 09:53:21 EDT 2009
On Sun, Oct 25, 2009 at 11:21 PM, Brian Hammond <brian at brianhammond.com> wrote:
> If I want to update the links associated with some stored value, I have to
> first read the value, update the links, and write back the entire value.
> I'm concerned if that my data has "large" values that this
> read-update-write technique will cause a fair amount of overhead.
> I am wondering if there's any merit to this concern in practice. I suppose
> the answer is dependent on one's definition of large, network configuration,
> etc. Anyway, perhaps you guys can keep it in the back of your head that
> perhaps CRUD operations on links might be useful.
> More generically, perhaps some form of "JSON object *merging*" would be in
Hi, Brian. I see exactly where you're coming from. The main problem,
though, is the need to maintain vclocks so causality can be determined
in the case of conflicting data. This means that you *have* to
perform a read so you know what vclock to use when you perform your
I can think of two ways to avoid the large-data overhead, though.
One option is to keep the links for that large piece of data in a
completely separate object. That way you only have to read the
links-object in order to change links. Having done this a couple of
times, I found it convenient to put links objects in a separate
bucket, then use the same key as the data object. So data would live
in /mydata/KEY, and its links would live in /mydata-links/KEY. This
meant that there was no lookup necessary to determine where that
secondary links object was.
Another option is to take advantage of the sibling creation that
happens when Riak encounters non-descending vclocks. That is, if you
just perform a write with an empty vclock (or any other form of
non-descendant vclock), Riak will consider that write to not be
descended from a value already stored. When a read is performed on
that key, Riak will hand the client both the original value and the
newly-written value. At that point, your application could merge the
links lists is the appropriate fashion. Note: this will require
setting the 'allow_mult' bucket property to 'true', and possibly also
implementing a Jiak bucket module if you're using the HTTP interface
(see jiak_example.erl). There are many quirks to this approach, so
beware that it may not work for you.
We have also talked about exposing ways to read/write only certain
parts of Riak objects (metadata, data, links), but we haven't
implemented the functionality end-to-end yet. It's great feedback to
hear from someone who is thinking about the same problem.
More information about the riak-users