Two quick questions about X-RIak-Meta-* headers....

Dmitri Zagidulin dzagidulin at basho.com
Mon Dec 7 12:33:48 EST 2015


Hi Joe,

1. Yes, it's possible (with the HTTP HEAD request, or the client library
equivalent (I'm pretty sure all the client libraries expose the 'return
only the headers' part of the object fetch -- see the Optional Parameters
head=true section of the PB API
http://docs.basho.com/riak/latest/dev/references/protocol-buffers/fetch-object/
)).

However, this is not going to be very helpful in your case. Behind the
scenes, a HEAD request still requests all replicas of an object -- *full*
replicas, including the value. It's just that the node coordinating the
request drops the actual object value before returning the metadata/headers
to the client.
So, if you use this 'just give me the metadata' request, you're only saving
on the cost of shipping the object value down the wire from the cluster to
the client. But you're still incurring the cost of all 3 copies of the
object (so, 3-4MB, in your case) being transferred over the network between
the nodes, as a result of that HEAD request.

2. I don't know that there is a specific size limit on the object header
values. However, this question is definitely a red flag -- it's very likely
that you're trying to use the custom headers in a way that they weren't
intended for.

Can you describe in more detail what's your use case? (As in, what are you
trying to store as metadata, and why would retrieving just the headers be
useful to you.)

Don't forget, that if you need to store a LOT of metadata on an object, and
you can't do it within the object value itself (for example, when storing
binary images), you can simply store a separate metadata object, in a
different bucket, using the same key as the object.
For example, if I'm storing my primary objects in the 'blobs' bucket, I can
also store a JSON object with corresponding metadata in a 'blobs-meta'
object, like so:

/buckets/blobs/keys/blob123   -->  binary object value
/buckets/blobs-meta/keys/blob123   --> json metadata object

The downsides of this setup is that you're now doing 2 writes for each
object (one to the blobs bucket, and one to the meta bucket).
But the benefits are considerable:

- You can store arbitrarily large metadata object (you're not abusing
object headers by stuffing large values into them)
- Since the metadata object is likely to be much smaller than the object
it's referring to, you can use the metadata object to check for an object's
existence (or to get the actual headers that you care about) without the
cost of requesting the full giant blob.

Dmitri



On Mon, Nov 30, 2015 at 12:18 PM, Joe Olson <technology at nododos.com> wrote:

> Two quick questions about X-RIak-Meta-* headers:
>
> 1. Is it possible to pull the headers for a key without pulling the key
> itself? The reason I am interested in this is because the values for our
> keys are in the 1.2-1.6 MB range, so the headers are a lot smaller in
> comparison. I know I can index the headers using Solr or 2i, but I am
> trying to involve the overhead of doing that.
>
> 2. What are the size limits on the headers values that are strings?
>
> As always, thanks in advance!
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20151207/d51d816f/attachment-0002.html>


More information about the riak-users mailing list