Fwd: Map Reduce Requirements

bill robertson billrobertson42 at gmail.com
Mon Aug 22 15:19:08 EDT 2011


Forgot to reply-all...

---------- Forwarded message ----------
From: bill robertson <billrobertson42 at gmail.com>
Date: Mon, Aug 22, 2011 at 3:18 PM
Subject: Re: Map Reduce Requirements
To: Jeremiah Peschka <jeremiah.peschka at gmail.com>


That makes sense.

Suppose I have a query called Q1. I would like to specify Q1 in Javascript.
Assume that I can write an Erlang function called F that will translate the
raw GPB bytes into the appropriate JSON for use by Q1. How would I hook F
into the processing of Q1?

I guess that the Javascript function be passed the GPB bytes in the reduce
phase at which point I could call my translation function and operate on the
JSON, and possibly pass on a structure containing the JSON and the GPB to
the next phase.

Does that make sense? Is it possible to invoke arbitrary Erlang functions
within Javascript like this? If so, are there examples?

Additionally, are secondary indexes meta-data?  i.e. If I built some
secondary indices, these are stored in some form internal to Riak, and
therefore available for query regardless of the type of data its associated
with. Is this correct?

Thanks,
Bill Robertson

On Mon, Aug 22, 2011 at 2:57 PM, Jeremiah Peschka <
jeremiah.peschka at gmail.com> wrote:

> You can MR across whatever kind of data you'd like. JSON is typically used
> because it's very easy to show people how to query JSON and the structure
> makes sense to many programmers.
>
> To MR across anything else, you'll want a library that will translate your
> protocol buffers encoded data into objects that can be parsed in either
> JavaScript or Erlang. That is to say that you'll need a
> Serialization/Deserialization function to translate between data at rest
> (protobufs) to data that the MR program can understand.
>
> Since there are protocol buffer libraries for many languages, this should
> be doable in either JavaScript or Erlang. I don't know of any examples, but
> it shouldn't be much more difficult than Riak.mapValuesJson - provided that
> you can find some easy magic to translate objects for you ;)
> ---
> Jeremiah Peschka - Founder, Brent Ozar PLF, LLC
> Microsoft SQL Server MVP
>
> On Aug 22, 2011, at 11:51 AM, bill robertson wrote:
>
> > In order to run a map reduce query v.s. Riak, does the data need to be
> stored in JSON? If this isn't a requirement, then how would I run a query
> against data stored in a google protocol buffer format? Is there an example
> of this somewhere?
> >
> > Thanks!
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110822/f7063221/attachment.html>


More information about the riak-users mailing list