mr_queue gone wild

Sylvain Niles sylvain.niles at gmail.com
Tue Jul 12 21:56:36 EDT 2011


Thanks again for all the pointers. After much digging I found the
issue. We have UTF8 validation on the most data we put into riak (our
events bucket), but weren't validating UTF8 on some objects that are
linked to by many other objects (group bucket). Consequently some
malformed UTF8 (that displays fine via console, it's only spidermonkey
that has a problem) causes spidermonkey to die a horrible death when
it pulls an object that has a bad link. Does anyone have some example
code for returning a list of keys that have a common link? In this
case we only had one-way links.
Example:

Object1  ---link--->  BadUTF8.Object
Object2  ---link--->  BadUTF8.Object

I'll have to do this via the erlang client as doing it in Javascript
crashes every time :(

Any ideas?

PS: Is there any plan to make bad UTF8 not a riak-killer? This is the
second time it's bitten us and all of the examples we've seen don't
cause ruby or erlang any problems, only spidermonkey.

Bad UTF8: <<"Afro-Belly Boogie®  Fitness and Wellness-1800400">>


-Sylvain


On Tue, Jul 5, 2011 at 12:37 PM, Bryan Fink <bryan at basho.com> wrote:
> On Thu, Jun 30, 2011 at 4:52 PM, Sylvain Niles <sylvain.niles at gmail.com> wrote:
>> Is there a way to list the m/r jobs in the queue in case there's
>> something else going on? Is there a reason they never get removed?
>
> Hi, Sylvain.  As Aphyr noted, the mr_queue is a bitcask.  Because
> bitcask is append-only storage, its size alone does not give a good
> indication of the active dataset.  More specifically, unless you have
> tweaked your bitcask parameters, you can expect the mr_queue directory
> to grow to at least 2GB before purging unused data.  This is normal
> and expected.
>
> A few things worth noting:
>
>  - The mr_queue is only used for Javascript MapReduce phases.  Erlang
>   phases never touch it.  This is because there are a limited number
>   of Javascript VMs available on a node, and all vnodes compete for
>   them.  The mr_queue provides a place to offload the backlog of
>   pending requeusts for Javascript interpreters, rather than keeping
>   them in memory.
>
>  - To check the depth of the mr_queue, connect to any Riak node's
>   console (bin/riak attach), and call
>   riak_kv_map_master:queue_depth/0.  It should show you the active
>   depth of the mr_queue on each node in the cluster:
>
>   $ bin/riak attach
>   (dev1 at 127.0.0.1)> riak_kv_map_master:queue_depth().
>   [{'dev1 at 127.0.0.1',0},
>    {'dev2 at 127.0.0.1',0},
>    {'dev3 at 127.0.0.1',0},
>    {'dev4 at 127.0.0.1',0}]
>
>   A result like the above means there are no pending map requests
>   waiting.  A result like the following means there are 242 map
>   requests waiting on the dev1 node, 128 on dev2, etc.:
>
>   [{'dev1 at 127.0.0.1',242},
>    {'dev2 at 127.0.0.1',128},
>    {'dev3 at 127.0.0.1',212},
>    {'dev4 at 127.0.0.1',230}]
>
>   These numbers may be slightly confusing because a single logical
>   map phase gets split into a separate map request for each vnode.
>   So, you may have only one MapReduce request outstanding, but see
>   numbers greater than 1 in this output.
>
> Hope this helps,
>
> Bryan Fink
> Senior Software Developer,
> Basho Technologies
>




More information about the riak-users mailing list