Riak Search VS other query systems
sargun at sargun.me
Wed Aug 20 22:28:20 EDT 2014
I second John's opinions. Generally, I would have have one key which
is the secondary index, being an observe-remove OR-Set (or a relevant
type for your application, be a register, g-set, or a plain old
OR-set) pointing to back to the keys. Unfortunately, this mechanism
can become quite unwieldy in when you have a term with a high
Now, moving onto the Twitter use case, you care a lot about the speed.
With this strategy, if you're doing this from a client where you (1)
read the 2i OR-set, and then (2) read the keys, that can be expensive
as you have to read the entire 2i value back to the Riak client before
reading any of the keys. An example, the hashtag #beiber, would have
high cardinality would result in a super large value, and reading that
back over the network would be less than awesome. Also, having to pass
this value around disterl would be poor. Fortunately, the folks at
Basho have invented riak_pipe. Riak_pipe is a method to allow running
the read locally on the node the 2i lives on, and then streaming reads
for all of those keys to the nodes that they live on, and then all
back to the reader. It's actually the framework that Riak MR uses
under the hood.
Talk: https://vimeo.com/53910999 (there might be newer ones as well)
Also, to deal with high-cardinality values, there are a variety of
work arounds, such as sharding the secondary index to some known set
of keys, and doing a read across these list of keys. Also, you can
postfix a nonce to the 2i-key, and ensure that they all end up on one
node (custom hashing function), or a subset of nodes, and utilize
leveldb's key iteration over a range to handle this.
The general patterns I like for 2i atop Riak is to specialize Peter
Bailis, from UC Berkeley's work for RAMPs. If you build the framework
for this, it'll be all sorts of useful in the future. One warning is
that there is no easy way to garbage collect in Riak today.
None of these methods gracefully handle range queries. You can do
clever things with your 2i to handle this, but it the Twitter use case
wouldn't need ranges.
On Wed, Aug 20, 2014 at 12:28 PM, John Daily <jdaily at basho.com> wrote:
> I don't have benchmarks to discuss query performance for different tools at
> different sizes, but I'd like to point out that the ultimate search tool for
> Riak is to not search at all.
> Riak Search, 2i, MapReduce are all capable tools, but they don't scale
> nearly as well as straight key/value requests, and it is often possible to
> model your data around the latter.
> I covered this in https://basho.com/riak-development-anti-patterns/ and the
> next edition of Eric Redmond's Little Riak Book (http://littleriakbook.com)
> will have more discussion on the topic, but if at all possible, create your
> query results as reports as the data is ingested, instead of attempting to
> find it all later.
> On Wed, Aug 20, 2014 at 3:21 PM, Alex De la rosa <alex.rosa.box at gmail.com>
>> Any thoughts about this?
>> One thing it worries me about Riak Search is that if one index has several
>> millions of object to search for maybe it becomes slow? 2i might be faster
>> On Tue, Aug 19, 2014 at 8:47 AM, Alex De la rosa <alex.rosa.box at gmail.com>
>>> Hi there,
>>> I had been seeing lately Riak Search as an ultimate way to query Riak...
>>> and it seems recommended to use over MapReduce and even 2i... said so...
>>> should we try to always use Riak Search over the other systems?
>>> Is there any situation in which MapReduce could be a better approach than
>>> Riak Search?
>>> Same goes for 2i... I believe 2i is an optimal approach if you just want
>>> keys and know very well what are you looking for, but out of that, should
>>> Riak Search try to replace all 2i uses?
>>> Practical example: If you are twitter and want to get twits for the
>>> hashtag #Riak, what would be the best approach? 2i? Riak Search? MapReduce?
>> riak-users mailing list
>> riak-users at lists.basho.com
> riak-users mailing list
> riak-users at lists.basho.com
More information about the riak-users