Proper data filtration and pagination

Anton Yakushin anton at
Wed Jan 15 09:01:30 EST 2014


Seems yokozuna is what I looked for.
The only problem is our project is already in production.
Are there any other ways like additional buckets serving as indexes or
something built-in to achieve similar functionality?
Is there a way in Riak to get values for several known keys without using

Many thanks for your quick response!

Best regards,

On Mon, Jan 13, 2014 at 7:39 PM, Eric Redmond <eredmond at> wrote:

> Anton,
> Depending on how soon you plan to be in production, this sounds like a
> good usecase for yokozuna (the new Riak Search) coming in 2.0 (sometime in
> Q1). It has builtin support for handling semistructured data like JSON or
> XML, nested even, and will allow you to query by multiple fields at once.
> The most recent version also has excellent pagination support.
> To toy with it right now, you can find more information on
> Eric
> On Jan 13, 2014, at 5:46 AM, Anton Yakushin <anton at>
> wrote:
> Hi,
> Could anyone advice on how to properly organize data structure so
> it could be efficiently filtered and results paginated?
> A structure example from our current system.
> We have products stored in the following JSON:
> {
>     "name": <string>
>     "category": {
>         "key" : <string>,
>         "name": <string>
>     },
>     "sub_categories": [
>         {
>             "key" : <string>,
>             "name": <string>
>         }
>     ],
>     ... lots of other attributes: strings, arrays, objects.
> }
> What we need is be able to fetch filtered by their attributes
> products with paginated results.
> What we have tried:
> 1. Implemented filtration using key filters. So currently product
> keys look like: "<unique_id>-c:<category_key>-sc:<subcategory_key>..."
> Disadvantages were that the keys could possibly change and they
> had quite a large length. Also it was hard to add a new attribute to a key.
> 2. MapReduce was used to filter by all attributes that were not in
> product keys. Also results from key filters were passed through
> MapReduce to decrease amount of requests.
> 3. SecondaryIndexes were used for some attributes. But it's very
> inconvenient that one can't filter by several indexes at a time.
> 4. At the first moment seemed that Riak Search could solve all these
> problems, but it transforms product objects into a flat structure
> making them hardly usable. Case with Search + MapReduce was too slow
> and didn't allow to paginate data.
> 5. Pagination with MapReduce was too slow and put exceptional load
> on our servers.
> So the issue which holds our system from growing and feature
> increasing is how should the data be re-organized to make them easily and
> efficiently
> filtered with pagination?
> Would highly appreciate any suggestions. It's really a big issue now
> so we even think about switching to another database (e.g. MongoDB) if
> we would not find a suitable solution in Riak.
> Best regards,
> Anton
> _______________________________________________
> riak-users mailing list
> riak-users at
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the riak-users mailing list