Proper data filtration and pagination

Anton Yakushin anton at istylemyself.com
Mon Jan 13 08:46:24 EST 2014


Hi,

Could anyone advice on how to properly organize data structure so
it could be efficiently filtered and results paginated?

A structure example from our current system.
We have products stored in the following JSON:
{
    "name": <string>
    "category": {
        "key" : <string>,
        "name": <string>
    },
    "sub_categories": [
        {
            "key" : <string>,
            "name": <string>
        }
    ],
    ... lots of other attributes: strings, arrays, objects.
}

What we need is be able to fetch filtered by their attributes
products with paginated results.

What we have tried:
1. Implemented filtration using key filters. So currently product
keys look like: "<unique_id>-c:<category_key>-sc:<subcategory_key>..."
Disadvantages were that the keys could possibly change and they
had quite a large length. Also it was hard to add a new attribute to a key.

2. MapReduce was used to filter by all attributes that were not in
product keys. Also results from key filters were passed through
MapReduce to decrease amount of requests.

3. SecondaryIndexes were used for some attributes. But it's very
inconvenient that one can't filter by several indexes at a time.

4. At the first moment seemed that Riak Search could solve all these
problems, but it transforms product objects into a flat structure
making them hardly usable. Case with Search + MapReduce was too slow
and didn't allow to paginate data.

5. Pagination with MapReduce was too slow and put exceptional load
on our servers.

So the issue which holds our system from growing and feature
increasing is how should the data be re-organized to make them easily and
efficiently
filtered with pagination?

Would highly appreciate any suggestions. It's really a big issue now
so we even think about switching to another database (e.g. MongoDB) if
we would not find a suitable solution in Riak.

Best regards,
Anton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140113/b2523c36/attachment.html>


More information about the riak-users mailing list