Migration from memcachedb to riak

Edgar Veiga edgarmveiga at gmail.com
Wed Jul 10 05:35:00 EDT 2013


Hi Damien,

We have ~1100000000 keys and we are using ~2TB of disk space.
(The average object length will be ~2000 bytes).

This is a lot to fit in memory (We have bad past experiencies with
couchDB...).

Thanks for the rest of the tips!


On 10 July 2013 10:13, damien krotkine <dkrotkine at gmail.com> wrote:

>
> ( first post here, hi everybody... )
>
> If you don't need MR, 2i, etc, then BitCask will be faster. You just need
> to make sure all your keys fit in memory, which should not be a problem.
> How many keys do you have and what's their average length ?
>
> About the values,you can save a lot of space by choosing an appropriate
> serialization. We use Sereal[1] to serialize our data, and it's small
> enough that we don't need to compress it further (it can automatically use
> snappy to compress further). There is a php client [2]
>
> If you use leveldb, it can compress using snappy, but I've been a bit
> disappointed by snappy, because it didn't work well with our data. If you
> serialize your php object as verbose string (I don't know what's the usual
> way to serialize php objects), then you should probably benchmark different
> compressions algorithms on the application side.
>
>
> [1]: https://github.com/Sereal/Sereal/wiki/Sereal-Comparison-Graphs
> [2]: https://github.com/tobyink/php-sereal/tree/master/PHP
>
> On 10 July 2013 10:49, Edgar Veiga <edgarmveiga at gmail.com> wrote:
>
>>  Hello all!
>>
>> I have a couple of questions that I would like to address all of you
>> guys, in order to start this migration the best as possible.
>>
>> Context:
>> - I'm responsible for the migration of a pure key/value store that for
>> now is being stored on memcacheDB.
>> - We're serializing php objects and storing them.
>> - The total size occupied it's ~2TB.
>>
>> - The idea it's to migrate this data to a riak cluster with elevelDB
>> backend (starting with 6 nodes, 256 partitions. This thing is scaling very
>> fast).
>> - We only need to access the information by key. *We won't need neither
>> map/reduces, searches or secondary indexes*. It's a pure key/value store!
>>
>> My questions are:
>> - Do you have any riak fine tunning tip regarding this use case (due to
>> the fact that we will only use the key/value capabilities of riak)?
>> - It's expected that those 2TB would be reduced due to the levelDB
>> compression. Do you think we should compress our objects to on the client?
>>
>> Best regards,
>> Edgar Veiga
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130710/59ceefc5/attachment.html>


More information about the riak-users mailing list