Migration from memcachedb to riak

Edgar Veiga edgarmveiga at gmail.com
Wed Jul 10 07:04:23 EDT 2013


Hi Damien,

Well let's dive into this a little bit.

I told you guys that bitcask was not an option due to a bad past
experiencie with couchbase (sorry, in the previous post I wrote couchdb),
that uses the same architecture as bitcask, keys in memory and values in
disk.

We started the migration to couchbase, and were already using 3 physical
nodes and only had imported 5% of the real data! That was one of the main
reasons for choosing a solution like riak + leveldb, to delete the keys fit
in memory bottleneck..

Now, here's a tipical key (It's large :D, x=letter and 0=numbers ):

xxxxxxx_xxxxxxxx_xxxxxxxx_00000xxxxx_0000000000_00000000000_0000000xxxxxx

We are using the php serialize native function!

Best regards



On 10 July 2013 11:43, damien krotkine <dkrotkine at gmail.com> wrote:

>
>
>
> On 10 July 2013 11:03, Edgar Veiga <edgarmveiga at gmail.com> wrote:
>
>> Hi Guido.
>>
>> Thanks for your answer!
>>
>> Bitcask it's not an option due to the amount of ram needed.. We would
>> need a lot more of physical nodes so more money spent...
>>
>
> Why is it not an option?
>
> If you use Bitcask, then each node needs to store its keys in memory. It's
> usually not a lot. In a precedent email I asked you the average lenght of
> *keys*, but you gave us the average length of *values* :)
>
> We have 1 billion keys and fits on a 5 nodes Ring. ( check out
> http://docs.basho.com/riak/1.2.0/references/appendices/Bitcask-Capacity-Planning/). Our bucket names are 1 letter, our keys are 10 chars long.
>
> What does a typical key look like ? Also, what are you using to serialize
> your php objects? Maybe you could paste a typical value somewhere as well
>
> Damien
>
>
>>
>> Instead we're using less machines with SSD disks to improve elevelDB
>> performance.
>>
>> Best regards
>>
>>
>>
>> On 10 July 2013 09:58, Guido Medina <guido.medina at temetra.com> wrote:
>>
>>>  Well, I rushed my answer before, if you want performance, you probably
>>> want Bitcask, if you want compression then LevelDB, the following links
>>> should help you decide better:
>>>
>>> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/Bitcask/
>>> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/LevelDB/
>>>
>>> Or multi, use one as default and then the other for specific buckets:
>>>
>>> http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/Multi/
>>>
>>> HTH,
>>>
>>> Guido.
>>>
>>>
>>>
>>> On 10/07/13 09:53, Guido Medina wrote:
>>>
>>> Then you are better off with Bitcask, that will be the fastest in your
>>> case (no 2i, no searches, no M/R)
>>>
>>> HTH,
>>>
>>> Guido.
>>>
>>> On 10/07/13 09:49, Edgar Veiga wrote:
>>>
>>> Hello all!
>>>
>>>  I have a couple of questions that I would like to address all of you
>>> guys, in order to start this migration the best as possible.
>>>
>>>  Context:
>>> - I'm responsible for the migration of a pure key/value store that for
>>> now is being stored on memcacheDB.
>>> - We're serializing php objects and storing them.
>>> - The total size occupied it's ~2TB.
>>>
>>>  - The idea it's to migrate this data to a riak cluster with elevelDB
>>> backend (starting with 6 nodes, 256 partitions. This thing is scaling very
>>> fast).
>>> - We only need to access the information by key. *We won't need neither
>>> map/reduces, searches or secondary indexes*. It's a pure key/value
>>> store!
>>>
>>>  My questions are:
>>> - Do you have any riak fine tunning tip regarding this use case (due to
>>> the fact that we will only use the key/value capabilities of riak)?
>>>  - It's expected that those 2TB would be reduced due to the levelDB
>>> compression. Do you think we should compress our objects to on the client?
>>>
>>>  Best regards,
>>> Edgar Veiga
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing listriak-users at lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130710/86bde107/attachment.html>


More information about the riak-users mailing list