Riak on SAN

Dave Martorana dave at flyclops.com
Fri Oct 4 13:47:47 EDT 2013


We're in the middle of building out a Riak cluster with OmniOS and ZFS
snapshot backups offsite with the help some friends that have deployed the
same basic solution (not Riak, but ZFS snapshots for hot-backup) at a
tremendous scale.

You get some other nice bits along with "hot" backups - ZFS snapshots are
basically diffs, so you can snapshot as often as once-per-second if you're
so inclined. Vertical scaling is also interesting in that you can launch a
beefier replacement machine for node and restore the older machine's ZFS
snapshot. The node won't be wise to the fact it's not the same machine,
which makes vertical scaling much faster than the traditional replace
method. (This is where someone at Basho can scream "NOOO! DON'T DO THAT!")

You can continue down that line of thinking to plenty of advantages of
using ZFS if backup is of ultimate importance - including being able to
restore a cluster even if the building hosting your physical boxes are hit
by a daisy cutter. :)

Here's a bit of information on how we're going about it, so far:
http://tech.flyclops.com/building-riak-on-omnios-360

Dave


On Thu, Oct 3, 2013 at 8:45 PM, Pedram Nimreezi <mc at majorcomputing.com>wrote:

> I consider that the main use case ;p
>
>
> On Thu, Oct 3, 2013 at 8:38 PM, Mike Oxford <moxford at gmail.com> wrote:
>
>> One more use-case for backups:  If you're running a big cluster and UserX
>> makes a bad code deploy which horks a bunch of data ... restore may be the
>> only option.
>>
>> It happens.
>>
>> -mox
>>
>>
>> On Wed, Oct 2, 2013 at 12:12 PM, John E. Vincent <
>> lusis.org+riak-users at gmail.com> wrote:
>>
>>> I'm going to take a competing view here.
>>>
>>> SAN is a bit overloaded of a term at this point. Nothing precludes a SAN
>>> from being performant or having SSDs. Yes the cost is overkill for fiber
>>> but iSCSI is much more realistic. Alternately you can even do ATAoE.
>>>
>>> From a hardware perspective, if I have 5 pizza boxes as riak nodes, I
>>> can only fit so many disks in them. Meanwhile I can add another shelf to my
>>> SAN and expand as needed. Additionally backup of a SAN is MUCH easier than
>>> backup of a riak node itself. It's a snapshot and you're done. Mind you
>>> nothing precludes you from doing LVM snapshots in the OS but you still need
>>> to get the data OFF that system for it to be truly backed up.
>>>
>>> I love riak and other distributed stores but backing them up is NOT a
>>> solved problem. Walking all keys, coordinating the take down of all your
>>> nodes in a given order or whatever your strategy is a serious pain point.
>>>
>>> Using a SAN or local disk also doesn't excuse you from watching I/O
>>> performance. With a SAN I get multiple redundant paths to a block device
>>> and I don't get that necessarily with local storage.
>>>
>>> Just my two bits.
>>>
>>>
>>>
>>> On Wed, Oct 2, 2013 at 2:18 AM, Jeremiah Peschka <
>>> jeremiah.peschka at gmail.com> wrote:
>>>
>>>> Could you do it? Sure.
>>>>
>>>> Should you do it? No.
>>>>
>>>> An advantage of Riak is that you can avoid the cost of SAN storage by
>>>> getting duplication at the machine level rather than rely on your storage
>>>> vendor to provide it.
>>>>
>>>> Running Riak on a SAN also exposes you to the SAN becoming your
>>>> bottleneck; you only have so many fiber/iSCSI ports and a fixed number of
>>>> disks. The risk of storage contention is high, too, so you can run into
>>>> latency issues that are difficult to diagnose without looking into both
>>>> Riak as well as the storage system.
>>>>
>>>> Keeping cost in mind, too, SAN storage is about 10x the cost of
>>>> consumer grade SSDs. Not to mention feature licensing and support... The
>>>> cost comparison isn't favorable.
>>>>
>>>> Please note: Even though your vendor calls it a SAN, that doesn't mean
>>>> it's a SAN.
>>>>  On Oct 1, 2013 11:08 PM, "Guy Morton" <Guy.Morton at bksv.com> wrote:
>>>>
>>>>> Does this make sense?
>>>>>
>>>>> --
>>>>> Guy Morton
>>>>> Web Development Manager
>>>>> Brüel & Kjær EMS
>>>>>
>>>>> This e-mail is confidential and may be read, copied and used only by
>>>>> the intended recipient. If you have received it in error, please contact
>>>>> the sender immediately by return e-mail. Please then delete the e-mail and
>>>>> do not disclose its contents to any other person.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> riak-users at lists.basho.com
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> /* Sincerely
> --------------------------------------------------------------
> Pedram Nimreezi - Chief Technology Officer  */
>
> // The hardest part of design … is keeping features out. - Donald Norman
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131004/e34df5e4/attachment.html>


More information about the riak-users mailing list