Riak for streaming video

Jared Morrow jared at basho.com
Mon Jun 30 01:37:24 EDT 2014


Alex,

That is correct.  There are some additional docs we have on load-balancing
and such on our docs page
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/,
but generally it acts and feels like a Riak installation.  One additional
special node is required (Stanchion) and that is also described on our docs
page
http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Stanchion/
.

Keep the questions coming, if I don't know the details I'll hand off to
Engineers more knowledgable with RiakCS.

-Jared




On Sun, Jun 29, 2014 at 5:58 AM, Alex De la rosa <alex.rosa.box at gmail.com>
wrote:

> Hi there,
>
> One more question regarding Riak CS, does it behave like standard Riak?
> storing 3 copies (default) and needing a minimum of 5 servers to operate?
>
> Cheers,
> Alex
>
>
> On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <jared at basho.com> wrote:
>
>> Luwak is definitely EOL'd and RiakCS is our large object store going
>> forward.  It is a far superior design compared to Luwak and handles very
>> large file sizes.
>>
>>
>>
>> On Thursday, June 26, 2014, Jason Campbell <xiaclo at xiaclo.net> wrote:
>>
>>> Riak CS is designed to be the same interface as Amazon S3.  It can be
>>> run anywhere you can run Riak.
>>>
>>> Think of it as a way to run your own S3.
>>>
>>> I think the hard limit on Riak objects is 50MB, but the recommended size
>>> is less than 1MB per object.  The main reason for that is latency.  It's
>>> faster to get 100x1MB chunks from the entire cluster than it is to get
>>> 1x100MB chunk from a single server.  It's also easier for disks to pull a
>>> 1MB object without delaying any other requests than pulling a 100MB object,
>>> which may end up queuing other requests made after.
>>>
>>> There used to be a project called luwak that was built into Riak and
>>> handled chunking files like this, but it has been EOL'd.  Perhaps someone
>>> can shed some light on this.  Is RiakCS the final solution for binary
>>> storage?  Or is there another way to store larger objects in Riak?
>>>
>>> RiakCS is wonderful, but it does seem overkill for some uses where you
>>> don't need to expose an S3 interface, don't need the S3 permission
>>> structure, and really just want file storage.
>>>
>>> ----- Original Message -----
>>> From: "Alex De la rosa" <alex.rosa.box at gmail.com>
>>> To: "Hector Castro" <hector at basho.com>
>>> Cc: "riak-users" <riak-users at lists.basho.com>
>>> Sent: Friday, 27 June, 2014 6:55:13 AM
>>> Subject: Re: Riak for streaming video
>>>
>>>
>>>
>>> Hi Hector,
>>>
>>>
>>> Thanks for your response :) highly appreciated. I have more questions
>>> though.
>>>
>>>
>>> Riak CS has to be implemented in Amazon S3? Or can I house it myself
>>> with own managed servers?
>>>
>>>
>>> What is the limit for an object in Standard Riak? Large objects should
>>> go to Riak CS, but what would be a limit for the Standard Riak? I may want
>>> to save pictures, for example, in binary data... not a RAW version, but
>>> like processed photo uploads from users... I think Standard Riak can hold
>>> that with no problem, but it would be good to know the size limit for a
>>> key's content.
>>>
>>>
>>> Thanks,
>>> Alex
>>>
>>>
>>>
>>> On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < hector at basho.com >
>>> wrote:
>>>
>>>
>>> Hey Alex,
>>>
>>> My response are inline below.
>>>
>>> --
>>> Hector
>>>
>>>
>>>
>>> On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
>>> < alex.rosa.box at gmail.com > wrote:
>>> > Hi Hector,
>>> >
>>> > I see, I always thought that Riak CS was the same as Riak just that
>>> you run
>>> > it in Amazon S3... they are actually different...
>>>
>>> Correct, but it is important to note that Riak is still at the core of
>>> Riak CS.
>>>
>>>
>>> > So... if I have a social network and one of the features is that they
>>> can
>>> > share video, I would use a normal Riak cluster for the webapp and data
>>> and a
>>> > Riak CS cluster for the video storage/streaming... am I right?
>>>
>>> That seems like a reasonable conclusion.
>>>
>>> Riak is fundamentally a distributed key/value store for low latency
>>> access to smaller pieces of data. Riak CS is an S3/Swift
>>> API-compatible object storage platform with a little higher request
>>> latency, but also the added ability to house very large objects
>>> (backups, raw images, video).
>>>
>>>
>>> > By the way, couldn't this "range header" be implemented in standard
>>> Riak?
>>> > might be a good thing to have so you don't need 2 clusters for this
>>> matter.
>>>
>>> It could some day. Right now, separating the clusters is best because
>>> tuning a single cluster for both use cases would be difficult.
>>>
>>>
>>>
>>> > Thanks,
>>> > Alex
>>> >
>>> >
>>> > On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < hector at basho.com >
>>> wrote:
>>> >>
>>> >> Hi Alex,
>>> >>
>>> >> For this type of problem, you may want to look into Riak CS. [0] It is
>>> >> an object storage platform built on top of Riak.
>>> >>
>>> >> When you GET an object from Riak CS, you can specify a Range header to
>>> >> get the object by its byte offset. [1]
>>> >>
>>> >> --
>>> >> Hector
>>> >>
>>> >> [0] http://docs.basho.com/riakcs/latest/
>>> >> [1]
>>> >>
>>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>> >>
>>> >> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>>> >> < alex.rosa.box at gmail.com > wrote:
>>> >> > Hi there,
>>> >> >
>>> >> > Imagine that I want to use Riak for a video service like Youtube
>>> and I
>>> >> > save
>>> >> > the video file in Riak... is there a way to stream the contents of 1
>>> >> > key? I
>>> >> > know we can stream keys... but what about its content?
>>> >> >
>>> >> > It would be pretty troublesome to have to wait for the full data to
>>> be
>>> >> > downloaded $bucket->get('myvideo') to be able to serve it... as
>>> they can
>>> >> > be
>>> >> > pretty big files.
>>> >> >
>>> >> > Is there any recommendation to store/get big files like videos? or
>>> it
>>> >> > would
>>> >> > be better to use some other system than Riak for the job?
>>> >> >
>>> >> > Thanks,
>>> >> > Alex
>>> >> >
>>> >> > _______________________________________________
>>> >> > riak-users mailing list
>>> >> > riak-users at lists.basho.com
>>> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> >> >
>>> >
>>> >
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140629/b58145f3/attachment.html>


More information about the riak-users mailing list