Riak for streaming video

Jared Morrow jared at basho.com
Thu Jun 26 20:43:23 EDT 2014


Luwak is definitely EOL'd and RiakCS is our large object store going
forward.  It is a far superior design compared to Luwak and handles very
large file sizes.



On Thursday, June 26, 2014, Jason Campbell <xiaclo at xiaclo.net> wrote:

> Riak CS is designed to be the same interface as Amazon S3.  It can be run
> anywhere you can run Riak.
>
> Think of it as a way to run your own S3.
>
> I think the hard limit on Riak objects is 50MB, but the recommended size
> is less than 1MB per object.  The main reason for that is latency.  It's
> faster to get 100x1MB chunks from the entire cluster than it is to get
> 1x100MB chunk from a single server.  It's also easier for disks to pull a
> 1MB object without delaying any other requests than pulling a 100MB object,
> which may end up queuing other requests made after.
>
> There used to be a project called luwak that was built into Riak and
> handled chunking files like this, but it has been EOL'd.  Perhaps someone
> can shed some light on this.  Is RiakCS the final solution for binary
> storage?  Or is there another way to store larger objects in Riak?
>
> RiakCS is wonderful, but it does seem overkill for some uses where you
> don't need to expose an S3 interface, don't need the S3 permission
> structure, and really just want file storage.
>
> ----- Original Message -----
> From: "Alex De la rosa" <alex.rosa.box at gmail.com <javascript:;>>
> To: "Hector Castro" <hector at basho.com <javascript:;>>
> Cc: "riak-users" <riak-users at lists.basho.com <javascript:;>>
> Sent: Friday, 27 June, 2014 6:55:13 AM
> Subject: Re: Riak for streaming video
>
>
>
> Hi Hector,
>
>
> Thanks for your response :) highly appreciated. I have more questions
> though.
>
>
> Riak CS has to be implemented in Amazon S3? Or can I house it myself with
> own managed servers?
>
>
> What is the limit for an object in Standard Riak? Large objects should go
> to Riak CS, but what would be a limit for the Standard Riak? I may want to
> save pictures, for example, in binary data... not a RAW version, but like
> processed photo uploads from users... I think Standard Riak can hold that
> with no problem, but it would be good to know the size limit for a key's
> content.
>
>
> Thanks,
> Alex
>
>
>
> On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < hector at basho.com
> <javascript:;> > wrote:
>
>
> Hey Alex,
>
> My response are inline below.
>
> --
> Hector
>
>
>
> On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
> < alex.rosa.box at gmail.com <javascript:;> > wrote:
> > Hi Hector,
> >
> > I see, I always thought that Riak CS was the same as Riak just that you
> run
> > it in Amazon S3... they are actually different...
>
> Correct, but it is important to note that Riak is still at the core of
> Riak CS.
>
>
> > So... if I have a social network and one of the features is that they can
> > share video, I would use a normal Riak cluster for the webapp and data
> and a
> > Riak CS cluster for the video storage/streaming... am I right?
>
> That seems like a reasonable conclusion.
>
> Riak is fundamentally a distributed key/value store for low latency
> access to smaller pieces of data. Riak CS is an S3/Swift
> API-compatible object storage platform with a little higher request
> latency, but also the added ability to house very large objects
> (backups, raw images, video).
>
>
> > By the way, couldn't this "range header" be implemented in standard Riak?
> > might be a good thing to have so you don't need 2 clusters for this
> matter.
>
> It could some day. Right now, separating the clusters is best because
> tuning a single cluster for both use cases would be difficult.
>
>
>
> > Thanks,
> > Alex
> >
> >
> > On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < hector at basho.com
> <javascript:;> > wrote:
> >>
> >> Hi Alex,
> >>
> >> For this type of problem, you may want to look into Riak CS. [0] It is
> >> an object storage platform built on top of Riak.
> >>
> >> When you GET an object from Riak CS, you can specify a Range header to
> >> get the object by its byte offset. [1]
> >>
> >> --
> >> Hector
> >>
> >> [0] http://docs.basho.com/riakcs/latest/
> >> [1]
> >>
> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
> >>
> >> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
> >> < alex.rosa.box at gmail.com <javascript:;> > wrote:
> >> > Hi there,
> >> >
> >> > Imagine that I want to use Riak for a video service like Youtube and I
> >> > save
> >> > the video file in Riak... is there a way to stream the contents of 1
> >> > key? I
> >> > know we can stream keys... but what about its content?
> >> >
> >> > It would be pretty troublesome to have to wait for the full data to be
> >> > downloaded $bucket->get('myvideo') to be able to serve it... as they
> can
> >> > be
> >> > pretty big files.
> >> >
> >> > Is there any recommendation to store/get big files like videos? or it
> >> > would
> >> > be better to use some other system than Riak for the job?
> >> >
> >> > Thanks,
> >> > Alex
> >> >
> >> > _______________________________________________
> >> > riak-users mailing list
> >> > riak-users at lists.basho.com <javascript:;>
> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >
> >
> >
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com <javascript:;>
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com <javascript:;>
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140626/ad9f8410/attachment.html>


More information about the riak-users mailing list