Riak for streaming video

Jason Campbell xiaclo at xiaclo.net
Thu Jun 26 18:41:59 EDT 2014

Riak CS is designed to be the same interface as Amazon S3.  It can be run anywhere you can run Riak.

Think of it as a way to run your own S3.

I think the hard limit on Riak objects is 50MB, but the recommended size is less than 1MB per object.  The main reason for that is latency.  It's faster to get 100x1MB chunks from the entire cluster than it is to get 1x100MB chunk from a single server.  It's also easier for disks to pull a 1MB object without delaying any other requests than pulling a 100MB object, which may end up queuing other requests made after.

There used to be a project called luwak that was built into Riak and handled chunking files like this, but it has been EOL'd.  Perhaps someone can shed some light on this.  Is RiakCS the final solution for binary storage?  Or is there another way to store larger objects in Riak?

RiakCS is wonderful, but it does seem overkill for some uses where you don't need to expose an S3 interface, don't need the S3 permission structure, and really just want file storage.

----- Original Message -----
From: "Alex De la rosa" <alex.rosa.box at gmail.com>
To: "Hector Castro" <hector at basho.com>
Cc: "riak-users" <riak-users at lists.basho.com>
Sent: Friday, 27 June, 2014 6:55:13 AM
Subject: Re: Riak for streaming video

Hi Hector, 

Thanks for your response :) highly appreciated. I have more questions though. 

Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers? 

What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content. 


On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < hector at basho.com > wrote: 

Hey Alex, 

My response are inline below. 


On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa 
< alex.rosa.box at gmail.com > wrote: 
> Hi Hector, 
> I see, I always thought that Riak CS was the same as Riak just that you run 
> it in Amazon S3... they are actually different... 

Correct, but it is important to note that Riak is still at the core of Riak CS. 

> So... if I have a social network and one of the features is that they can 
> share video, I would use a normal Riak cluster for the webapp and data and a 
> Riak CS cluster for the video storage/streaming... am I right? 

That seems like a reasonable conclusion. 

Riak is fundamentally a distributed key/value store for low latency 
access to smaller pieces of data. Riak CS is an S3/Swift 
API-compatible object storage platform with a little higher request 
latency, but also the added ability to house very large objects 
(backups, raw images, video). 

> By the way, couldn't this "range header" be implemented in standard Riak? 
> might be a good thing to have so you don't need 2 clusters for this matter. 

It could some day. Right now, separating the clusters is best because 
tuning a single cluster for both use cases would be difficult. 

> Thanks, 
> Alex 
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < hector at basho.com > wrote: 
>> Hi Alex, 
>> For this type of problem, you may want to look into Riak CS. [0] It is 
>> an object storage platform built on top of Riak. 
>> When you GET an object from Riak CS, you can specify a Range header to 
>> get the object by its byte offset. [1] 
>> -- 
>> Hector 
>> [0] http://docs.basho.com/riakcs/latest/ 
>> [1] 
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples 
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa 
>> < alex.rosa.box at gmail.com > wrote: 
>> > Hi there, 
>> > 
>> > Imagine that I want to use Riak for a video service like Youtube and I 
>> > save 
>> > the video file in Riak... is there a way to stream the contents of 1 
>> > key? I 
>> > know we can stream keys... but what about its content? 
>> > 
>> > It would be pretty troublesome to have to wait for the full data to be 
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can 
>> > be 
>> > pretty big files. 
>> > 
>> > Is there any recommendation to store/get big files like videos? or it 
>> > would 
>> > be better to use some other system than Riak for the job? 
>> > 
>> > Thanks, 
>> > Alex 
>> > 
>> > _______________________________________________ 
>> > riak-users mailing list 
>> > riak-users at lists.basho.com 
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>> > 

riak-users mailing list
riak-users at lists.basho.com

More information about the riak-users mailing list