Riak for streaming video

Alex De la rosa alex.rosa.box at gmail.com
Mon Jun 30 13:00:21 EDT 2014


Hi Kota,

返事をありがとうございました! :)

Ok, understood, so 1 Stanchion server shouldn't be a problem at all, good
to know :)

Cheers,
Alex


On Mon, Jun 30, 2014 at 5:03 PM, Kota Uenishi <kota at basho.com> wrote:

> There are a lot of variety, in definition of being SPOF. It also
> depends on not only software implementation but also on setup
> configuration and operation.
>
> When it comes to Stanchion, it just serializes creation, modification
> and deletion of buckets and users, and all data is stored in Riak,
> which means, Stanchion failure does not cause any data loss but just
> downtime of these operations. As Brian points out that all other
> operations on objects will work fine.
>
> Even you can run two Act-Standby Stanchion processes at once. But the
> only thing operators should not do is, to let any two CS nodes see
> different Stanchion at once. After the operator make sure 'primary'
> Stanchion killed or stopped, it's okay to do any style of failover.
>
> On Mon, Jun 30, 2014 at 7:27 PM, Brian Akins <brian at akins.org> wrote:
> > Stanchion is only used for user modification and creating/deleting
> buckets.
> > The cluster will function just fine without it except for those
> functions.
> >
> >
> > On Jun 30, 2014, at 3:10 AM, Alex De la rosa <alex.rosa.box at gmail.com>
> > wrote:
> >
> > Hi Jared,
> >
> > Thanks for your explanation and links... my next question is... Is
> Stanchion
> > not a single-point of failure? One of the things I like in Standard Riak
> is
> > that every node has the same level, there are no masters, no slaves... if
> > one fail it doesn't matter, the rest can do the job. If Riak CS requires
> un
> > especial node, what happens if that one fails? system is down then? Can
> you
> > configure the system with more than one Stanchion node?
> >
> > Cheers,
> > Alex
> >
> >
> > On Mon, Jun 30, 2014 at 7:37 AM, Jared Morrow <jared at basho.com> wrote:
> >>
> >> Alex,
> >>
> >> That is correct.  There are some additional docs we have on
> load-balancing
> >> and such on our docs page
> >>
> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/
> ,
> >> but generally it acts and feels like a Riak installation.  One
> additional
> >> special node is required (Stanchion) and that is also described on our
> docs
> >> page
> >>
> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Stanchion/
> .
> >>
> >> Keep the questions coming, if I don't know the details I'll hand off to
> >> Engineers more knowledgable with RiakCS.
> >>
> >> -Jared
> >>
> >>
> >>
> >>
> >> On Sun, Jun 29, 2014 at 5:58 AM, Alex De la rosa <
> alex.rosa.box at gmail.com>
> >> wrote:
> >>>
> >>> Hi there,
> >>>
> >>> One more question regarding Riak CS, does it behave like standard Riak?
> >>> storing 3 copies (default) and needing a minimum of 5 servers to
> operate?
> >>>
> >>> Cheers,
> >>> Alex
> >>>
> >>>
> >>> On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <jared at basho.com> wrote:
> >>>>
> >>>> Luwak is definitely EOL'd and RiakCS is our large object store going
> >>>> forward.  It is a far superior design compared to Luwak and handles
> very
> >>>> large file sizes.
> >>>>
> >>>>
> >>>>
> >>>> On Thursday, June 26, 2014, Jason Campbell <xiaclo at xiaclo.net> wrote:
> >>>>>
> >>>>> Riak CS is designed to be the same interface as Amazon S3.  It can be
> >>>>> run anywhere you can run Riak.
> >>>>>
> >>>>> Think of it as a way to run your own S3.
> >>>>>
> >>>>> I think the hard limit on Riak objects is 50MB, but the recommended
> >>>>> size is less than 1MB per object.  The main reason for that is
> latency.
> >>>>> It's faster to get 100x1MB chunks from the entire cluster than it is
> to get
> >>>>> 1x100MB chunk from a single server.  It's also easier for disks to
> pull a
> >>>>> 1MB object without delaying any other requests than pulling a 100MB
> object,
> >>>>> which may end up queuing other requests made after.
> >>>>>
> >>>>> There used to be a project called luwak that was built into Riak and
> >>>>> handled chunking files like this, but it has been EOL'd.  Perhaps
> someone
> >>>>> can shed some light on this.  Is RiakCS the final solution for binary
> >>>>> storage?  Or is there another way to store larger objects in Riak?
> >>>>>
> >>>>> RiakCS is wonderful, but it does seem overkill for some uses where
> you
> >>>>> don't need to expose an S3 interface, don't need the S3 permission
> >>>>> structure, and really just want file storage.
> >>>>>
> >>>>> ----- Original Message -----
> >>>>> From: "Alex De la rosa" <alex.rosa.box at gmail.com>
> >>>>> To: "Hector Castro" <hector at basho.com>
> >>>>> Cc: "riak-users" <riak-users at lists.basho.com>
> >>>>> Sent: Friday, 27 June, 2014 6:55:13 AM
> >>>>> Subject: Re: Riak for streaming video
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hi Hector,
> >>>>>
> >>>>>
> >>>>> Thanks for your response :) highly appreciated. I have more questions
> >>>>> though.
> >>>>>
> >>>>>
> >>>>> Riak CS has to be implemented in Amazon S3? Or can I house it myself
> >>>>> with own managed servers?
> >>>>>
> >>>>>
> >>>>> What is the limit for an object in Standard Riak? Large objects
> should
> >>>>> go to Riak CS, but what would be a limit for the Standard Riak? I
> may want
> >>>>> to save pictures, for example, in binary data... not a RAW version,
> but like
> >>>>> processed photo uploads from users... I think Standard Riak can hold
> that
> >>>>> with no problem, but it would be good to know the size limit for a
> key's
> >>>>> content.
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>> Alex
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < hector at basho.com >
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>> Hey Alex,
> >>>>>
> >>>>> My response are inline below.
> >>>>>
> >>>>> --
> >>>>> Hector
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
> >>>>> < alex.rosa.box at gmail.com > wrote:
> >>>>> > Hi Hector,
> >>>>> >
> >>>>> > I see, I always thought that Riak CS was the same as Riak just that
> >>>>> > you run
> >>>>> > it in Amazon S3... they are actually different...
> >>>>>
> >>>>> Correct, but it is important to note that Riak is still at the core
> of
> >>>>> Riak CS.
> >>>>>
> >>>>>
> >>>>> > So... if I have a social network and one of the features is that
> they
> >>>>> > can
> >>>>> > share video, I would use a normal Riak cluster for the webapp and
> >>>>> > data and a
> >>>>> > Riak CS cluster for the video storage/streaming... am I right?
> >>>>>
> >>>>> That seems like a reasonable conclusion.
> >>>>>
> >>>>> Riak is fundamentally a distributed key/value store for low latency
> >>>>> access to smaller pieces of data. Riak CS is an S3/Swift
> >>>>> API-compatible object storage platform with a little higher request
> >>>>> latency, but also the added ability to house very large objects
> >>>>> (backups, raw images, video).
> >>>>>
> >>>>>
> >>>>> > By the way, couldn't this "range header" be implemented in standard
> >>>>> > Riak?
> >>>>> > might be a good thing to have so you don't need 2 clusters for this
> >>>>> > matter.
> >>>>>
> >>>>> It could some day. Right now, separating the clusters is best because
> >>>>> tuning a single cluster for both use cases would be difficult.
> >>>>>
> >>>>>
> >>>>>
> >>>>> > Thanks,
> >>>>> > Alex
> >>>>> >
> >>>>> >
> >>>>> > On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < hector at basho.com
> >
> >>>>> > wrote:
> >>>>> >>
> >>>>> >> Hi Alex,
> >>>>> >>
> >>>>> >> For this type of problem, you may want to look into Riak CS. [0]
> It
> >>>>> >> is
> >>>>> >> an object storage platform built on top of Riak.
> >>>>> >>
> >>>>> >> When you GET an object from Riak CS, you can specify a Range
> header
> >>>>> >> to
> >>>>> >> get the object by its byte offset. [1]
> >>>>> >>
> >>>>> >> --
> >>>>> >> Hector
> >>>>> >>
> >>>>> >> [0] http://docs.basho.com/riakcs/latest/
> >>>>> >> [1]
> >>>>> >>
> >>>>> >>
> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
> >>>>> >>
> >>>>> >> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
> >>>>> >> < alex.rosa.box at gmail.com > wrote:
> >>>>> >> > Hi there,
> >>>>> >> >
> >>>>> >> > Imagine that I want to use Riak for a video service like Youtube
> >>>>> >> > and I
> >>>>> >> > save
> >>>>> >> > the video file in Riak... is there a way to stream the contents
> of
> >>>>> >> > 1
> >>>>> >> > key? I
> >>>>> >> > know we can stream keys... but what about its content?
> >>>>> >> >
> >>>>> >> > It would be pretty troublesome to have to wait for the full data
> >>>>> >> > to be
> >>>>> >> > downloaded $bucket->get('myvideo') to be able to serve it... as
> >>>>> >> > they can
> >>>>> >> > be
> >>>>> >> > pretty big files.
> >>>>> >> >
> >>>>> >> > Is there any recommendation to store/get big files like videos?
> or
> >>>>> >> > it
> >>>>> >> > would
> >>>>> >> > be better to use some other system than Riak for the job?
> >>>>> >> >
> >>>>> >> > Thanks,
> >>>>> >> > Alex
> >>>>> >> >
> >>>>> >> > _______________________________________________
> >>>>> >> > riak-users mailing list
> >>>>> >> > riak-users at lists.basho.com
> >>>>> >> >
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>>> >> >
> >>>>> >
> >>>>> >
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> riak-users mailing list
> >>>>> riak-users at lists.basho.com
> >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>>>
> >>>>> _______________________________________________
> >>>>> riak-users mailing list
> >>>>> riak-users at lists.basho.com
> >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>
> >>>
> >>
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
>
>
> --
> Kota UENISHI / @kuenishi
> Basho Japan KK
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140630/6da96a39/attachment.html>


More information about the riak-users mailing list