RiakCS poor s3 upload speeds 2MB/s

Luke Bakken lbakken at basho.com
Tue Jan 20 23:22:38 EST 2015


Hi Toby -

Are you using the stock bitcask configuration for merging?

--
Luke Bakken
Engineer
lbakken at basho.com

On Tue, Jan 20, 2015 at 5:07 PM, Toby Corkindale <toby at dryft.net> wrote:
> Hi Kota,
> I had a bit of an off-list chat about this a while ago, plus continued
> to investigate locally, and eventually achieved some faster speeds,
> around 15MByte/sec writes.
> Things that were changed:
>  * Adjusted Riak CS GC to be spread out over the cluster much more.
>  * Tweaked up the put buffers and concurrency further
>  * Moved most of the files out of CS and into Amazon S3+Glacier
>  * Switched from nginx to haproxy
>  * simplified firewalling for internal clients
>
> Each one of those changes made a small to modest improvement, but
> overall combined to make a quite noticeable improvement.
>
> I did notice something odd though -- despite moving most of the data
> out of the cluster, the disk-space-in-use by Riak is still very large
> compared to the amount stored. I mean, we moved more than 90% of the
> data out of the cluster, yet the actual disk space used only halved.
> For every gigabyte of file stored in CS, dozens of gigabytes are
> actually on disk!
>
> Either the garbage collection algorithm is very, very lazy, or somehow
> something has gone a bit wrong in the past, which might have explained
> part of the performance problems.
>
> We're going to look at redeploying a new, fresh cluster based on Riak
> 2 in the not too distant future, once Riak CS looks like it's approved
> for use there, and maybe that'll clear all of this up.
>
> Toby




More information about the riak-users mailing list