RiakCS poor s3 upload speeds 2MB/s

Toby Corkindale toby at dryft.net
Wed Jan 21 07:40:12 EST 2015


On 21 January 2015 at 15:22, Luke Bakken <lbakken at basho.com> wrote:
> Hi Toby -
>
> Are you using the stock bitcask configuration for merging?

Hi Luke,
Yes, pretty much.

> On Tue, Jan 20, 2015 at 5:07 PM, Toby Corkindale <toby at dryft.net> wrote:
>> Hi Kota,
>> I had a bit of an off-list chat about this a while ago, plus continued
>> to investigate locally, and eventually achieved some faster speeds,
>> around 15MByte/sec writes.
>> Things that were changed:
>>  * Adjusted Riak CS GC to be spread out over the cluster much more.
>>  * Tweaked up the put buffers and concurrency further
>>  * Moved most of the files out of CS and into Amazon S3+Glacier
>>  * Switched from nginx to haproxy
>>  * simplified firewalling for internal clients
>>
>> Each one of those changes made a small to modest improvement, but
>> overall combined to make a quite noticeable improvement.
>>
>> I did notice something odd though -- despite moving most of the data
>> out of the cluster, the disk-space-in-use by Riak is still very large
>> compared to the amount stored. I mean, we moved more than 90% of the
>> data out of the cluster, yet the actual disk space used only halved.
>> For every gigabyte of file stored in CS, dozens of gigabytes are
>> actually on disk!
>>
>> Either the garbage collection algorithm is very, very lazy, or somehow
>> something has gone a bit wrong in the past, which might have explained
>> part of the performance problems.
>>
>> We're going to look at redeploying a new, fresh cluster based on Riak
>> 2 in the not too distant future, once Riak CS looks like it's approved
>> for use there, and maybe that'll clear all of this up.
>>
>> Toby



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world




More information about the riak-users mailing list