Fwd: RiakCS 504 Timeout on s3cmd for certain keys

Kelly McLaughlin kelly at basho.com
Tue Aug 19 12:27:18 EDT 2014


Alex,

The value you had set for the fold_objects_for_list_keys setting is one 
I was very interested to see and I highly recommend setting it to true 
for your cluster.
The impact of setting this to true should be to make bucket listing 
operations generally more efficient. There should be no detrimental 
effects. There are
also some optimizations for bucket listing queries that use the prefix 
request parameter so I would expect queries to list specific 
subdirectories in a bucket to
show improved perforamnce as well. Changing the app.config and 
restarting the CS node is the correct way to have it take effect.


As for GC performance, I would recommend to add an entry to your 
app.config file to se||||t gc_paginated_indexes to true. This option 
causes the GC process
to use a more efficient process for determining data that is eligible 
for collection and generally results in far fewer timeouts and better 
success for
users.


Kelly

On 08/19/2014 07:32 AM, Alex Millar wrote:
> Hey Kota,
>
> We’re currently using the following versions;
>
> # Download RiakCS
> # Version: 1.4.5
> # OS: Ubuntu 12.04 (Precise) AMD 64
> curl -O 
> http://s3.amazonaws.com/downloads.basho.com/riak-cs/1.4/1.4.5/ubuntu/precise/riak-cs_1.4.5-1_amd64.deb
>
> # Download Riak
> # Version: 1.4.8
> # OS: Ubuntu 12.04 (Precise) AMD 64
> curl -O 
> http://s3.amazonaws.com/downloads.basho.com/riak/1.4/1.4.8/ubuntu/precise/riak_1.4.8-1_amd64.deb
>
> I checked our RiakCS app.config and fold_objects_for_list_keys is set 
> to false. What impact would it have my cluster if I flip that to true? 
> Would I simply update the app.config and restart RiakCS?
>
> As for the consideration on garbage collection, the slow performance 
> is happening consistently over the span of a week (since we noticed it 
> as we don’t often list buckets). I suspect its not the case regarding 
> large amounts of objects being deleted as generally all data going 
> into that bucket is write-once (we process PDFs pages to .JPG and PUT 
> them in that bucket, the only time overwrites occur is if we manually 
> re-trigger the processing script to run on a specific document)
>
> *Adding kelly at basho.com* as we have another thread going on about this 
> same topic, I figured we could merge the discussion to reduce 
> duplicate effort here.
>
> Bonfire Logo 	*Alex Millar*, CTO
> Office: 1-800-354-8010 ext. 704 <tel:+18003548010>
> Mobile: 519-729-2539 <tel:+15197292539>
> *GoBonfire*.com <http://GoBonfire.com>
>
>
> From: Kota Uenishi <kota at basho.com> <mailto:kota at basho.com>
> Reply: Kota Uenishi <kota at basho.com>> <mailto:kota at basho.com>
> Date: August 18, 2014 at 10:03:40 PM
> To: Alex Millar <alex at gobonfire.com>> <mailto:alex at gobonfire.com>
> Cc: Charlie Voiselle <cvoiselle at basho.com>> 
> <mailto:cvoiselle at basho.com>, Tad Bickford <tbickford at basho.com>> 
> <mailto:tbickford at basho.com>, Riak-Users <riak-users at lists.basho.com>> 
> <mailto:riak-users at lists.basho.com>, Brandon Noad 
> <brandon at gobonfire.com>> <mailto:brandon at gobonfire.com>
> Subject: Re: Fwd: RiakCS 504 Timeout on s3cmd for certain keys
>
>> Alex,
>>
>> Riak CS 1.4.5 and 1.5.0 had a lot of improvement after those articles 
>> you put the URL, not it is not using Riak's bucket listing but using 
>> Riak's internal API for more efficient listing. What version of Riak 
>> CS are you using? I want you to make sure you're using those versions 
>> and a line `{fold_objects_for_list_keys, true},` at riak_cs section 
>> of app.config (assuming all other Riak part correctly configured).
>>
>> >Based on this I’m thinking that cost of this type of query is only 
>> going to get worse over time as we add more keys to this bucket 
>> (unless secondary indexes can be added). Or am I totally out to lunch 
>> here and there’s some other underlying problem?
>>
>> The strange part is s3cmd. Riak CS has incremental bucket listing API 
>> that requires clients to iterate on every 1000 objects (common 
>> prefixes), but s3cmd iterates all the specified bucket before 
>> printing them all. You can observe how s3cmd and Riak CS interacts if 
>> you specify '-d' option like this:
>>
>> ```
>> s3cmd -d -c yours.s3cfg ls -r s3://yourbucket/yourdir/
>> ```
>>
>> I would expect Riak CS's listing API is not much slow  as to need 5 
>> seconds (or, say, >10 seconds) because, on each request it just 
>> returns 1000 objects.
>>
>> There might be another possibility on slow query - if you had many 
>> (say, more than 10 thousands) deleted objects on the same bucket it 
>> might affect each 1000 listing. This will eventually be solved as 
>> Riak CS's garbage collection removes deleted manifests, which is just 
>> marked as deleted (and to be ignored correctly).
>>
>> [1] 
>> http://www.quora.com/Riak/Is-it-really-expensive-for-Riak-to-list-all-buckets-Why
>>
>> On Thu, Aug 14, 2014 at 6:05 AM, Alex Millar <alex at gobonfire.com 
>> <mailto:alex at gobonfire.com>> wrote:
>>
>>     Good afternoon Charlie,
>>
>>     So the issue we’re having is only with bucket listing.
>>
>>     alxndrmlr at alxndrmlr-mbp $ time s3cmd -c .s3cfg-riakcs-admin ls
>>     s3://bonfirehub-resources-can-east-doc-conversion
>>            DIR
>>     s3://bonfirehub-resources-can-east-doc-conversion/organizations/
>>
>>     real 2m0.747s
>>     user 0m0.076s
>>     sys 0m0.030s
>>
>>     where as…
>>
>>     alxndrmlr at alxndrmlr-mbp $ time s3cmd -c .s3cfg-riakcs-admin ls
>>     s3://bonfirehub-resources-can-east-doc-conversion/organizations/OrganizationID-1/documents/proposals
>>            DIR
>>     s3://bonfirehub-resources-can-east-doc-conversion/organizations/OrganizationID-1/documents/proposals/
>>
>>     real 0m10.262s
>>     user 0m0.075s
>>     sys 0m0.028s
>>
>>     The contents of this bucket contains a lot of very small files
>>     (basically for each PDF we receive I split it to .JPG foreach
>>     page and store them here. Based on the my latest counts it looks
>>     like we have around *170,000* .JPG files in that bucket.
>>
>>     Here’s a snippet from the HAProxy log for the 504 timeouts…
>>
>>     Aug 12 16:01:34
>>     <http://airmail.calendar/2014-08-12%2016:01:34%20EDT> localhost.localdomain
>>     haproxy[4718]: 192.0.223.236:48457 <http://192.0.223.236:48457>
>>     [12/Aug/2014:16:01:24.454] riak_cs~ riak_cs_backend/riak3
>>     161/0/0/-1/10162 504 194 - - sH-- 0/0/0/0/0 0/0
>>     {bonfirehub-resources-can-east-doc-conversion.bf-riakcs.com
>>     <http://bonfirehub-resources-can-east-doc-conversion.bf-riakcs.com/>}
>>     "GET /?delimiter=/ HTTP/1.1"
>>
>>     I’ve put together a video showing off the top results of each of
>>     the 5 riak nodes while performing $ time s3cmd -c
>>     .s3cfg-riakcs-admin
>>     lss3://bonfirehub-resources-can-east-doc-conversion
>>
>>     https://dl.dropboxusercontent.com/u/5723659/RiakCS%20ls%20monitoring%20results.mov
>>
>>     Now I’ve had a hunch this is just a fundamentally expensive
>>     operation which exceeds the 5000ms response time threshold set in
>>     our HAProxy config (which I raised during the video to illustrate
>>     what’s going on). After reading
>>     http://www.quora.com/Riak/Is-it-really-expensive-for-Riak-to-list-all-buckets-Why and
>>     http://www.paperplanes.de/2011/12/13/list-all-of-the-riak-keys.html I’m
>>     feeling like this is just a fundamental issue with the data
>>     structure in Riak.
>>
>>     Based on this I’m thinking that cost of this type of query is
>>     only going to get worse over time as we add more keys to this
>>     bucket (unless secondary indexes can be added). Or am I totally
>>     out to lunch here and there’s some other underlying problem?
>>
>>     I’ve cc’d the mailing list on this as suggested.
>>
>>     Bonfire Logo 	*Alex Millar*, CTO
>>     Office: 1-800-354-8010 ext. 704 <tel:+18003548010>
>>     Mobile: 519-729-2539 <tel:+15197292539>
>>     *GoBonfire*.com <http://GoBonfire.com>
>>
>>
>>     From: Charlie Voiselle <cvoiselle at basho.com>
>>     <mailto:cvoiselle at basho.com>
>>     Reply: Charlie Voiselle <cvoiselle at basho.com>>
>>     <mailto:cvoiselle at basho.com>
>>     Date: August 13, 2014 at 10:36:51 AM
>>     To: Alex Millar <alex at gobonfire.com>> <mailto:alex at gobonfire.com>
>>     Cc: Tad Bickford <tbickford at basho.com>> <mailto:tbickford at basho.com>
>>     Subject: Fwd: RiakCS 504 Timeout on s3cmd for certain keys
>>
>>>
>>
>>     _______________________________________________
>>     riak-users mailing list
>>     riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>     http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>>
>> --
>> Kota UENISHI / @kuenishi
>> Basho Japan KK

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140819/9660b32d/attachment.html>


More information about the riak-users mailing list