Brokenness specific to nginx and jets3t with Riak CS

Toby Corkindale toby at dryft.net
Thu Sep 25 03:42:47 EDT 2014


Following up further again, with a solution.

Turns out that Nginx canonicalises the URLs when proxying them, if and
only if, the proxy_pass parameter includes a path component, including
a plain slash.
Meanwhile, Jets3t URL-encodes the object path so that slashes become
%2F, but Nginx converts them to actual slashes before forwarding to
Riak CS.

Fix was to change
proxy_pass http://upstreamservers/;
to
proxy_pass http://upstreamservers;

-Toby

On 25 September 2014 16:24, Toby Corkindale <toby at dryft.net> wrote:
> Quite update to mention that by removing some extra (unneeded) custom
> headers from the proxy configuration and fiddling with some other
> nginx options, I'm at a point where Riak CS throws an actual error
> back rather than disconnecting prematurely.
> The error now returned is below. I'm wondering if something else nginx
> is doing is causing the request to get mangled enough that it doesn't
> work.. but hard to tell what, or why :/
>
> ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml
> version=\"1.0\"
> encoding=\"UTF-8\"?><Error><Code>AccessDenied</Code><Message>Access
> Denied</Message><Resource>/upload-service/da417126-8651-4be5-b552-6d125bb7b27c/coreos_production_ami_hvm.txt</Resource><RequestId></RequestId></Error>
>
>
> On 25 September 2014 15:59, Toby Corkindale <toby at dryft.net> wrote:
>> Hi,
>> I've hit an issue that only seems to occur when using the Jets3t
>> client library and Nginx as a load balancer in front of Riak CS.
>> The issue is NOT present when using other S3 libraries, nor is it
>> present if I switch out Nginx for haproxy.
>>
>> Unfortunately, in this instance it is desirable to use both, unless it
>> turns out to be *really* troublesome.
>> I wondered if anyone here can offer advice?
>>
>> The problem is that reads and writes to objects results in Riak CS,
>> apparently, disconnecting the client prematurely.
>> Nginx reports "connection reset by peer" and retries upstream Riak CS
>> servers several times before giving up.
>> The Riak CS access logs indicate what look like valid URLs, with a 403
>> status indicated.
>>
>>
>> Accessing the same buckets with the same auth and transferring the
>> same files is fine when done with s3cmd, and we have other apps that
>> have been running through nginx for a while using other S3 client
>> libraries.
>>
>> A simple "check if bucket exists" command through jets3t seems to work OK.
>>
>> Any thoughts on what could be going on, or is this all just a bit too vague?
>>
>> Toby
>
>
>
> --
> Turning and turning in the widening gyre
> The falcon cannot hear the falconer
> Things fall apart; the center cannot hold
> Mere anarchy is loosed upon the world



-- 
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world




More information about the riak-users mailing list