Brokenness specific to nginx and jets3t with Riak CS

Luke Bakken lbakken at basho.com
Mon Sep 29 12:55:39 EDT 2014


Hi Toby,

Thanks a ton for the detailed information. I'll make sure it's made
more obvious on this page:

http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/

--
Luke Bakken
Engineer / CSE
lbakken at basho.com


On Thu, Sep 25, 2014 at 12:42 AM, Toby Corkindale <toby at dryft.net> wrote:
> Following up further again, with a solution.
>
> Turns out that Nginx canonicalises the URLs when proxying them, if and
> only if, the proxy_pass parameter includes a path component, including
> a plain slash.
> Meanwhile, Jets3t URL-encodes the object path so that slashes become
> %2F, but Nginx converts them to actual slashes before forwarding to
> Riak CS.
>
> Fix was to change
> proxy_pass http://upstreamservers/;
> to
> proxy_pass http://upstreamservers;
>
> -Toby
>
> On 25 September 2014 16:24, Toby Corkindale <toby at dryft.net> wrote:
>> Quite update to mention that by removing some extra (unneeded) custom
>> headers from the proxy configuration and fiddling with some other
>> nginx options, I'm at a point where Riak CS throws an actual error
>> back rather than disconnecting prematurely.
>> The error now returned is below. I'm wondering if something else nginx
>> is doing is causing the request to get mangled enough that it doesn't
>> work.. but hard to tell what, or why :/
>>
>> ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml
>> version=\"1.0\"
>> encoding=\"UTF-8\"?><Error><Code>AccessDenied</Code><Message>Access
>> Denied</Message><Resource>/upload-service/da417126-8651-4be5-b552-6d125bb7b27c/coreos_production_ami_hvm.txt</Resource><RequestId></RequestId></Error>
>>
>>
>> On 25 September 2014 15:59, Toby Corkindale <toby at dryft.net> wrote:
>>> Hi,
>>> I've hit an issue that only seems to occur when using the Jets3t
>>> client library and Nginx as a load balancer in front of Riak CS.
>>> The issue is NOT present when using other S3 libraries, nor is it
>>> present if I switch out Nginx for haproxy.
>>>
>>> Unfortunately, in this instance it is desirable to use both, unless it
>>> turns out to be *really* troublesome.
>>> I wondered if anyone here can offer advice?
>>>
>>> The problem is that reads and writes to objects results in Riak CS,
>>> apparently, disconnecting the client prematurely.
>>> Nginx reports "connection reset by peer" and retries upstream Riak CS
>>> servers several times before giving up.
>>> The Riak CS access logs indicate what look like valid URLs, with a 403
>>> status indicated.
>>>
>>>
>>> Accessing the same buckets with the same auth and transferring the
>>> same files is fine when done with s3cmd, and we have other apps that
>>> have been running through nginx for a while using other S3 client
>>> libraries.
>>>
>>> A simple "check if bucket exists" command through jets3t seems to work OK.
>>>
>>> Any thoughts on what could be going on, or is this all just a bit too vague?
>>>
>>> Toby
>>
>>
>>
>> --
>> Turning and turning in the widening gyre
>> The falcon cannot hear the falconer
>> Things fall apart; the center cannot hold
>> Mere anarchy is loosed upon the world
>
>
>
> --
> Turning and turning in the widening gyre
> The falcon cannot hear the falconer
> Things fall apart; the center cannot hold
> Mere anarchy is loosed upon the world
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




More information about the riak-users mailing list