Re: HA Proxy as Amazon S3 Content Switch

On Sat, Sep 27, 2008 at 04:46:56PM -0700, John Singleton wrote:
> Hi All,
>
> I've been using HA Proxy as a straight load balancer for some time now
> and decided to check out the Layer 7 features in version 1.3.
> Specifically I am trying to get content switching working with Amazon
> S3 as the backend. I've tried just about everything but I can't seem
> to get it going right and I suspect the reason has something to do
> with the way S3 handles http.
>
> The setup:
>
> global
> log 127.0.0.1 local0
> log 127.0.0.1 local1 notice
> #log loghost local0 info
> maxconn 4096
> chroot /usr/share/haproxy
> uid 99
> gid 99
> daemon
> #debug
> #quiet
>
> defaults
> log global
> mode http
> option httplog
> option dontlognull
> retries 3
> redispatch
> maxconn 2000
> contimeout 5000
> clitimeout 300000
> srvtimeout 300000
>
>
> ##
> ## HTTP Only Frontend
> ##
>
> frontend app X.X.X.X:80
>
> mode http
> acl image_content url_sub images #simple example to trigger
> the static cdn
> use_backend static_content if image_content
> default_backend dynamic_content
>
> backend static_content
> mode http
> option httpclose
> balance roundrobin
> reqrep ^([^\ ]*)\ /(.*) \1\ /my-content-cache/web/\2
>
> server CDN s3.amazonaws.com:80
>
>
> Here's what SHOULD happen.
> 1) Url for /images/foo-image.jpg is requested
> 2) HA Proxy forwards the request to the backend, static_content
> 3) Rewrite rule rewrites request from /images/foo-image.jpg -> /my-
> content-cache/web/images/foo-image.jpg
> 4) File is served!
>
> However, I don't get that result. I simply get a "could not find
> bucket" error from S3. S3 is very picky about bucket naming, (ie, /
> web//images/image.jpg is not the same thing as /web/images/image.jpg)
> but I don't think that's the problem. Testing this setup against a
> regular apache server works fine---any idea what I am doing wrong?

If you try the request by hand by telneting to the s3 server, does it work ? Also, is it possible that the server simply does not accept your Host: header ?

Your replacement rule looks correct to me, and it should not even have a problem with double slashes nor anything like that.

Do you also have "option httpclose" on the other backend ? It would be possible that the static request is always sent as part of a keep-alive connection directed to the dynamic backend instead of being switched to the static one. In doubt, you should set "option httpclose" in the frontend.

Otherwise your conf looks OK to me. I don't have any knowledge about S3, so I could not comment on what might be wrong starting at that point.

Regards,
Willy Received on 2008/09/28 08:36