X-Forwarded-For fun...

It would appear that sometimes, Apache isn't seeing any data for X-Forwarded-For in requests.

Most of the time, I get full information expected from X-Forwarded-For in apache logs, from 'CLIENTIP - - [Date]' to 'CLIENTIP, PROXYIP, PROXYIP, OURPROXYIP - - [Date]'. This has also been proven in scripting, pulling data from X-Forwarded-For.

The problem comes when there is NO CLIENTIP information, ala. '- - - [Date]' This behavior has been most common on machines in our organization.

Explanation of architecture time... LB1 has a publicly available IP on the outside interface (155.98.X.X), with a secondary interface on a private subnet that the web servers are on (192.168.8.5).

So, while requests from the outside world to 155.98.X.X are handled as expected and for the most part show information correctly in X-Forwarded-For, requests from other machines in 155.98.X.X or 155.98.Y.Y are handled by haproxy correctly, but over half the time do not have information in X-Forwarded-For (ie. '- - - [Date]'). When requests show OURPROXYIP (ie. the example in the second paragraph) or when X-Forwarded-For is not enabled for logging (using standard %h instead of %{X-Forwarded-For}i) they show as the IP of the private subnet interface on LB1 (192.168.8.5), as expected.

Another note on behavior, when a machine within our organization visits the site for the first time, it is almost guaranteed to report the X-Forwarded information correctly. Once the client starts browsing within the site from there, the X-Forwarded-For information disappears.

Relevant configurations:

Apache logging:
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

haproxy.cfg:
defaults

        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        redispatch
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000
listen www
        bind 155.98.X.X:80
        mode http
        balance roundrobin
        option forwardfor
        option httpchk HEAD /check.txt HTTP/1.0
        server web1 192.168.8.15:80 check
        server web2 192.168.8.16:80 check