HAProxy is running, but backend is down in stats and cannot access server
-
Hi all,
I've been over the configuration a million time and read every post on this forum.
I followed the two templates that are available in PiBa's git and although it should be pretty straightforward, I cannot make it work.Let me tell you what I have and what works:
I have a pfSense box that I use as main WAN router, and until now I was forwarding traffic to a HAproxy container to do reverse proxying. Now I saw that an HAproxy is available for my pfsense and I wanted to transfer this functionality.
My container haproxy.cfg file looks like this:
global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon maxconn 4096 tune.ssl.default-dh-param 2048 # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ # An alternative list with additional directives can be obtained from # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 defaults log global mode http option httplog option dontlognull option forwardfor option http-server-close timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http #--------------------------------------------------------------------- # SSL secured routing #--------------------------------------------------------------------- frontend secured bind *:443 ssl crt /etc/ssl/web/ mode http reqadd X-Forwarded-Proto:\ https option httplog use_backend aeshome-build if { hdr_dom(host) -i jenkins.aeshome.uk } { dst_port 443 } #--------------------------------------------------------------------- # Unsecured redirection to Secured #--------------------------------------------------------------------- frontend unsecured bind *:80 reqadd X-Forwarded-Proto:\ http redirect scheme https code 301 if !{ ssl_fc } #--------------------------------------------------------------------- # AEShome Build #--------------------------------------------------------------------- backend aeshome-build mode http server node 192.168.1.17:8080 checkThis configuration works fine, very performant and no issues. Here are the NAT rules that go with this:
(I know the :80 port rule is disabled, because I am now using a native firewall to redirect to "This Firewall", more on this later)I followed the template to create multiple sites under a single IP and configured like for like.
The new rule in firewall looks like this:
But I get a 503 error that the server is not found. Also, the stats page shows the backend down with:```
L7STS/403 - Layer 7 wrong status: ForbiddenHere is the configured haproxy.cfg file on pfSense:Automaticaly generated, dont edit manually.
Generated on: 2017-06-02 11:16
global
maxconn 10000
stats socket /tmp/haproxy.socket level admin
gid 80
nbproc 1
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_statelisten HAProxyLocalStats
bind 127.0.0.1:4045 name localstats
mode http
stats enable
stats admin if TRUE
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000frontend Frontend-merged
bind 81.111.17.37:80 name 81.111.17.37:80
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
maxconn 10
timeout client 30000
acl jenkins.aeshome.uk hdr(host) -i jenkins.aeshome.uk
use_backend jenkins_http_ipv4 if jenkins.aeshome.uk
default_backend jenkins_http_ipv4backend jenkins_http_ipv4
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4@ usesrc clientip
option httpchk OPTIONS /
server node 192.168.1.17:8080 check inter 1000Please let me know if it's something obvious or if I'm missing any step. I've been trialling many different settings and combinations without success. As I said, the standalone one works flawlessly. Thanks in advance for your input and support.     -
The main differences between your working and the nonworking config are probably these:
option httpchk OPTIONS / source ipv4@ usesrc clientipThe first makes haproxy perform a OPTIONS http request to the website, but it gets a 403 response which is considered an 'error'.. On possible way to 'solve' this add in the advanced section the textual option:
http-check expect status 200Another could be to make a request to a special /haproxystatus.php page on your website that would not need authentication.
If that makes the 'status' show the server as 'up' but still have problems accessing the site from the lan network then disable the transparent-client-ip option in gui for the backend. It needs to be disabled it if you need to access the site from within the same network and still have trouble as it would create a asynchronous traffic flow which cant work with a proxy.
-
Thanks PiBA for taking time to work this out. However, it doesn't want to play ball.
I give up, it was more convenience than anything else, but I'll keep my container which works flawless.
Cheers! -
Sorry i read my reply again, and of-course the proposed change should have read "http-check expect status 403" so it sees that code as valid..
Anyhow if your willing to dig further into the problem, lemme know, ill try and help.