HAProxy is running, but backend is down in stats and cannot access server

aes · Jun 2, 2017, 11:18 AM

Hi all,

I've been over the configuration a million time and read every post on this forum.
I followed the two templates that are available in PiBa's git and although it should be pretty straightforward, I cannot make it work.

Let me tell you what I have and what works:

I have a pfSense box that I use as main WAN router, and until now I was forwarding traffic to a HAproxy container to do reverse proxying. Now I saw that an HAproxy is available for my pfsense and I wanted to transfer this functionality.

My container haproxy.cfg file looks like this:


global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        maxconn 4096
        tune.ssl.default-dh-param 2048

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option forwardfor
        option http-server-close
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

#---------------------------------------------------------------------
# SSL secured routing
#---------------------------------------------------------------------
frontend secured
        bind *:443 ssl crt /etc/ssl/web/
        mode http
        reqadd X-Forwarded-Proto:\ https
        option httplog
        use_backend aeshome-build if { hdr_dom(host) -i jenkins.aeshome.uk }     { dst_port 443 }

#---------------------------------------------------------------------
# Unsecured redirection to Secured
#---------------------------------------------------------------------
frontend unsecured
        bind *:80
        reqadd X-Forwarded-Proto:\ http
        redirect scheme https code 301 if !{ ssl_fc }

#---------------------------------------------------------------------
# AEShome Build
#---------------------------------------------------------------------
backend aeshome-build
        mode http
        server node 192.168.1.17:8080 check

This configuration works fine, very performant and no issues. Here are the NAT rules that go with this:

(I know the :80 port rule is disabled, because I am now using a native firewall to redirect to "This Firewall", more on this later)

I followed the template to create multiple sites under a single IP and configured like for like.
The new rule in firewall looks like this:

But I get a 503 error that the server is not found. Also, the stats page shows the backend down with:```
L7STS/403 - Layer 7 wrong status: Forbidden


Here is the configured haproxy.cfg file on pfSense:

Automaticaly generated, dont edit manually.

Generated on: 2017-06-02 11:16

global
maxconn 10000
stats socket /tmp/haproxy.socket level admin
gid 80
nbproc 1
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
bind 127.0.0.1:4045 name localstats
mode http
stats enable
stats admin if TRUE
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend Frontend-merged
bind 81.111.17.37:80 name 81.111.17.37:80
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
maxconn 10
timeout client 30000
acl jenkins.aeshome.uk hdr(host) -i jenkins.aeshome.uk
use_backend jenkins_http_ipv4 if jenkins.aeshome.uk
default_backend jenkins_http_ipv4

backend jenkins_http_ipv4
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4@ usesrc clientip
option httpchk OPTIONS /
server node 192.168.1.17:8080 check inter 1000


Please let me know if it's something obvious or if I'm missing any step. I've been trialling many different settings and combinations without success. As I said, the standalone one works flawlessly.

Thanks in advance for your input and support.
![NAT_rules.png](/public/_imported_attachments_/1/NAT_rules.png)
![NAT_rules.png_thumb](/public/_imported_attachments_/1/NAT_rules.png_thumb)
![Firewall_rule.png](/public/_imported_attachments_/1/Firewall_rule.png)
![Firewall_rule.png_thumb](/public/_imported_attachments_/1/Firewall_rule.png_thumb)

PiBa · Jun 10, 2017, 8:11 PM

The main differences between your working and the nonworking config are probably these:


option			httpchk OPTIONS /
source ipv4@ usesrc clientip

The first makes haproxy perform a OPTIONS http request to the website, but it gets a 403 response which is considered an 'error'.. On possible way to 'solve' this add in the advanced section the textual option:

http-check expect status 200

Another could be to make a request to a special /haproxystatus.php page on your website that would not need authentication.

If that makes the 'status' show the server as 'up' but still have problems accessing the site from the lan network then disable the transparent-client-ip option in gui for the backend. It needs to be disabled it if you need to access the site from within the same network and still have trouble as it would create a asynchronous traffic flow which cant work with a proxy.

aes · Jun 19, 2017, 3:45 PM

Thanks PiBA for taking time to work this out. However, it doesn't want to play ball.
I give up, it was more convenience than anything else, but I'll keep my container which works flawless.
Cheers!

PiBa · Jun 19, 2017, 5:37 PM

Sorry i read my reply again, and of-course the proposed change should have read "http-check expect status 403" so it sees that code as valid..
Anyhow if your willing to dig further into the problem, lemme know, ill try and help.