haproxy - host is responding to IP, but URL returns a secure 503 - why?

NickyDoes

I'm new to haproxy, and to setting up reverse proxies. I'm unclear how to set up good haproxy logging, so that may be the first question.

background

The network is a WAN plus 2 LANs

homelan: 10.71.73.0/24
prolan: 10.10.1.0/24
rules allow outbound traffic from home and pro to WAN, and allow select traffic between home and pro.

firewall is on homelan 10.71.73.1
host is on prolan 10.10.1.15:80
cert is wildcard to prolan *.example.com
dns overrides work - digging host.example.com returns 10.10.1.15

Trying to reach host from LAN gives 503 with valid certificate.
Browsing to the host IP responds normally (unsecured)
haproxy backend health check fails, but curl responds normally.

WAN to ha_frontend is SSL (https://host.example.com:443)
ha_backend to host is responds / 10.10.1.15:80

browsing to https://host.example.com gives 503 error
browsing to 10.10.1.15:80 returns the correct full page
from pfsense shell: curl -I http://10.10.1.15:80 -> HTTP/1.1 200 OK
ha_backend health check fails
firewall logs show no blocked traffic on 10.10.1.15.

questions

I'm stuck because all tests say 10.10.1.15:80 works, and I can reach https://host.example.com securely. It seems like haproxy just isn't making the connection.

how to view haproxy activity?
why is health check failing? (is it a HTTP/1.0 vs HTTP/1.1 thing? How to fix?)
???

viragomann

@NickyDoes
Don't let the health check fail!
Either configure it properly, so that it succeed, or disable the health check.

Basic health check should work in your case. It just probes the TCP connection to the backend IP:port.

NickyDoes

@viragomann I understand, and that's why I posted this. I tried disabling health checks, too.

Got any specific guidance? I'm still searching and experimenting.

viragomann

@NickyDoes
Error 503 basically means, "service unavailable". If this response is coming from HAproxy the backend device doesn't respond to forwarded requests.
So how did you configure the backend exactly?

NickyDoes

I created two VMs with simple nginx servers on them: 10.71.73.200/24 and 10.71.73.201/24 and set them up as backends in haproxy. I only get 503 errors, and the health checks never pass. I tried disabling health checks, but still get 503s.

This should be simple. Either something is wrong with the system, or I'm missing something.

When I attempt to apply changes:

curl test from the firewall

[2.7.2-RELEASE][root@firewall.demarcohome.home.arpa]/var/etc/haproxy: curl 10.71.73.200
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body>
<h1>Welcome to haproxy1</h1>
</body>
</html>

haproxy.cfg

# Automaticaly generated, dont edit manually.
# Generated on: 2024-08-15 13:17
global
        maxconn                 500
        stats socket /tmp/haproxy.socket level admin  expose-fd listeners
        uid                     80
        gid                     80
        nbthread                        1
        hard-stop-after         15m
        chroot                          /tmp/haproxy_chroot
        daemon
        server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
        bind 127.0.0.1:10000 name localstats
        mode http
        stats enable
        stats admin if TRUE
        stats show-legends
        stats uri /haproxy/haproxy_stats.php?haproxystats=1
        timeout client 5000
        timeout connect 5000
        timeout server 5000

frontend wan_frontend
        bind                    136.54.116.71:443 name 136.54.116.71:443   ssl crt-list /var/etc/haproxy/wan_frontend.crt_list
        mode                    http
        log                     global
        option                  httplog
        option                  http-keep-alive
        timeout client          30000
        acl                     haproxy1        var(txn.txnhost) -m str -i haproxy1.demarcohome.com
        acl                     haproxy2        var(txn.txnhost) -m str -i haproxy2.demarcohome.com
        acl                     aclcrt_wan_frontend     var(txn.txnhost) -m reg -i ^([^\.]*)\.demarcohome\.com(:([0-9]){1,5})?$
        http-request set-var(txn.txnhost) hdr(host)
        use_backend haproxy1_ipvANY  if  haproxy1 aclcrt_wan_frontend
        use_backend haproxy2_ipvANY  if  haproxy2 aclcrt_wan_frontend

backend haproxy1_ipvANY
        mode                    http
        id                      100
        log                     global
        option                  log-health-checks
        http-check              send meth GET
        timeout connect         30000
        timeout server          30000
        retries                 3
        load-server-state-from-file     global
        option                  httpchk
        server                  haproxy1 10.71.73.200:80 id 101 check inter 1000

backend haproxy2_ipvANY
        mode                    http
        id                      102
        log                     global
        timeout connect         30000
        timeout server          30000
        retries                 3
        load-server-state-from-file     global
        server                  haproxy2 10.71.73.201:80 id 103

NickyDoes

Responding to my own issue. Solved by completely purging the haproxy package and configuration, then reinstalling haproxy and reconfiguring.

My steps to purge haproxy from pfSense:

Remove all backends, then remove all frontends.
Remove the haproxy package
Archive (or remove) the haproxy config at /var/etc/haproxy
Examine the pfsense config for haproxy details cat /conf/config.xml | grep haproxy. Expect some haproxy config details, but nothing specific to your installation.

Next, reinstall the haproxy package.