HAproxy 503 error on secondary domain

[accidentallyadmin]

Grettings fellow pfsense users.

I'm having a bit of a problem with haproxy on pfsense. Please bear with me since English is not my first language.
I set up haproxy to serve 3 web servers from 2 different domains. Which in this thread I post as 1st and 2nd. The pfsense/haproxy itself is registered in 1st domain (hostname net1.1st). The frontend is serving from WAN address 11.22.33.44 (virtual IP) and LAN address (192.168.5.1) both on port 443. The web server from 1st domain (devoffice.1st and pve.1st) can be accessed successfully from LAN and internet, but web server from 2nd domain (www.butiktrip.2nd) is inaccessible with 503 error from both LAN and internet. For the record, all web server is accessible from LAN when using each IP address. Also, I set the host overrides in DNS resolver on pfsense to point to haproxy.
Here is the setting for the frontend:

# Automaticaly generated, dont edit manually.
# Generated on: 2024-12-03 15:53
global
	maxconn			1000
	log			/var/run/log	local0	info
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	log-send-hostname		haproxy1
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats refresh 10
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

resolvers globalresolvers
	nameserver Localhost 127.0.0.1:53
	resolve_retries 3
	timeout retry 5
	timeout resolve 10

frontend static
	bind			11.22.33.44:443 name 11.22.33.44:443   ssl crt-list /var/etc/haproxy/static.crt_list  
	bind			192.168.5.1:443 name 192.168.5.1:443   ssl crt-list /var/etc/haproxy/static.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			devoffice	var(txn.txnhost) -m str -i devoffice.1st
	acl			pve	var(txn.txnhost) -m str -i pve.1st
	acl			butiktrip	var(txn.txnhost) -m str -i www.butiktrip.2nd
	acl			aclcrt_static	var(txn.txnhost) -m reg -i ^([^\.]*)\.1st(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend devoffice_ipvANY  if  devoffice aclcrt_static
	use_backend pve_ipvANY  if  pve aclcrt_static
	use_backend butiktrip_ipvANY  if  butiktrip aclcrt_static

backend devoffice_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	load-server-state-from-file	global
	server			devoffice 192.168.5.51:443 id 101 ssl check inter 1000  verify none resolvers globalresolvers 

backend pve_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	load-server-state-from-file	global
	server			pve 192.168.5.50:8006 id 103 ssl check inter 1000  verify none resolvers globalresolvers 

backend butiktrip_ipvANY
	mode			http
	id			108
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	load-server-state-from-file	global
	server			butiktrip 192.168.5.53:443 id 109 ssl check inter 1000  verify none  resolvers globalresolvers

All SSL certificates are generated using ACME package (Let's Encrypt).

It will be gratly appreciated if anyone can point the error of my way. Thank you.

[viragomann]

@accidentallyadmin
Error 503 commonly means, that HAproxy cannot reach the backend service for whatever reason.

Did you enable the health checks in the backend?

Is the backend shown as up on the stats page?

[accidentallyadmin]

@viragomann Thank you for your reply. And yes, I did enable basic check for each server as shown in the screen capture below:

All the backends are shown as up in stats page:

Also, where do I find the details of the following part (as in what packet send to which backend, frontend response, backend response, etc)?

[viragomann]

@accidentallyadmin
Seems that HAproxy cannot access the backend, however.

What do you get if you run

curl -k -H "Host:www.butiktrip.2nd" -I https://192.168.5.53:443

from pfSense?

This shows only the reply code and headers. If you run it without "-I" it should load the page.

[accidentallyadmin]

@viragomann said in HAproxy 503 error on secondary domain:

curl -k -H "Host:www.butiktrip.2nd" -I https://192.168.5.53:443

both commands executed successfully as shown below:

[viragomann]

@accidentallyadmin
So the 503 isn't reasonable to me. If the health check is good and the backend is accessible from pfSense,it should work in my opinion.

Maybe there are some hints to find in log after enhancing the "Syslog level".

[accidentallyadmin]

@viragomann Exactly what I'm thinking, but obviously there's something missing. Could you please take another pass at screen capture below, in case there's some incorrect settings.

Frontend

Backend - devoffice.1st

Backend - pve.1st

Backend - www.butiktrip.2nd

Host overrides

Do you think I should create a new frontend for 2nd domain? Will it work, considering it will point to the same IP addresses?

[viragomann]

@accidentallyadmin
I'm missing the internal IP in the frontend, which your DNS directs to.
But I think, the issue is the same from inside your LAN and from outside. So this might not be responsible.

[accidentallyadmin]

@viragomann The internal IP is shown as "LAN address (IPv4)" in frontend settings page. But you can see it in frontend listing page like the screen capture below:

I also changed the syslog level as you suggested:

But I don't see any significant message in system log:

[accidentallyadmin]

I got it working after creating a new server to replace the one serving butiktrip.2nd