Haproxy to DMZ not working



  • Everyone,

    I'm trying to setup Haproxy to load balance between 4 servers hosting an NGNIX server listening on port 3001 that I have set up in my DMZ. I've tested each of the servers from the Lan and they are working properly. I've also checked Haproxy and it is also reporting that the servers are alive as well. However, when I try to access the site from the internet I get a 503 error and I can't seem to find anything in the logs to correspond with the error.

    To see if I did something wrong in the DMZ, I moved the servers on to my LAN. I then reconfigured the Haproxy backends to point to the servers in the LAN. In this configuration, I was able to successfully reach the servers, so I've determined its probably something I haven't configured correctly in the DMZ. Currently, my DMZ is locked down and I'm only allowing egress traffic over a few ports to allow the web applications to work. Are there any other rules I need to add to allow Haproxy to route the requests properly?

    I'm new to Haproxy & pfSense so I apologize if I'm missing something obvious.

    Thanks in advance,
    Harry



  • @hhbarnes
    What does your haproxy.cfg file look like? (can be seen on the bottom of settings tab of haproxy)



  • Sure thing, I've attached two configurations, the first is when the server is on my LAN, the second is when it is on the DMZ. I've also attached the firewall logs for the two configurations as well. I noticed that the DMZ configuration is blocking communication from the firewall to the client, but there are no issues when it is connected to the LAN. Another observation is that on the Haproxy dashboard widget I can see the session number increase when I request the page from my browser so it looks like Haproxy is doing its job correctly.

    ===========================================
    Configuration when Server is on LAN (Works)
    ===========================================
    Content
    # Automaticaly generated, dont edit manually.
    # Generated on: 2019-02-02 18:30
    global
    	maxconn			1000
    	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
    	gid			80
    	nbproc			1
    	nbthread			1
    	hard-stop-after		15m
    	chroot				/tmp/haproxy_chroot
    	daemon
    	tune.ssl.default-dh-param	2048
    	server-state-file /tmp/haproxy_server_state
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:2200 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats show-legends
    	stats uri /haproxy/haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend shared-frontend-merged
    	bind			71.XXX.XXX.XXX:443 name 71.XXX.XXX.XXX:443   ssl crt-list /var/etc/haproxy/shared-frontend.crt_list  
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^AAAAAA\.in(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.AAAAAA\.in(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.BBBBBB\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.CCCCCC\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.DDDDDD\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^BBBBBB\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^CCCCCC\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^DDDDDD\.com(:([0-9]){1,5})?$
    	acl			BBBBBB.com	var(txn.txnhost) -m end -i BBBBBB.com
    	http-request set-var(txn.txnhost) hdr(host)
    	use_backend BBBBBB.LAN_ipv4  if  BBBBBB.com 
    
    frontend http-to-https
    	bind			71.XXX.XXX.XXX:80 name 71.XXX.XXX.XXX:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	http-request redirect scheme https 
    
    backend BBBBBB.LAN_ipv4
    	mode			http
    	id			10107
    	log			global
    	option			log-health-checks
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk GET / 
    	server			pine-04 192.168.1.44:3001 id 10101 check inter 1000
    	
    ====================================================
    Firewall Logs for Page when server is on LAN (Works)
    ====================================================
    Feb 2 18:37:59	WAN	 Allow HTTPS Inbound (1546899421)	  107.182.230.243:33164	  71.XXX.XXX.XXX:443	TCP:S
    Feb 2 18:37:59	WAN	 Allow HTTPS Inbound (1546899421)	  107.182.230.243:33165	  71.XXX.XXX.XXX:443	TCP:S
    Feb 2 18:37:59	WAN	 Allow HTTPS Inbound (1546899421)	  107.182.230.243:33163	  71.XXX.XXX.XXX:443	TCP:S
    Feb 2 18:37:59	WAN	 Allow HTTPS Inbound (1546899421)	  107.182.230.243:33162	  71.XXX.XXX.XXX:443	TCP:S
    Feb 2 18:37:59	WAN	 Allow HTTPS Inbound (1546899421)	  107.182.230.243:33160	  71.XXX.XXX.XXX:443	TCP:S
    Feb 2 18:37:59	WAN	 Allow HTTPS Inbound (1546899421)	  107.182.230.243:33141	  71.XXX.XXX.XXX:443	TCP:S
    
    
    
    ==================================================
    Configuration when Server is on DMZ (Doesn't work)
    ==================================================
    # Automaticaly generated, dont edit manually.
    # Generated on: 2019-02-02 18:47
    global
    	maxconn			1000
    	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
    	gid			80
    	nbproc			1
    	nbthread			1
    	hard-stop-after		15m
    	chroot				/tmp/haproxy_chroot
    	daemon
    	tune.ssl.default-dh-param	2048
    	server-state-file /tmp/haproxy_server_state
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:2200 name localstats
    	mode http
    	stats enable
    	stats admin if TRUE
    	stats show-legends
    	stats uri /haproxy/haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend shared-frontend-merged
    	bind			71.XXX.XXX.XXX:443 name 71.XXX.XXX.XXX:443   ssl crt-list /var/etc/haproxy/shared-frontend.crt_list  
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^AAAAAA\.in(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.AAAAAA\.in(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.BBBBBBBBBBBBB\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.CCCCCCC\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^([^\.]*)\.DDDDDDD\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^BBBBBBBBBBBBB\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^CCCCCCC\.com(:([0-9]){1,5})?$
    	acl			aclcrt_shared-frontend	var(txn.txnhost) -m reg -i ^DDDDDDD\.com(:([0-9]){1,5})?$
    	acl			BBBBBBBBBBBBB.com	var(txn.txnhost) -m end -i BBBBBBBBBBBBB.com
    	http-request set-var(txn.txnhost) hdr(host)
    	use_backend BBBBBBBBBBBBB.com_ipv4  if  BBBBBBBBBBBBB.com 
    
    frontend http-to-https
    	bind			71.XXX.XXX.XXX:80 name 71.XXX.XXX.XXX:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	http-request redirect scheme https 
    
    backend BBBBBBBBBBBBB.com_ipv4
    	mode			http
    	id			10102
    	log			global
    	option			log-health-checks
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source ipv4@ usesrc clientip
    	option			httpchk GET / 
    	server			pine-01 10.0.1.41:3001 id 10101 check inter 1000
    	
    	
    ===========================================================
    Firewall Logs for Page when server is on DMZ (Doesn't Work)
    ===========================================================
    Feb 2 19:01:21	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:45513	  107.182.230.242:53148	TCP:SA
    Feb 2 19:01:19	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:12325	  107.182.230.242:18353	TCP:SA
    Feb 2 19:01:18	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:10087	  107.182.230.242:53148	TCP:SA
    Feb 2 19:01:17	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:7195	  107.182.230.242:53148	TCP:SA
    Feb 2 19:01:15	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:12836	  107.182.230.242:53148	TCP:SA
    Feb 2 19:01:13	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:13321	  107.182.230.242:53148	TCP:SA
    Feb 2 19:01:12	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:53919	  107.182.230.242:53148	TCP:SA
    Feb 2 19:01:12	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:15333	  107.182.230.242:18362	TCP:SA
    Feb 2 19:01:05	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:39372	  107.182.230.242:18353	TCP:SA
    Feb 2 19:01:04	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:56889	  107.182.230.242:18362	TCP:SA
    Feb 2 19:01:02	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:6576	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:58	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:29797	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:57	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:25852	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:57	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:38859	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:54	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:35318	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:54	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:48667	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:51	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:53569	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:50	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:44300	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:49	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:3502	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:48	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:37541	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:47	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:53128	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:47	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:14512	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:45	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:7715	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:44	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:15218	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:43	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:47133	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:42	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:16575	  107.182.230.242:18362	TCP:SA
    Feb 2 19:00:42	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:63709	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:41	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:48244	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:40	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:2504	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:38	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:18308	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:36	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:55553	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:35	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:10284	  107.182.230.242:18353	TCP:SA
    Feb 2 19:00:34	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:31715	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:34	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:8533	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:32	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:41424	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:28	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:12983	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:27	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:25773	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:25	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:19495	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:25	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:40138	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:22	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:49320	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:21	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:9931	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:20	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:49203	  107.182.230.242:58725	TCP:SA
    Feb 2 19:00:19	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:3118	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:18	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:31800	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:16	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:1879	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:16	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:37333	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:15	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:55368	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:13	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:51370	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:12	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:48781	  107.182.230.242:18332	TCP:SA
    Feb 2 19:00:11	► WAN	Default deny rule IPv4 (1000000104)	  71.XXX.XXX.XXX:60948	  107.182.230.242:18324	TCP:SA
    Feb 2 19:00:12	WAN	 Allow HTTPS Inbound (1546899421)	  107.182.230.242:48384	  71.XXX.XXX.XXX:443	TCP:S


  • @hhbarnes
    As you are using the 'Transparent ClientIP' please make sure that when you change the IP of the server in the configuration, you also sset the correct interface to put the reply-capture-rule on in the backend advanced setting. If this is not set to the right interface, that would explain the Syn-Ack packets from the backend not getting redirected to the haproxy process but trying to leave on the wan interface.



  • @piba I owe you one, that was exactly the problem. I had it configured on the LAN first and then copied the configuration and I thought I changed everything to point to the DMZ. Once I changed the interface on the transparent IP it worked.

    Thanks again!