Haproxy to DMZ not working
-
Everyone,
I'm trying to setup Haproxy to load balance between 4 servers hosting an NGNIX server listening on port 3001 that I have set up in my DMZ. I've tested each of the servers from the Lan and they are working properly. I've also checked Haproxy and it is also reporting that the servers are alive as well. However, when I try to access the site from the internet I get a 503 error and I can't seem to find anything in the logs to correspond with the error.
To see if I did something wrong in the DMZ, I moved the servers on to my LAN. I then reconfigured the Haproxy backends to point to the servers in the LAN. In this configuration, I was able to successfully reach the servers, so I've determined its probably something I haven't configured correctly in the DMZ. Currently, my DMZ is locked down and I'm only allowing egress traffic over a few ports to allow the web applications to work. Are there any other rules I need to add to allow Haproxy to route the requests properly?
I'm new to Haproxy & pfSense so I apologize if I'm missing something obvious.
Thanks in advance,
Harry
-
@hhbarnes
What does your haproxy.cfg file look like? (can be seen on the bottom of settings tab of haproxy)
-
Sure thing, I've attached two configurations, the first is when the server is on my LAN, the second is when it is on the DMZ. I've also attached the firewall logs for the two configurations as well. I noticed that the DMZ configuration is blocking communication from the firewall to the client, but there are no issues when it is connected to the LAN. Another observation is that on the Haproxy dashboard widget I can see the session number increase when I request the page from my browser so it looks like Haproxy is doing its job correctly.
=========================================== Configuration when Server is on LAN (Works) =========================================== Content # Automaticaly generated, dont edit manually. # Generated on: 2019-02-02 18:30 global maxconn 1000 stats socket /tmp/haproxy.socket level admin expose-fd listeners gid 80 nbproc 1 nbthread 1 hard-stop-after 15m chroot /tmp/haproxy_chroot daemon tune.ssl.default-dh-param 2048 server-state-file /tmp/haproxy_server_state listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats show-legends stats uri /haproxy/haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend shared-frontend-merged bind 71.XXX.XXX.XXX:443 name 71.XXX.XXX.XXX:443 ssl crt-list /var/etc/haproxy/shared-frontend.crt_list mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^AAAAAA\.in(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.AAAAAA\.in(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.BBBBBB\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.CCCCCC\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.DDDDDD\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^BBBBBB\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^CCCCCC\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^DDDDDD\.com(:([0-9]){1,5})?$ acl BBBBBB.com var(txn.txnhost) -m end -i BBBBBB.com http-request set-var(txn.txnhost) hdr(host) use_backend BBBBBB.LAN_ipv4 if BBBBBB.com frontend http-to-https bind 71.XXX.XXX.XXX:80 name 71.XXX.XXX.XXX:80 mode http log global option http-keep-alive timeout client 30000 http-request redirect scheme https backend BBBBBB.LAN_ipv4 mode http id 10107 log global option log-health-checks timeout connect 30000 timeout server 30000 retries 3 source ipv4@ usesrc clientip option httpchk GET / server pine-04 192.168.1.44:3001 id 10101 check inter 1000 ==================================================== Firewall Logs for Page when server is on LAN (Works) ==================================================== Feb 2 18:37:59 WAN Allow HTTPS Inbound (1546899421) 107.182.230.243:33164 71.XXX.XXX.XXX:443 TCP:S Feb 2 18:37:59 WAN Allow HTTPS Inbound (1546899421) 107.182.230.243:33165 71.XXX.XXX.XXX:443 TCP:S Feb 2 18:37:59 WAN Allow HTTPS Inbound (1546899421) 107.182.230.243:33163 71.XXX.XXX.XXX:443 TCP:S Feb 2 18:37:59 WAN Allow HTTPS Inbound (1546899421) 107.182.230.243:33162 71.XXX.XXX.XXX:443 TCP:S Feb 2 18:37:59 WAN Allow HTTPS Inbound (1546899421) 107.182.230.243:33160 71.XXX.XXX.XXX:443 TCP:S Feb 2 18:37:59 WAN Allow HTTPS Inbound (1546899421) 107.182.230.243:33141 71.XXX.XXX.XXX:443 TCP:S ================================================== Configuration when Server is on DMZ (Doesn't work) ================================================== # Automaticaly generated, dont edit manually. # Generated on: 2019-02-02 18:47 global maxconn 1000 stats socket /tmp/haproxy.socket level admin expose-fd listeners gid 80 nbproc 1 nbthread 1 hard-stop-after 15m chroot /tmp/haproxy_chroot daemon tune.ssl.default-dh-param 2048 server-state-file /tmp/haproxy_server_state listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats show-legends stats uri /haproxy/haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend shared-frontend-merged bind 71.XXX.XXX.XXX:443 name 71.XXX.XXX.XXX:443 ssl crt-list /var/etc/haproxy/shared-frontend.crt_list mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^AAAAAA\.in(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.AAAAAA\.in(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.BBBBBBBBBBBBB\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.CCCCCCC\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^([^\.]*)\.DDDDDDD\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^BBBBBBBBBBBBB\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^CCCCCCC\.com(:([0-9]){1,5})?$ acl aclcrt_shared-frontend var(txn.txnhost) -m reg -i ^DDDDDDD\.com(:([0-9]){1,5})?$ acl BBBBBBBBBBBBB.com var(txn.txnhost) -m end -i BBBBBBBBBBBBB.com http-request set-var(txn.txnhost) hdr(host) use_backend BBBBBBBBBBBBB.com_ipv4 if BBBBBBBBBBBBB.com frontend http-to-https bind 71.XXX.XXX.XXX:80 name 71.XXX.XXX.XXX:80 mode http log global option http-keep-alive timeout client 30000 http-request redirect scheme https backend BBBBBBBBBBBBB.com_ipv4 mode http id 10102 log global option log-health-checks timeout connect 30000 timeout server 30000 retries 3 source ipv4@ usesrc clientip option httpchk GET / server pine-01 10.0.1.41:3001 id 10101 check inter 1000 =========================================================== Firewall Logs for Page when server is on DMZ (Doesn't Work) =========================================================== Feb 2 19:01:21 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:45513 107.182.230.242:53148 TCP:SA Feb 2 19:01:19 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:12325 107.182.230.242:18353 TCP:SA Feb 2 19:01:18 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:10087 107.182.230.242:53148 TCP:SA Feb 2 19:01:17 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:7195 107.182.230.242:53148 TCP:SA Feb 2 19:01:15 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:12836 107.182.230.242:53148 TCP:SA Feb 2 19:01:13 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:13321 107.182.230.242:53148 TCP:SA Feb 2 19:01:12 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:53919 107.182.230.242:53148 TCP:SA Feb 2 19:01:12 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:15333 107.182.230.242:18362 TCP:SA Feb 2 19:01:05 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:39372 107.182.230.242:18353 TCP:SA Feb 2 19:01:04 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:56889 107.182.230.242:18362 TCP:SA Feb 2 19:01:02 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:6576 107.182.230.242:18362 TCP:SA Feb 2 19:00:58 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:29797 107.182.230.242:18362 TCP:SA Feb 2 19:00:57 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:25852 107.182.230.242:18332 TCP:SA Feb 2 19:00:57 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:38859 107.182.230.242:18353 TCP:SA Feb 2 19:00:54 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:35318 107.182.230.242:18362 TCP:SA Feb 2 19:00:54 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:48667 107.182.230.242:18353 TCP:SA Feb 2 19:00:51 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:53569 107.182.230.242:18362 TCP:SA Feb 2 19:00:50 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:44300 107.182.230.242:18353 TCP:SA Feb 2 19:00:49 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:3502 107.182.230.242:18324 TCP:SA Feb 2 19:00:48 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:37541 107.182.230.242:18362 TCP:SA Feb 2 19:00:47 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:53128 107.182.230.242:18353 TCP:SA Feb 2 19:00:47 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:14512 107.182.230.242:18362 TCP:SA Feb 2 19:00:45 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:7715 107.182.230.242:18362 TCP:SA Feb 2 19:00:44 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:15218 107.182.230.242:18353 TCP:SA Feb 2 19:00:43 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:47133 107.182.230.242:18362 TCP:SA Feb 2 19:00:42 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:16575 107.182.230.242:18362 TCP:SA Feb 2 19:00:42 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:63709 107.182.230.242:18332 TCP:SA Feb 2 19:00:41 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:48244 107.182.230.242:18353 TCP:SA Feb 2 19:00:40 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:2504 107.182.230.242:18353 TCP:SA Feb 2 19:00:38 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:18308 107.182.230.242:18353 TCP:SA Feb 2 19:00:36 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:55553 107.182.230.242:18353 TCP:SA Feb 2 19:00:35 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:10284 107.182.230.242:18353 TCP:SA Feb 2 19:00:34 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:31715 107.182.230.242:18324 TCP:SA Feb 2 19:00:34 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:8533 107.182.230.242:18332 TCP:SA Feb 2 19:00:32 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:41424 107.182.230.242:18332 TCP:SA Feb 2 19:00:28 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:12983 107.182.230.242:18332 TCP:SA Feb 2 19:00:27 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:25773 107.182.230.242:18324 TCP:SA Feb 2 19:00:25 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:19495 107.182.230.242:18332 TCP:SA Feb 2 19:00:25 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:40138 107.182.230.242:18324 TCP:SA Feb 2 19:00:22 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:49320 107.182.230.242:18332 TCP:SA Feb 2 19:00:21 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:9931 107.182.230.242:18324 TCP:SA Feb 2 19:00:20 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:49203 107.182.230.242:58725 TCP:SA Feb 2 19:00:19 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:3118 107.182.230.242:18332 TCP:SA Feb 2 19:00:18 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:31800 107.182.230.242:18324 TCP:SA Feb 2 19:00:16 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:1879 107.182.230.242:18332 TCP:SA Feb 2 19:00:16 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:37333 107.182.230.242:18332 TCP:SA Feb 2 19:00:15 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:55368 107.182.230.242:18324 TCP:SA Feb 2 19:00:13 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:51370 107.182.230.242:18332 TCP:SA Feb 2 19:00:12 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:48781 107.182.230.242:18332 TCP:SA Feb 2 19:00:11 ► WAN Default deny rule IPv4 (1000000104) 71.XXX.XXX.XXX:60948 107.182.230.242:18324 TCP:SA Feb 2 19:00:12 WAN Allow HTTPS Inbound (1546899421) 107.182.230.242:48384 71.XXX.XXX.XXX:443 TCP:S
-
@hhbarnes
As you are using the 'Transparent ClientIP' please make sure that when you change the IP of the server in the configuration, you also sset the correct interface to put the reply-capture-rule on in the backend advanced setting. If this is not set to the right interface, that would explain the Syn-Ack packets from the backend not getting redirected to the haproxy process but trying to leave on the wan interface.
-
@piba I owe you one, that was exactly the problem. I had it configured on the LAN first and then copied the configuration and I thought I changed everything to point to the DMZ. Once I changed the interface on the transparent IP it worked.
Thanks again!