Question on HAProxy, VIPs, DHCP, and Firewall Rules

[bjk002]

I have 7 distinct subdomains (e.g. site1.example.com, site2.example.com, etc,...) defined at DNS provider.

I have Let's Encrypt certs for each of the 7 subdomains.

All traffic is forwarded from DNS provider on port 8843 for each sub-domain.

I have 3 interfaces (plus WAN) -> LAN(10.1.1.0/24), DMZ(10.1.2.0/24), DMZ_WEB(10.1.3.0/24)

DHCP for each interface is enabled, with each range set from 10.1.x.12 - 10.1.x.199

I have a VIP defined with IP 10.1.3.200 and an alias "HAProxy_DMZ_WEB_VIP" defined for it.

I have a "web_ports" alias (80,443, 8843)

I have NAT setup as follows, and I have Pure NAT for NAT reflection.:

I have HAProxy setup with a single frontend, with 7 distinct ACLs and corresponding actions to each of the 7 distinct backends.

I have 7 "Host Overrides" defined under DNS resolver, one for each of the subdomains, pointing to the VIP noted earlier.

Here are my WAN Rules:

Here are my LAN rules:

Here are my DMz rules:

And here are my DMZ_WEB rules:

If on LAN..., everything is working as expected. I am reaching the haproxy backends defined for each of the respective ACLs and being served the appropriate certificates for the respective subdomain.

If on WAN..., I am only reaching one of the ACLs.

I am guessing my rules are not quite right, but I've been staring for quite some time and cannot see a solution.

The question I had pertained to this note from the documentation:

I've read that 100 times. I'm still not clear. My rules are using "Interface NET" for sources, but NAT routes to the HAProxy VIP. Is the VIP included in the "Interface NET", or not?

And, if you see some other obvious error, please chime in. I'm really not a networking expert by any stretch of the imagination.

[PiBa]

@bjk002
You have some 'provider' that sends all webrequests for the 7 domains to your one single WANIP:8843 .. Is that http or https traffic for those domains.?.

If traffic is handled for 1 of those domain at your local haproxy that is listening for the traffic, then than you can stop looking at firewall and nat rules on that side... As all the traffic would be passed along using the same rule which apparently seems to be correct.. (though i'm not 100% sure about the use of a port-alias in the portforward rule... perhaps try and make 3 separate rules.?.) but still is traffic arriving on 8843 or on 80?

As for the interface-net, if you want the details, checkout the /tmp/rules.debug file it should have all implicit pfSense rules written out to a 'pf' configuration file there.. Though i doubt it matters..

All backends are 'up' in the haproxy stats? Sure the local connections are really passing through haproxy? What config does haproxy have?

[bjk002]

Hi PiBa,

All traffic is forwarded to 8843 and HTTPS.

The one subdomain that was handled happened to be the default backend... so, after further consideration, this may not have been working for any of the domains.

But your comment on the NAT rules and port aliases got my head unstuck. This was the problem.

I separated out the NAT rules and everything is working for both LAN and WAN.

I am very happy. Thank you for the assist. And thanks for the tip regarding /tmp/rules.debug. That will come in handy in the future.

[bjk002]

So I need to resurrect this thread...

I can confirm that a backend with HTTP is working. HOWEVER, a backend with HTTPS is not working, resulting in 503 at all times.

So while the NAT rules and aliases was a problem that is now corrected, I still have an issue.

If the backend server FQDN is MYSERVER.example.com, and the web server is listening on both port 8080 (HTTP) and port 443 (HTTPS) then if I set the backend as:

I am able to reach the backend server correctly, with valid cert, and everything is wonderful.

If I flip the port to 443 in the backend and set "Encrypt(SSL)"

I am unable to reach the backend server and instead get the 503 error.

Yes, I can confirm I can reach the backend server directly via either port, using either HTTP or HTTPS. But something in the HAPROXY configuration is not allowing the HTTPS connection.

I am at a loss and can use some assistance/ideas on where to look.

Finally, for the record, the use cases require using HTTPS, so just using an HTTP backend is not a solution here.

[PiBa]

@bjk002
What does haproxy LastChk column say on the status page?

[bjk002]

Hi PiBa,

LastChk is blank. Not sure if it matters, but I do not have health checks enabled on any backend.

[PiBa]

@bjk002
Enabling http health-checks could help determine if haproxy is able to properly connect, negotiate ssl, and send a http request. And tell on which of those steps its possibly failing.

[bjk002]

OK.

Here is the full list of backends and frontend stats, with blackouts for identifying info.

Every backend defined with red status in below image is an HTTPS connection. Those in green are HTTP.

Some of those red status backends (the actual servers) are listening on both HTTP and HTTPS ports (80 / 443). I can flip the backend from HTTPS port to HTTP port and everything works (enabling/disabling SSL check as appropriate).

So I am at a loss...

I've got to believe this is some problem in my firewall rules, but I just do not see it.

[bjk002]

to prove the point, I flipped the "docs" backend to HTTP. Below is the result:

[PiBa]

@bjk002
he problem is NOT a firewall-rules problem, as all the LastChk says "L6RSP in 3ms". This means that a TCP-connection was made succesfully. But the certificate/cipher negotiation failed.. If you 'hover' with your mouse over that line it might tell a little bit more about the reason that failed.

If you are only switching the 'port' on a already configured server it 'might' remember the old port.. If you hover with your mouse over the servername it should show the ip:port its trying to connect to verify that its showing the 443 port there..

Finally can you paste the configuration of a backend as it appears in haproxy.cfg (bottom of settings tab) here when its configured for the ssl-checks?

[bjk002]

@PiBa said in Question on HAProxy, VIPs, DHCP, and Firewall Rules:

haproxy.cfg

PiBa,

Thanks for sticking with me. I hear you regarding firewall/rules.

Hovering over the "L6RSP in 6ms" yields "Layer6 invalid response: SSL handshake failure" for each backend.

I hovered over server name affiliated with each failed backend, and the server:port were correct for each.

Below is the content of haproxy.cfg for one backend with SSL. I changed domain and server names to obscure.

backend worklogs.example.com_ipvANY
	mode			http
	id			100
	log			global
	option			log-health-checks
	errorfile			503 /var/etc/haproxy/errorfile_worklogs.example.com_ipvANY_503_ErrorPage503
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			EXAMPLESERVER EXAMPLESERVER.example.com:443 id 101 ssl check inter 10000  ca-file /var/etc/haproxy/ca_5ea74728e9b92.pem crt /var/etc/haproxy/server_clientcert_5ea9847021976.pem resolvers globalresolvers

[PiBa]

@bjk002
I see you configured both a CA-certificate and a client-certificate for the server.. Are you sure a client-certificate is needed to connect to the server? Can you try without those?
If that works then only add the CA-certificate, and try again. If it still works 'maybe' add the client cert back.?.

[bjk002]

Let me give that a try. I'll report back.

[bjk002]

Confirmed. it works with no client cert.

I will need to add client cert back and retry, but I've got to run for now. TY!!!

[PiBa]

@bjk002
Okay thats good news ;) . As for the client-certificate, when you connect with a browser to the servers, do they ask you for sending/selecting a client certificate from your browser? (This would mean most people cant visit those sites when they would connect to the servers, but maybe that intended.?.)

[bjk002]

@PiBa - real quick - yes, good news. =)

And, yes, originally, you get a prompt for the client cert. it is intended.

I'll fiddle with it later this evening and see if I can get everything to work.