SSL certs handling and HAproxy

[kiokoman]

@lewis
about the port he was talking about this settings

the default is 80 or 443 for https, if you don't chage this port the pfsense gui take control of the traffic instead of haproxy

if you saw the pfsense cert you are near the solution, change that setting and you should see the cert of haproxy

[lewis]

One thing I mentioned and will again is that the pfsense GUI is set to a custom port without an SSL certificate.
Nothing will hit the pfsense GUI other than if someone got the correct port set in the Advanced/Admin Access/TCP port settings.
That port is also blocked and allowed to only one or two IPs so no one can get to the GUI.

Unless I'm missing something again?

Also, I double checked and there are no services on port 80 or 443 for the primary pfsense IP, all services are using ports with their own VIPs.

[stephenw10]

Great so that can't be an issue then.

So do you see states? Do you see the ACME cert on the client?

[lewis]

@stephenw10
That's the problem with posts when they get so long. I had mentioned that once or twice :).

I can check the states but we're talking about the single server that has the cert on pfsense ACME now right?
As mentioned, I can get back to the load balanced stuff later. It would be nice to see this working first.

On that web server, I've now added a self signed cert so that it can respond to https requests without needing the real FQDN cert on the server.
That cert is on pfsense ACME.

The server accepts port 80 and 443, no redirect enabled right now so I can see any traffic coming to it.

When I run an ssl test, I see the the correct ACME cert details showing up but I also see;
Trusted: We were unable to verify this certificate
In the browser, I see; Error code: SEC_ERROR_UNKNOWN_ISSUER

Oddly, I do see some connections getting to the server;

www.domain.com 10.0.0.1 - - [20/Dec/2023:14:36:19 -0700] "GET /action/social/login/facebook?ossn_ts=1702991375&ossn_token=xxx HTTP/1.1" 301 20 159565 159325 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"
and
www.domain.com 10.0.0.1 - - [20/Dec/2023:14:37:25 -0700] "GET / HTTP/1.1" 301 4691 181606 181062 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"

Notice those connections are coming from 10.0.0.1 which in this case, is the firewall so I think that's a good thing. It means it's intercepting the traffic or haproxy is, right?

Next, looking at states while trying to connect to said web server, I see nothing what so ever.

[lewis]

I'm not sure why I didn't notice before or maybe there weren't any when I looked but;
I do see connections but they are all port 80 no matter if I use http or https from a remote browser/client.

[lewis]

Using TOR, I always get an HSTS notice but using other browser, I only get a cert warning.
As mentioned, I do not have HSTS enabled on this server at the moment.

I'm confused on why some traffic is showing up however.

[stephenw10]

Ok that mostly looks good. The LAN side states should be http if HAProxy is configured to use that. It's independent of whatever the front end connections are.

I expect to see states on WAN too though, from the client to HAProxy on the VIP?

That cert error look like it might just be a cert problem. Unknown issuer is odd though. ACME setup not quite right? Still using the testing CA maybe?

[lewis]

Ok that mostly looks good. The LAN side states should be http if HAProxy is configured to use that. It's independent of
whatever the front end connections are.

The haproxy front end is set to both 443 and 80. I shared an image above of some of the settings.

I expect to see states on WAN too though, from the client to HAProxy on the VIP?

None, all LAN in the states view.

That cert error look like it might just be a cert problem. Unknown issuer is odd though. ACME setup not quite right? Still
using the testing CA maybe?

I don't see any options for live or testing. How can I check that?

[lewis]

[stephenw10]

It's ACME that has options for production and testing.

You are seeing no WAN side states at all?

If so are you seeing blocked traffic on WAN?

[lewis]

@stephenw10 said in SSL certs handling and HAproxy:

It's ACME that has options for production and testing.

I looked though the settings but don't see anything about testing.
The only options are enable or disable in a couple of sections.

You are seeing no WAN side states at all?
If so are you seeing blocked traffic on WAN?

There's lots of WAN traffic, just nothing to that server.

[stephenw10]

Hmm, no states on WAN is an issue. How are you testing?

Do you see states or bytes on the WAN firewall rue that should be passing that?

The acme testing option is here:

But of nothing is even reaching WAN that's probably not the issue

[lewis]

@stephenw10 said in SSL certs handling and HAproxy:

Hmm, no states on WAN is an issue. How are you testing?

Do you see states or bytes on the WAN firewall rue that should be passing that?

The acme testing option is here:

But of nothing is even reaching WAN that's probably not the issue

The WAN seems to have a lot of traffic, just not to this web server.
Thanks for the image, it turns out it is in testing mode.
I've changed it to LE production but still no go.

[lewis]

Just to confirm, I should be seeing traffic on WAN to the web server, in this case, 10.0.0.180.

[lewis]

Alright, I do see traffic on the WAN from public IPs to the VIP.
It's looking like the problem is only on the web server now?

[stephenw10]

Yes to the VIP where the HAProxy front end is listening.

Ok those states look better, two way traffic at least. They are not going to stay established if the client is dropping the connection due to a bad cert.

If you examine the cert on the client when you try to connect is it using a valid CA?

If ACME was set to use the staging server it wouldn't be valid. It would need to update with the real production cert before clients recognise it as signed by a valid CA.

[lewis]

Yes, I think we're getting somewhere.
This is what I'm seeing about the certificate;

---
Server certificate
subject=CN = domain.com

issuer=C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4661 bytes and written 409 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)

[stephenw10]

Aha! OK then that's the first hurdle. The client is connecting to HAProxy at least.

So you need to renew the cert now it's set to the production server.

Then see what is broken next.

[lewis]

Ok, I've renewed it and no change. Still shows as staging even though I've changed it to production.
The one difference however is now I'm seeing connections to the web server but all of them are getting a 301.

The bad thing about doing this is if all I'm going to see are connections from the pfsense device IP, that's going to remove all logging data that is always needed to better understand traffic and so on.

TOR still gets an HSTS error and firefox is still complaining "Warning: Potential Security Risk Ahead".

SSL test still shows;

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)

I'm not convinced the problem is with the web server because if I set up a NAT rule again, then it works fine.
This implies there is still something on ACME or haproxy that is not allowing the SSL cert to be used properly.

Could it have something to do with how the cert is being generated by ACME since there seems to be some small mismatch between the host/hostname/domain or something and the ACME cert being generated.

There are no errors when ACME renews the cert, all green, all confirmed, looks correct.

[lewis]

I wonder if it has something to do with the fact that the cert is being handled by ACME but once the browser hits the web server, that has it's own self signed cert so maybe a mismatch?

However, I cannot fire up the service without a cert if I allow https connections so added teh self signed cert.
Why is this so complicated? No wonder I wasn't able to follow any article I could find to get this working.