SSL certs handling and HAproxy

[lewis]

@stephenw10 said in SSL certs handling and HAproxy:

It's ACME that has options for production and testing.

I looked though the settings but don't see anything about testing.
The only options are enable or disable in a couple of sections.

You are seeing no WAN side states at all?
If so are you seeing blocked traffic on WAN?

There's lots of WAN traffic, just nothing to that server.

[stephenw10]

Hmm, no states on WAN is an issue. How are you testing?

Do you see states or bytes on the WAN firewall rue that should be passing that?

The acme testing option is here:

But of nothing is even reaching WAN that's probably not the issue

[lewis]

@stephenw10 said in SSL certs handling and HAproxy:

Hmm, no states on WAN is an issue. How are you testing?

Do you see states or bytes on the WAN firewall rue that should be passing that?

The acme testing option is here:

But of nothing is even reaching WAN that's probably not the issue

The WAN seems to have a lot of traffic, just not to this web server.
Thanks for the image, it turns out it is in testing mode.
I've changed it to LE production but still no go.

[lewis]

Just to confirm, I should be seeing traffic on WAN to the web server, in this case, 10.0.0.180.

[lewis]

Alright, I do see traffic on the WAN from public IPs to the VIP.
It's looking like the problem is only on the web server now?

[stephenw10]

Yes to the VIP where the HAProxy front end is listening.

Ok those states look better, two way traffic at least. They are not going to stay established if the client is dropping the connection due to a bad cert.

If you examine the cert on the client when you try to connect is it using a valid CA?

If ACME was set to use the staging server it wouldn't be valid. It would need to update with the real production cert before clients recognise it as signed by a valid CA.

[lewis]

Yes, I think we're getting somewhere.
This is what I'm seeing about the certificate;

---
Server certificate
subject=CN = domain.com

issuer=C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4661 bytes and written 409 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)

[stephenw10]

Aha! OK then that's the first hurdle. The client is connecting to HAProxy at least.

So you need to renew the cert now it's set to the production server.

Then see what is broken next.

[lewis]

Ok, I've renewed it and no change. Still shows as staging even though I've changed it to production.
The one difference however is now I'm seeing connections to the web server but all of them are getting a 301.

The bad thing about doing this is if all I'm going to see are connections from the pfsense device IP, that's going to remove all logging data that is always needed to better understand traffic and so on.

TOR still gets an HSTS error and firefox is still complaining "Warning: Potential Security Risk Ahead".

SSL test still shows;

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)

I'm not convinced the problem is with the web server because if I set up a NAT rule again, then it works fine.
This implies there is still something on ACME or haproxy that is not allowing the SSL cert to be used properly.

Could it have something to do with how the cert is being generated by ACME since there seems to be some small mismatch between the host/hostname/domain or something and the ACME cert being generated.

There are no errors when ACME renews the cert, all green, all confirmed, looks correct.

[lewis]

I wonder if it has something to do with the fact that the cert is being handled by ACME but once the browser hits the web server, that has it's own self signed cert so maybe a mismatch?

However, I cannot fire up the service without a cert if I allow https connections so added teh self signed cert.
Why is this so complicated? No wonder I wasn't able to follow any article I could find to get this working.

[lewis]

Weird, searching online for help on this leads to so many things but mostly, our own thread now.
I cannot find anything that helps me to solve this.

[lewis]

One thing I notice is that even with the 'production' setting, the re-generated cert always shows staging;
issuer=C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3

When I re-issue and it shows the green page, there is no mention of staging anywhere.

It's nuts how many articles contradict each other when trying to find anything on this.

[kiokoman]

@lewis

so you have registered for production

now you should see something with Issuer

Acmecert: O=Let's Encrypt, CN=R3, C=US

under system / certificates /certificates

if you see it than you need to go to haproxy and change, under ssl offloading , the certificate

[lewis]

Holy crap, I finally got it working.
I don't even know where to start to show what I did because I tried so many things.
The last thing I did was in the backend, changing the server list Table from port 80 to port 443 and enabling Encrypt(SSL).

The main problem I have now is still not knowing what the correct configuration of the web server should be.
It's using a self signed cert now but this is a stand alone web server. Some of them have lots of virtualhosts.

And, what SSL options will be used? The ones on ACME/haproxy or the ones set in ssl.conf/httpd.conf on the web server?

Plus, now I'd like to fix the 'load balanced' server that never actually worked.

And, how can I forward the visiting IPs? I only see the firewall's IP in the logs which is not useful.
I found this but it doesn't send the visiting IP;

[kiokoman]

@lewis
if you can, don't use cert on web server just point to port 80
people will connect to haproxy and use the cert from ACME/haproxy but haproxy will speak to the web server with the self signed cert

on the backend add all the server that will do balance under "server list", set a Weight (10 / 20 /30 etc etc) and select round robin under load balancing

backend but carefully read the warning

[lewis]

I can't point to port 80 because all of the sites are built using https only.
Any sites I'd like to add would be https.

[kiokoman]

@lewis
not a big problem, it's just for convenience, create a self signed cert with all the domains you have

[lewis]

@kiokoman said in SSL certs handling and HAproxy:

@lewis
not a big problem, it's just for convenience, create a self signed cert with all the domains you have

What does that mean, 'with all of the domains'?
Do you mean you have to generate a self signed cert for each domain or there's a way to do one that has all of the names?

And more importantly, will https use the always updates SSL options that LE offers or will visitors end up using what ever the Apache server is configured with?

[kiokoman]

@lewis
visitors will use and see only the ACME cert

with openssl you can create 1 cert with all domains name

make a copy and edit openssl.cfg

Under the [ CA_default ] section, change the line (uncomment the line) # copy_extensions = copy to copy_extensions = copy

Under the [ v3_ca ] section,

change the line # subjectAltName=email:copy to subjectAltName=@alt_names

add a new line with extendedKeyUsage = serverAuth

Go to the end of the file and add
[ alt_names ]

if the server has multiple address expected to be used for connection, you should add multiple entries under [ alt_names ]. For example,

[ alt_names ]

DNS.1 = app.example.com

DNS.2 = app2.example.com

DNS.3 = app3.example.com

IP.1 = 192.168.1.1

IP.2 = 10.10.10.10

under cmd if you use windows:

set OPENSSL_CONF=C:\OpenSSL-Win64\bin\openssl.cfg (change the directory to where you have your modified copy of openssl.cfg)
openssl genrsa -out ServerKey.key 2048
openssl req -x509 -new -key ServerKey.key -sha256 -days 825 -out myCA.cer

if you need pkcs12

openssl pkcs12 -export -out keystore.p12 -inkey ServerKey.key -in myCA.cer

These are notes I have and they should still work... probably ...

[lewis]

Getting the visitor IP is simple enough.
In httpd.conf, I changed;

LogFormat "%v %h %l %u %t "%r" %>s %b %D %^FB "%{Referer}i" "%{User-Agent}i"" combined
to
LogFormat "%v %a %l %u %t "%r" %>s %b %D %^FB "%{Referer}i" "%{User-Agent}i"" combined
(%h to %a)

And added this directive;
RemoteIPHeader X-Forwarded-For

In haproxy, Frontend, Advanced, I enabled;

Now I see the actual visitor IP.