SSL certs handling and HAproxy
-
@stephenw10
So the setup is fairly simple right now.
Using a rule with the vip and it's set to https.
Then haproxy is in play but no varnish. The haproxy should be sending to the web server using http.
On the front end, all described above.
On the back end, all described above.The really weird thing is that I have the back end up to use HTTP for the health check but as shown above, the connections to the server are https.
Why and how could that even happen? The web server is absolutely responding to http requests as well but none come from haproxy.11:07:21.958831 ens18 In IP _gateway.14123 > r9ty01.phx.loc.https: Flags [S], seq 1069863897, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1387418 ecr 0], length 0 11:07:21.959032 ens18 Out IP r9ty01.phx.loc.https > _gateway.14123: Flags [S.], seq 3639861567, ack 1069863898, win 65160, options [mss 1460,sackOK,TS val 2268769424 ecr 1387418,nop,wscale 7], length 0 -
BTW it cannot work with https because the server is now using a self signed cert, ACME is taking care of the cert.
-
That capture shows a simple TCP SYN / SYN-ACK that happens to be on port 443. It's not an actual http or ssl check. How is the backend actually configured?
There's no reason HAProxy can't connect to the backend servers using a self signed cert as long as the CA has been imported into pfSense.
-
That capture shows a simple TCP SYN / SYN-ACK that happens to be on port 443. It's not an actual http or ssl check.
How is the backend actually configured?It only shows up when I enable the HTTP check and it's showing up at 9mn as set so what else could it be?
And more importantly, there isn't anything set up to connect to the web server on port https. Any traffic would come from haproxy.There's no reason HAProxy can't connect to the backend servers using a self signed cert as long as the CA has been imported into pfSense.
With the config as shown below, the correct acme cert is actually being used but I need the traffic to be sent using http since once this is working, I'll add varnish back into the mix. Varnish isn't working because it's receiving https traffic and the web server as things are now, is not receiving any data from clients.
The only thing not shown is the rule which I've described above. It's a rule, receiving https on the VIP so that haproxy can access.Backend;
Frontend;
-
Ok, you can see there that the http health check should perform an actual http options request against the server and see a reply.
The backend servers would not be using an ACME cert if they are selfsigned. The CA cert you need to import is whatever created the selfsigned certs.
You previously said you could not use http to the backend because the website was hardcoded with https links. Has that now been resolved?
-
@stephenw10 said in SSL certs handling and HAproxy:
Ok, you can see there that the http health check should perform an actual http options request against the server and see a reply.
Correct but it shows up on the web server as https as shown in the tcpdump.
The backend servers would not be using an ACME cert if they are selfsigned. The CA cert you need to import is whatever
created the selfsigned certs.According to the remote ssl tests I'm running, it's the correct acme cert on pfsense that is showing up. There are no ssl errors but the web server keeps getting https traffic rather than http from haproxy.
You previously said you could not use http to the backend because the website was hardcoded with https links. Has that now been resolved?
Yes, quite a number of comments back :). The web server is using a self signed cert but the acme cert is the one in play when public connections are coming in.
-
@lewis
take in mind that if initially you configured the backend as https and after that you modified the settings to use port 80 it does not work. you need to delete the backend and reconfigure it with the correct value.
there are some bugs lurking around -
First, I've been wondering if I'm coming across a bug again.
Second, I've been considering starting all over again but I've tried so many things that at this point, I think I'll make a mistake in the config.
I will give it a try now however. -
I tore it all down and rebuilt it.
There's definitely a bug alright. Doing that, now I see http heartbeat finally.
The ssl cert checks out as the one in acme on pfsense.Incoming connections to the web server are now http as they should be but, public connections are now getting;
The page isn’t redirecting properly
I know http works fine because I've tested it from other servers and the web server responds as it should.
I find it a little hard to believe that if this is a bug that it's not well known?
-
tcpdump on the web server;
12:32:54.007855 ens18 In IP _gateway.28554 > r9ty01.phx.loc.http: Flags [S], seq 3274415560, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 511869013 ecr 0], length 0 12:32:54.008023 ens18 Out IP r9ty01.phx.loc.http > _gateway.28554: Flags [S.], seq 266390645, ack 3274415561, win 65160, options [mss 1460,sackOK,TS val 2446701473 ecr 511869013,nop,wscale 7], length 0The redirection error is odd since it's not happening from any other internal client.
-
I checked using curl and a windows machine on the lan, all reach the web server as they should.
Only public connections are getting the 'redirect' error.- - [02/Jan/2024:12:55:32 -0700] "GET / HTTP/1.1" 301 4690 197049 196471 "-" "Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0" -
Just for giggles, I changed haproxy to point to varnish and that is now working too.
Meaning, it's sending the traffic to the web server using http.At this point, all I can think of is that there is something in the headers being sent from haproxy which is causing this constant 301.
-
@lewis said in SSL certs handling and HAproxy:
The page isn’t redirecting properly
How exactly are you testing when you see that? From some external IP? Using http?
Is HAProxy configured to redirect http to https?
-
I guess you missed some of my comments.
Is HAProxy configured to redirect http to https?
How exactly are you testing when you see that? From some external IP? Using http?The traffic is finally hitting the server as http so that part is solved.
I've tested from internal (LAN) clients using curl and firefox to http since the firewall is handling the ssl cert.
Internally, all is good, web pages are being served up using http.
Externally or public, everything gets a 301.From external connections, I'm connecting to https and can see the traffic being sent from haproxy to http.
-
@lewis
what are you running on the server? apache2 ?
that's usually a misconfiguration in the web server, maybe a redirection on the config that you don't need anymore -
I checked the incoming headers and for some reason, http is still being forwarded to https so I think the problem is with the web server at this point.
It's odd because there is no forwarding configured, at all on the web server so I'll spend some time on this and see what I can discover.It's darn close now that I've rebuilt the haproxy config.
-
@lewis
that error is usually from a redirect in the web server, haproxy try to comunicate with the web server via http but the server answer back via https but since haproxy is expecting HTTP traffic, it keeps resending the same request, resulting in a redirect loopnginx could have something like -> return 301 https://$server_name$request_uri;
apache2 could have something like -> RewriteEngine / RewriteCond / RewriteRule -
So what seems to be happening is that because the site was built with all urls being https, no matter if I hit it using http, the urls are all htps so the browser keeps getting https.
I dumped the application db, edited all the links to be http and it 'sort of' works.
I can fix the problems but it's not going to solve the main problem which is the sites cannot all be rebuilt to be http.
So, all this for nothing ti seems unless there's some way to get https pages working with haproxy as well. -
Hmm, if you only see that when connecting externally though that's only when the connection is going through HAProxy.
What I expected would be a redirect in the HAProxy front end. That should only ever be for http to https for client connections but it would be possible to have ot set the other way around accidentally.
-
HAProxy will work fine using https to the backends if you need it to. I have no idea if you can add Varnish to that though.
