SSL certs handling and HAproxy

lewis

The main problem I'm having right now is no matter what I do, I see https connections to the varnish server from haproxy.

I'm using a rule for the public ip to haproxy. acme is correctly handling the ssl cert.
In the hap front end, I have the address as the VIP with https offloading.
In the hap backend, I have the server list set to send to the varnish server on port 80.

No matter what I've done so far, I still see https getting to varnish.

stephenw10

Is varnish sending a redirect to https maybe?

But varnish can do load balancing so why not just use it for that and remove HAProxy from the setup?

lewis

@stephenw10
As mentioned, I've been doing a lot of searching and trial and error trying to solve this and what I've learned is that while varnish can do it, haproxy has a lot more functionality.

That said, I'm not sure I need all that functionality but I do like to keep things separated to avoid putting too much load on any one thing.
But it's a moot point because no matter which way I go, it's not working.
If I use a NAT rule to send to varnish or a rule to send to haproxy, the traffic to varnish is always https.
Even when I get rid of haproxy using NAT, the traffic is always https.

I've checked countless times and don't see why.

lewis

I've tried this as well.

I've disabled the haproxy service.
I've created a NAT rule to to accept incoming connections to the VIP on port https forwarded to http/varnish.

When testing the ssl cert from remote, I get;
"Handshake failed, we haven't received any certificates from the requested server."

So it seems acme cannot do this on its own.

I've read this;
pfSense can use the ACME package to issue and renew certificates, but it does not natively handle SSL termination for services behind it, like Varnish, without a load balancer or reverse proxy (like HAProxy). Typically, the SSL termination for incoming traffic before it reaches Varnish is handled by either a load balancer or a web server that supports SSL.

If you want pfSense/ACME to handle the SSL and then pass the traffic to Varnish, you would usually still need an SSL termination point on pfSense. This is not a built-in feature for pfSense and usually requires a package like HAProxy or stunnel.

Since you prefer not to use HAProxy, you might explore other methods such as:

Using stunnel on pfSense for SSL termination.
Setting up an SSL-capable reverse proxy on another server between pfSense and Varnish.
Directly terminating SSL at Varnish using hitch or nginx as a TLS proxy in front of Varnish.

Each of these methods would require installing and configuring additional software either on pfSense or another server in your environment.

lewis

I'm at the point where the main problem I'm facing is that haproxy will not use http to reach the web server.
Initially, I had it pointed to varnish which was getting the traffic but again over https so I simplified the setup to see what's going on.
It seems no matter how I set things up, haproxy always thinks that port 80 is down on the web server which it is not, tested many many times.
Even the hearbeat which is set to http hits the server using https.

I've probably had this config working several times but since the traffic won't get to the server as http, it never works.

I have a rule, no NAT, pointing to the VIP using port 443.
haproxy external IP is set to the VIP, port 443, ssl offloading and that's also the Type.
ACLs, certs, etc, all set correctly on the front end.

On the backend, I have Server list, Forwardto set to Address+Port, LAN IP of the web server, port 80, no ssl, or checks.
I have the correct client cert selected.

Health check is set to HTTP.

Still the connections from haproxy to the web server are https.
This is from tcpdump on the web server;

09:16:12.629794 ens18 In  IP _gateway.40275 > r9ty01.phx.loc.https: Flags [S], seq 3084197971, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1539290509 ecr 0], length 0
09:16:12.629960 ens18 Out IP r9ty01.phx.loc.https > _gateway.40275: Flags [S.], seq 1778053685, ack 3084197972, win 65160, options [mss 1460,sackOK,TS val 2262100095 ecr 1539290509,nop,wscale 7], length 0

Every time I save the haproxy config I get this warning;

NOTICE] (37014) : haproxy version is 2.7.8-58c657f
[NOTICE] (37014) : path to executable is /usr/local/sbin/haproxy
[WARNING] (37014) : config : Server Tyoubackend_ipvANY/typort80 is DOWN, changed from server-state after a reload. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

But it's not true, port 80 is definitely open to all traffic on that web server.

stephenw10

@lewis said in SSL certs handling and HAproxy:

I've created a NAT rule to to accept incoming connections to the VIP on port https forwarded to http/varnish.

I would expect that to fail since the client expects an https connection and the server receives it on the http port.

I thought the problem was that the actual website that is being accessed is hard coded to use https? In which case the backend connection from HAproxy would need to be https. Which should certainly be possible.

I would try to get this working with just HAProxy and one backend server first.

lewis

@stephenw10

That was a previous post, I've moved on since. I was trying all kinds of things, NAT and just rules.
My last comment explains where I'm at now and yes, I'm trying to get this working direct to the web server first to remove complexities.
Varnish was also getting https connections based on the last comment.

stephenw10

Ok so the backend server can now use http? Nothing there is hardcoded to use https?

It should still work with https.

lewis

@stephenw10
So the setup is fairly simple right now.
Using a rule with the vip and it's set to https.
Then haproxy is in play but no varnish. The haproxy should be sending to the web server using http.
On the front end, all described above.
On the back end, all described above.

The really weird thing is that I have the back end up to use HTTP for the health check but as shown above, the connections to the server are https.
Why and how could that even happen? The web server is absolutely responding to http requests as well but none come from haproxy.

11:07:21.958831 ens18 In  IP _gateway.14123 > r9ty01.phx.loc.https: Flags [S], seq 1069863897, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1387418 ecr 0], length 0
11:07:21.959032 ens18 Out IP r9ty01.phx.loc.https > _gateway.14123: Flags [S.], seq 3639861567, ack 1069863898, win 65160, options [mss 1460,sackOK,TS val 2268769424 ecr 1387418,nop,wscale 7], length 0

lewis

BTW it cannot work with https because the server is now using a self signed cert, ACME is taking care of the cert.

stephenw10

That capture shows a simple TCP SYN / SYN-ACK that happens to be on port 443. It's not an actual http or ssl check. How is the backend actually configured?

There's no reason HAProxy can't connect to the backend servers using a self signed cert as long as the CA has been imported into pfSense.

lewis

That capture shows a simple TCP SYN / SYN-ACK that happens to be on port 443. It's not an actual http or ssl check.
How is the backend actually configured?

It only shows up when I enable the HTTP check and it's showing up at 9mn as set so what else could it be?
And more importantly, there isn't anything set up to connect to the web server on port https. Any traffic would come from haproxy.

There's no reason HAProxy can't connect to the backend servers using a self signed cert as long as the CA has been imported into pfSense.

With the config as shown below, the correct acme cert is actually being used but I need the traffic to be sent using http since once this is working, I'll add varnish back into the mix. Varnish isn't working because it's receiving https traffic and the web server as things are now, is not receiving any data from clients.
The only thing not shown is the rule which I've described above. It's a rule, receiving https on the VIP so that haproxy can access.

Backend;

Frontend;

stephenw10

Ok, you can see there that the http health check should perform an actual http options request against the server and see a reply.

The backend servers would not be using an ACME cert if they are selfsigned. The CA cert you need to import is whatever created the selfsigned certs.

You previously said you could not use http to the backend because the website was hardcoded with https links. Has that now been resolved?

lewis

@stephenw10 said in SSL certs handling and HAproxy:

Ok, you can see there that the http health check should perform an actual http options request against the server and see a reply.

Correct but it shows up on the web server as https as shown in the tcpdump.

The backend servers would not be using an ACME cert if they are selfsigned. The CA cert you need to import is whatever
created the selfsigned certs.

According to the remote ssl tests I'm running, it's the correct acme cert on pfsense that is showing up. There are no ssl errors but the web server keeps getting https traffic rather than http from haproxy.

You previously said you could not use http to the backend because the website was hardcoded with https links. Has that now been resolved?

Yes, quite a number of comments back :). The web server is using a self signed cert but the acme cert is the one in play when public connections are coming in.

kiokoman

@lewis
take in mind that if initially you configured the backend as https and after that you modified the settings to use port 80 it does not work. you need to delete the backend and reconfigure it with the correct value.
there are some bugs lurking around

lewis

@kiokoman

First, I've been wondering if I'm coming across a bug again.
Second, I've been considering starting all over again but I've tried so many things that at this point, I think I'll make a mistake in the config.
I will give it a try now however.

lewis

I tore it all down and rebuilt it.
There's definitely a bug alright. Doing that, now I see http heartbeat finally.
The ssl cert checks out as the one in acme on pfsense.

Incoming connections to the web server are now http as they should be but, public connections are now getting;

The page isn’t redirecting properly

I know http works fine because I've tested it from other servers and the web server responds as it should.

I find it a little hard to believe that if this is a bug that it's not well known?

lewis

tcpdump on the web server;

12:32:54.007855 ens18 In  IP _gateway.28554 > r9ty01.phx.loc.http: Flags [S], seq 3274415560, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 511869013 ecr 0], length 0
12:32:54.008023 ens18 Out IP r9ty01.phx.loc.http > _gateway.28554: Flags [S.], seq 266390645, ack 3274415561, win 65160, options [mss 1460,sackOK,TS val 2446701473 ecr 511869013,nop,wscale 7], length 0

The redirection error is odd since it's not happening from any other internal client.

lewis

I checked using curl and a windows machine on the lan, all reach the web server as they should.
Only public connections are getting the 'redirect' error.

 - - [02/Jan/2024:12:55:32 -0700] "GET / HTTP/1.1" 301 4690 197049 196471 "-" "Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0"

lewis

Just for giggles, I changed haproxy to point to varnish and that is now working too.
Meaning, it's sending the traffic to the web server using http.

At this point, all I can think of is that there is something in the headers being sent from haproxy which is causing this constant 301.