SSL certs handling and HAproxy

lewis

Well, as you know, I've been able to get the one web server working but I've not touched the others for load balancing yet.
I quickly tried forwarding the traffic using rules instead of nat the other day but that didn't work. Those web servers already have their own ssl certs and I'd like to tackle that later.

I've been playing with trying to get varnish in the mix and wasting a huge number of hours on this.
Again, nothing I find actually works so learning is difficult since there is nothing working to see yet only confusion.

stephenw10

Hmm, if you're using Varnish I'm not really sure why you would use HAProxy to be honest. Just forward all the traffic to the Varnish proxy and let it handle everything.
Though I haven't used Varnish myself.

lewis

@stephenw10 said in SSL certs handling and HAproxy:

Hmm, if you're using Varnish I'm not really sure why you would use HAProxy to be honest. Just forward all the traffic to the Varnish proxy and let it handle everything.
Though I haven't used Varnish myself.

To clarify, I have three things I'm wanting to accomplish.

1: Fix the what I thought were load balanced servers which are using or should be using haproxy.
2: Wanted to add one single server using acme/haproxy to better understand how that works but also because of the following.
3 I need to send traffic through varnish for caching.

That traffic will be public to pfsense/acme then varnish.
But, in some cases, it could be public to pfsense/acme/haproxy then to varnish.

As I've had to spend so much time on this, and so know this now.
Varnish's load balancing and failover features are less sophisticated and customizable compared to a dedicated lb like HAProxy.
That might be enough in some cases of course.

However, varnish and HAProxy can be used together effectively. HAProxy can distribute incoming traffic across multiple varnish instances.
This combination leverages the strengths of both: load balancing from HAProxy and fast content delivery from varnish's cache.

No matter any of this, other than the server I've gotten working, nothing else has worked so far. No article/documentation seems to cover the complete working process, there's always something missing.

lewis

The main problem I'm having right now is no matter what I do, I see https connections to the varnish server from haproxy.

I'm using a rule for the public ip to haproxy. acme is correctly handling the ssl cert.
In the hap front end, I have the address as the VIP with https offloading.
In the hap backend, I have the server list set to send to the varnish server on port 80.

No matter what I've done so far, I still see https getting to varnish.

stephenw10

Is varnish sending a redirect to https maybe?

But varnish can do load balancing so why not just use it for that and remove HAProxy from the setup?

lewis

@stephenw10
As mentioned, I've been doing a lot of searching and trial and error trying to solve this and what I've learned is that while varnish can do it, haproxy has a lot more functionality.

That said, I'm not sure I need all that functionality but I do like to keep things separated to avoid putting too much load on any one thing.
But it's a moot point because no matter which way I go, it's not working.
If I use a NAT rule to send to varnish or a rule to send to haproxy, the traffic to varnish is always https.
Even when I get rid of haproxy using NAT, the traffic is always https.

I've checked countless times and don't see why.

lewis

I've tried this as well.

I've disabled the haproxy service.
I've created a NAT rule to to accept incoming connections to the VIP on port https forwarded to http/varnish.

When testing the ssl cert from remote, I get;
"Handshake failed, we haven't received any certificates from the requested server."

So it seems acme cannot do this on its own.

I've read this;
pfSense can use the ACME package to issue and renew certificates, but it does not natively handle SSL termination for services behind it, like Varnish, without a load balancer or reverse proxy (like HAProxy). Typically, the SSL termination for incoming traffic before it reaches Varnish is handled by either a load balancer or a web server that supports SSL.

If you want pfSense/ACME to handle the SSL and then pass the traffic to Varnish, you would usually still need an SSL termination point on pfSense. This is not a built-in feature for pfSense and usually requires a package like HAProxy or stunnel.

Since you prefer not to use HAProxy, you might explore other methods such as:

Using stunnel on pfSense for SSL termination.
Setting up an SSL-capable reverse proxy on another server between pfSense and Varnish.
Directly terminating SSL at Varnish using hitch or nginx as a TLS proxy in front of Varnish.

Each of these methods would require installing and configuring additional software either on pfSense or another server in your environment.

lewis

I'm at the point where the main problem I'm facing is that haproxy will not use http to reach the web server.
Initially, I had it pointed to varnish which was getting the traffic but again over https so I simplified the setup to see what's going on.
It seems no matter how I set things up, haproxy always thinks that port 80 is down on the web server which it is not, tested many many times.
Even the hearbeat which is set to http hits the server using https.

I've probably had this config working several times but since the traffic won't get to the server as http, it never works.

I have a rule, no NAT, pointing to the VIP using port 443.
haproxy external IP is set to the VIP, port 443, ssl offloading and that's also the Type.
ACLs, certs, etc, all set correctly on the front end.

On the backend, I have Server list, Forwardto set to Address+Port, LAN IP of the web server, port 80, no ssl, or checks.
I have the correct client cert selected.

Health check is set to HTTP.

Still the connections from haproxy to the web server are https.
This is from tcpdump on the web server;

09:16:12.629794 ens18 In  IP _gateway.40275 > r9ty01.phx.loc.https: Flags [S], seq 3084197971, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1539290509 ecr 0], length 0
09:16:12.629960 ens18 Out IP r9ty01.phx.loc.https > _gateway.40275: Flags [S.], seq 1778053685, ack 3084197972, win 65160, options [mss 1460,sackOK,TS val 2262100095 ecr 1539290509,nop,wscale 7], length 0

Every time I save the haproxy config I get this warning;

NOTICE] (37014) : haproxy version is 2.7.8-58c657f
[NOTICE] (37014) : path to executable is /usr/local/sbin/haproxy
[WARNING] (37014) : config : Server Tyoubackend_ipvANY/typort80 is DOWN, changed from server-state after a reload. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

But it's not true, port 80 is definitely open to all traffic on that web server.

stephenw10

@lewis said in SSL certs handling and HAproxy:

I've created a NAT rule to to accept incoming connections to the VIP on port https forwarded to http/varnish.

I would expect that to fail since the client expects an https connection and the server receives it on the http port.

I thought the problem was that the actual website that is being accessed is hard coded to use https? In which case the backend connection from HAproxy would need to be https. Which should certainly be possible.

I would try to get this working with just HAProxy and one backend server first.

lewis

@stephenw10

That was a previous post, I've moved on since. I was trying all kinds of things, NAT and just rules.
My last comment explains where I'm at now and yes, I'm trying to get this working direct to the web server first to remove complexities.
Varnish was also getting https connections based on the last comment.

stephenw10

Ok so the backend server can now use http? Nothing there is hardcoded to use https?

It should still work with https.

lewis

@stephenw10
So the setup is fairly simple right now.
Using a rule with the vip and it's set to https.
Then haproxy is in play but no varnish. The haproxy should be sending to the web server using http.
On the front end, all described above.
On the back end, all described above.

The really weird thing is that I have the back end up to use HTTP for the health check but as shown above, the connections to the server are https.
Why and how could that even happen? The web server is absolutely responding to http requests as well but none come from haproxy.

11:07:21.958831 ens18 In  IP _gateway.14123 > r9ty01.phx.loc.https: Flags [S], seq 1069863897, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1387418 ecr 0], length 0
11:07:21.959032 ens18 Out IP r9ty01.phx.loc.https > _gateway.14123: Flags [S.], seq 3639861567, ack 1069863898, win 65160, options [mss 1460,sackOK,TS val 2268769424 ecr 1387418,nop,wscale 7], length 0

lewis

BTW it cannot work with https because the server is now using a self signed cert, ACME is taking care of the cert.

stephenw10

That capture shows a simple TCP SYN / SYN-ACK that happens to be on port 443. It's not an actual http or ssl check. How is the backend actually configured?

There's no reason HAProxy can't connect to the backend servers using a self signed cert as long as the CA has been imported into pfSense.

lewis

That capture shows a simple TCP SYN / SYN-ACK that happens to be on port 443. It's not an actual http or ssl check.
How is the backend actually configured?

It only shows up when I enable the HTTP check and it's showing up at 9mn as set so what else could it be?
And more importantly, there isn't anything set up to connect to the web server on port https. Any traffic would come from haproxy.

There's no reason HAProxy can't connect to the backend servers using a self signed cert as long as the CA has been imported into pfSense.

With the config as shown below, the correct acme cert is actually being used but I need the traffic to be sent using http since once this is working, I'll add varnish back into the mix. Varnish isn't working because it's receiving https traffic and the web server as things are now, is not receiving any data from clients.
The only thing not shown is the rule which I've described above. It's a rule, receiving https on the VIP so that haproxy can access.

Backend;

Frontend;

stephenw10

Ok, you can see there that the http health check should perform an actual http options request against the server and see a reply.

The backend servers would not be using an ACME cert if they are selfsigned. The CA cert you need to import is whatever created the selfsigned certs.

You previously said you could not use http to the backend because the website was hardcoded with https links. Has that now been resolved?

lewis

@stephenw10 said in SSL certs handling and HAproxy:

Ok, you can see there that the http health check should perform an actual http options request against the server and see a reply.

Correct but it shows up on the web server as https as shown in the tcpdump.

The backend servers would not be using an ACME cert if they are selfsigned. The CA cert you need to import is whatever
created the selfsigned certs.

According to the remote ssl tests I'm running, it's the correct acme cert on pfsense that is showing up. There are no ssl errors but the web server keeps getting https traffic rather than http from haproxy.

You previously said you could not use http to the backend because the website was hardcoded with https links. Has that now been resolved?

Yes, quite a number of comments back :). The web server is using a self signed cert but the acme cert is the one in play when public connections are coming in.

kiokoman

@lewis
take in mind that if initially you configured the backend as https and after that you modified the settings to use port 80 it does not work. you need to delete the backend and reconfigure it with the correct value.
there are some bugs lurking around

lewis

@kiokoman

First, I've been wondering if I'm coming across a bug again.
Second, I've been considering starting all over again but I've tried so many things that at this point, I think I'll make a mistake in the config.
I will give it a try now however.

lewis

I tore it all down and rebuilt it.
There's definitely a bug alright. Doing that, now I see http heartbeat finally.
The ssl cert checks out as the one in acme on pfsense.

Incoming connections to the web server are now http as they should be but, public connections are now getting;

The page isn’t redirecting properly

I know http works fine because I've tested it from other servers and the web server responds as it should.

I find it a little hard to believe that if this is a bug that it's not well known?