SSL certs handling and HAproxy

lewis

I can't point to port 80 because all of the sites are built using https only.
Any sites I'd like to add would be https.

kiokoman

@lewis
not a big problem, it's just for convenience, create a self signed cert with all the domains you have

lewis

@kiokoman said in SSL certs handling and HAproxy:

@lewis
not a big problem, it's just for convenience, create a self signed cert with all the domains you have

What does that mean, 'with all of the domains'?
Do you mean you have to generate a self signed cert for each domain or there's a way to do one that has all of the names?

And more importantly, will https use the always updates SSL options that LE offers or will visitors end up using what ever the Apache server is configured with?

kiokoman

@lewis
visitors will use and see only the ACME cert

with openssl you can create 1 cert with all domains name

make a copy and edit openssl.cfg

Under the [ CA_default ] section, change the line (uncomment the line) # copy_extensions = copy to copy_extensions = copy

Under the [ v3_ca ] section,

change the line # subjectAltName=email:copy to subjectAltName=@alt_names

add a new line with extendedKeyUsage = serverAuth

Go to the end of the file and add
[ alt_names ]

if the server has multiple address expected to be used for connection, you should add multiple entries under [ alt_names ]. For example,

[ alt_names ]

DNS.1 = app.example.com

DNS.2 = app2.example.com

DNS.3 = app3.example.com

IP.1 = 192.168.1.1

IP.2 = 10.10.10.10

under cmd if you use windows:

set OPENSSL_CONF=C:\OpenSSL-Win64\bin\openssl.cfg (change the directory to where you have your modified copy of openssl.cfg)
openssl genrsa -out ServerKey.key 2048
openssl req -x509 -new -key ServerKey.key -sha256 -days 825 -out myCA.cer

if you need pkcs12

openssl pkcs12 -export -out keystore.p12 -inkey ServerKey.key -in myCA.cer

These are notes I have and they should still work... probably ...

lewis

Getting the visitor IP is simple enough.
In httpd.conf, I changed;

LogFormat "%v %h %l %u %t "%r" %>s %b %D %^FB "%{Referer}i" "%{User-Agent}i"" combined
to
LogFormat "%v %a %l %u %t "%r" %>s %b %D %^FB "%{Referer}i" "%{User-Agent}i"" combined
(%h to %a)

And added this directive;
RemoteIPHeader X-Forwarded-For

In haproxy, Frontend, Advanced, I enabled;

Now I see the actual visitor IP.

lewis

@kiokoman said in SSL certs handling and HAproxy:

visitors will use and see only the ACME cert

Not sure I understand this. One cert can handle one or more virtual hosts on one web server?

The web servers do not have more than one IP, just one but some have lots of virtualhosts, so, different domain names.
And of course, since I have to use port 80/443 for many of those domains, I have a bunch of VIPs on pfsense.

if you need pkcs12

Don't even know what that is unless I look it up so probably not :).

kiokoman

yes, both ACME and your self signed cert can handle more virtual hosts/domain names with only 1 cert, it's one of the reason why people use haproxy on the first place, 1 ip > many domains

lewis

In the case that we've been working on, it's one domain on one web server.
There is another that should be redirected (301) to the main domain in the httpd.conf but I commented that out while working on this.
So maybe that's a good test to try and understand what this new thing (to me) is about.

I definitely don't understand how I can use just one LE cert on pfsense/acme to do this but I kind of understand doing it on the web servers.
On the web servers, I could also just create a self singed cert for every domain that's very long lasting. That I understand.

Getting into editing openssl and other things like that, I don't and could easily lose track of stuff like that. I'm looking for the easiest way that is a no brainer to maintain as I jump from one location to another so have a hard time keeping track of highly nuanced configs.

When you say using one certificate, and let's say we're talking about LE, I've created certs where you use '-d domain1.com -d domain2.com' etc. So I understand that any number of domains can be inserted into one cert. Is this what you mean? I assume so.

stephenw10

Yes the LE cert used by the HAProxy front end has to be valid for all hosts/domains it is accepting traffic for.

lewis

Since I have the single server working, now, moving on to the load balanced servers.

Those web servers are using a san share to get their certificates.
They don't have any self signed cert of course so what would the first step be?

Since they were not going through the haproxy, should I address that first or just do it all at once?

lewis

My original plan was to have pfsense handle the cert encryption so that it could send all of the traffic to a varnish server on the LAN.
I guess that means that no matter what, I would have to remove all ssl from each site/server so that pfsense could handle it then send traffic to varnish using port 80.

stephenw10

No you can still use https between HAProxy and the backend servers if you have to. HAproxy sees all the traffic because the connection from clients are terminated there.

It's just more common to use http to the backends because that traffic doesn't need to be encrypted so it's unnecessary loading on the servers.

lewis

Well, as you know, I've been able to get the one web server working but I've not touched the others for load balancing yet.
I quickly tried forwarding the traffic using rules instead of nat the other day but that didn't work. Those web servers already have their own ssl certs and I'd like to tackle that later.

I've been playing with trying to get varnish in the mix and wasting a huge number of hours on this.
Again, nothing I find actually works so learning is difficult since there is nothing working to see yet only confusion.

stephenw10

Hmm, if you're using Varnish I'm not really sure why you would use HAProxy to be honest. Just forward all the traffic to the Varnish proxy and let it handle everything.
Though I haven't used Varnish myself.

lewis

@stephenw10 said in SSL certs handling and HAproxy:

Hmm, if you're using Varnish I'm not really sure why you would use HAProxy to be honest. Just forward all the traffic to the Varnish proxy and let it handle everything.
Though I haven't used Varnish myself.

To clarify, I have three things I'm wanting to accomplish.

1: Fix the what I thought were load balanced servers which are using or should be using haproxy.
2: Wanted to add one single server using acme/haproxy to better understand how that works but also because of the following.
3 I need to send traffic through varnish for caching.

That traffic will be public to pfsense/acme then varnish.
But, in some cases, it could be public to pfsense/acme/haproxy then to varnish.

As I've had to spend so much time on this, and so know this now.
Varnish's load balancing and failover features are less sophisticated and customizable compared to a dedicated lb like HAProxy.
That might be enough in some cases of course.

However, varnish and HAProxy can be used together effectively. HAProxy can distribute incoming traffic across multiple varnish instances.
This combination leverages the strengths of both: load balancing from HAProxy and fast content delivery from varnish's cache.

No matter any of this, other than the server I've gotten working, nothing else has worked so far. No article/documentation seems to cover the complete working process, there's always something missing.

lewis

The main problem I'm having right now is no matter what I do, I see https connections to the varnish server from haproxy.

I'm using a rule for the public ip to haproxy. acme is correctly handling the ssl cert.
In the hap front end, I have the address as the VIP with https offloading.
In the hap backend, I have the server list set to send to the varnish server on port 80.

No matter what I've done so far, I still see https getting to varnish.

stephenw10

Is varnish sending a redirect to https maybe?

But varnish can do load balancing so why not just use it for that and remove HAProxy from the setup?

lewis

@stephenw10
As mentioned, I've been doing a lot of searching and trial and error trying to solve this and what I've learned is that while varnish can do it, haproxy has a lot more functionality.

That said, I'm not sure I need all that functionality but I do like to keep things separated to avoid putting too much load on any one thing.
But it's a moot point because no matter which way I go, it's not working.
If I use a NAT rule to send to varnish or a rule to send to haproxy, the traffic to varnish is always https.
Even when I get rid of haproxy using NAT, the traffic is always https.

I've checked countless times and don't see why.

lewis

I've tried this as well.

I've disabled the haproxy service.
I've created a NAT rule to to accept incoming connections to the VIP on port https forwarded to http/varnish.

When testing the ssl cert from remote, I get;
"Handshake failed, we haven't received any certificates from the requested server."

So it seems acme cannot do this on its own.

I've read this;
pfSense can use the ACME package to issue and renew certificates, but it does not natively handle SSL termination for services behind it, like Varnish, without a load balancer or reverse proxy (like HAProxy). Typically, the SSL termination for incoming traffic before it reaches Varnish is handled by either a load balancer or a web server that supports SSL.

If you want pfSense/ACME to handle the SSL and then pass the traffic to Varnish, you would usually still need an SSL termination point on pfSense. This is not a built-in feature for pfSense and usually requires a package like HAProxy or stunnel.

Since you prefer not to use HAProxy, you might explore other methods such as:

Using stunnel on pfSense for SSL termination.
Setting up an SSL-capable reverse proxy on another server between pfSense and Varnish.
Directly terminating SSL at Varnish using hitch or nginx as a TLS proxy in front of Varnish.

Each of these methods would require installing and configuring additional software either on pfSense or another server in your environment.

lewis

I'm at the point where the main problem I'm facing is that haproxy will not use http to reach the web server.
Initially, I had it pointed to varnish which was getting the traffic but again over https so I simplified the setup to see what's going on.
It seems no matter how I set things up, haproxy always thinks that port 80 is down on the web server which it is not, tested many many times.
Even the hearbeat which is set to http hits the server using https.

I've probably had this config working several times but since the traffic won't get to the server as http, it never works.

I have a rule, no NAT, pointing to the VIP using port 443.
haproxy external IP is set to the VIP, port 443, ssl offloading and that's also the Type.
ACLs, certs, etc, all set correctly on the front end.

On the backend, I have Server list, Forwardto set to Address+Port, LAN IP of the web server, port 80, no ssl, or checks.
I have the correct client cert selected.

Health check is set to HTTP.

Still the connections from haproxy to the web server are https.
This is from tcpdump on the web server;

09:16:12.629794 ens18 In  IP _gateway.40275 > r9ty01.phx.loc.https: Flags [S], seq 3084197971, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1539290509 ecr 0], length 0
09:16:12.629960 ens18 Out IP r9ty01.phx.loc.https > _gateway.40275: Flags [S.], seq 1778053685, ack 3084197972, win 65160, options [mss 1460,sackOK,TS val 2262100095 ecr 1539290509,nop,wscale 7], length 0

Every time I save the haproxy config I get this warning;

NOTICE] (37014) : haproxy version is 2.7.8-58c657f
[NOTICE] (37014) : path to executable is /usr/local/sbin/haproxy
[WARNING] (37014) : config : Server Tyoubackend_ipvANY/typort80 is DOWN, changed from server-state after a reload. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

But it's not true, port 80 is definitely open to all traffic on that web server.