SSL certs handling and HAproxy

[lewis]

Holy crap, I finally got it working.
I don't even know where to start to show what I did because I tried so many things.
The last thing I did was in the backend, changing the server list Table from port 80 to port 443 and enabling Encrypt(SSL).

The main problem I have now is still not knowing what the correct configuration of the web server should be.
It's using a self signed cert now but this is a stand alone web server. Some of them have lots of virtualhosts.

And, what SSL options will be used? The ones on ACME/haproxy or the ones set in ssl.conf/httpd.conf on the web server?

Plus, now I'd like to fix the 'load balanced' server that never actually worked.

And, how can I forward the visiting IPs? I only see the firewall's IP in the logs which is not useful.
I found this but it doesn't send the visiting IP;

[kiokoman]

@lewis
if you can, don't use cert on web server just point to port 80
people will connect to haproxy and use the cert from ACME/haproxy but haproxy will speak to the web server with the self signed cert

on the backend add all the server that will do balance under "server list", set a Weight (10 / 20 /30 etc etc) and select round robin under load balancing

backend but carefully read the warning

[lewis]

I can't point to port 80 because all of the sites are built using https only.
Any sites I'd like to add would be https.

[kiokoman]

@lewis
not a big problem, it's just for convenience, create a self signed cert with all the domains you have

[lewis]

@kiokoman said in SSL certs handling and HAproxy:

@lewis
not a big problem, it's just for convenience, create a self signed cert with all the domains you have

What does that mean, 'with all of the domains'?
Do you mean you have to generate a self signed cert for each domain or there's a way to do one that has all of the names?

And more importantly, will https use the always updates SSL options that LE offers or will visitors end up using what ever the Apache server is configured with?

[kiokoman]

@lewis
visitors will use and see only the ACME cert

with openssl you can create 1 cert with all domains name

make a copy and edit openssl.cfg

Under the [ CA_default ] section, change the line (uncomment the line) # copy_extensions = copy to copy_extensions = copy

Under the [ v3_ca ] section,

change the line # subjectAltName=email:copy to subjectAltName=@alt_names

add a new line with extendedKeyUsage = serverAuth

Go to the end of the file and add
[ alt_names ]

if the server has multiple address expected to be used for connection, you should add multiple entries under [ alt_names ]. For example,

[ alt_names ]

DNS.1 = app.example.com

DNS.2 = app2.example.com

DNS.3 = app3.example.com

IP.1 = 192.168.1.1

IP.2 = 10.10.10.10

under cmd if you use windows:

set OPENSSL_CONF=C:\OpenSSL-Win64\bin\openssl.cfg (change the directory to where you have your modified copy of openssl.cfg)
openssl genrsa -out ServerKey.key 2048
openssl req -x509 -new -key ServerKey.key -sha256 -days 825 -out myCA.cer

if you need pkcs12

openssl pkcs12 -export -out keystore.p12 -inkey ServerKey.key -in myCA.cer

These are notes I have and they should still work... probably ...

[lewis]

Getting the visitor IP is simple enough.
In httpd.conf, I changed;

LogFormat "%v %h %l %u %t "%r" %>s %b %D %^FB "%{Referer}i" "%{User-Agent}i"" combined
to
LogFormat "%v %a %l %u %t "%r" %>s %b %D %^FB "%{Referer}i" "%{User-Agent}i"" combined
(%h to %a)

And added this directive;
RemoteIPHeader X-Forwarded-For

In haproxy, Frontend, Advanced, I enabled;

Now I see the actual visitor IP.

[lewis]

@kiokoman said in SSL certs handling and HAproxy:

visitors will use and see only the ACME cert

Not sure I understand this. One cert can handle one or more virtual hosts on one web server?

The web servers do not have more than one IP, just one but some have lots of virtualhosts, so, different domain names.
And of course, since I have to use port 80/443 for many of those domains, I have a bunch of VIPs on pfsense.

if you need pkcs12

Don't even know what that is unless I look it up so probably not :).

[kiokoman]

yes, both ACME and your self signed cert can handle more virtual hosts/domain names with only 1 cert, it's one of the reason why people use haproxy on the first place, 1 ip > many domains

[lewis]

In the case that we've been working on, it's one domain on one web server.
There is another that should be redirected (301) to the main domain in the httpd.conf but I commented that out while working on this.
So maybe that's a good test to try and understand what this new thing (to me) is about.

I definitely don't understand how I can use just one LE cert on pfsense/acme to do this but I kind of understand doing it on the web servers.
On the web servers, I could also just create a self singed cert for every domain that's very long lasting. That I understand.

Getting into editing openssl and other things like that, I don't and could easily lose track of stuff like that. I'm looking for the easiest way that is a no brainer to maintain as I jump from one location to another so have a hard time keeping track of highly nuanced configs.

When you say using one certificate, and let's say we're talking about LE, I've created certs where you use '-d domain1.com -d domain2.com' etc. So I understand that any number of domains can be inserted into one cert. Is this what you mean? I assume so.

[stephenw10]

Yes the LE cert used by the HAProxy front end has to be valid for all hosts/domains it is accepting traffic for.

[lewis]

Since I have the single server working, now, moving on to the load balanced servers.

Those web servers are using a san share to get their certificates.
They don't have any self signed cert of course so what would the first step be?

Since they were not going through the haproxy, should I address that first or just do it all at once?

[lewis]

My original plan was to have pfsense handle the cert encryption so that it could send all of the traffic to a varnish server on the LAN.
I guess that means that no matter what, I would have to remove all ssl from each site/server so that pfsense could handle it then send traffic to varnish using port 80.

[stephenw10]

No you can still use https between HAProxy and the backend servers if you have to. HAproxy sees all the traffic because the connection from clients are terminated there.

It's just more common to use http to the backends because that traffic doesn't need to be encrypted so it's unnecessary loading on the servers.

[lewis]

Well, as you know, I've been able to get the one web server working but I've not touched the others for load balancing yet.
I quickly tried forwarding the traffic using rules instead of nat the other day but that didn't work. Those web servers already have their own ssl certs and I'd like to tackle that later.

I've been playing with trying to get varnish in the mix and wasting a huge number of hours on this.
Again, nothing I find actually works so learning is difficult since there is nothing working to see yet only confusion.

[stephenw10]

Hmm, if you're using Varnish I'm not really sure why you would use HAProxy to be honest. Just forward all the traffic to the Varnish proxy and let it handle everything.
Though I haven't used Varnish myself.

[lewis]

@stephenw10 said in SSL certs handling and HAproxy:

Hmm, if you're using Varnish I'm not really sure why you would use HAProxy to be honest. Just forward all the traffic to the Varnish proxy and let it handle everything.
Though I haven't used Varnish myself.

To clarify, I have three things I'm wanting to accomplish.

1: Fix the what I thought were load balanced servers which are using or should be using haproxy.
2: Wanted to add one single server using acme/haproxy to better understand how that works but also because of the following.
3 I need to send traffic through varnish for caching.

That traffic will be public to pfsense/acme then varnish.
But, in some cases, it could be public to pfsense/acme/haproxy then to varnish.

As I've had to spend so much time on this, and so know this now.
Varnish's load balancing and failover features are less sophisticated and customizable compared to a dedicated lb like HAProxy.
That might be enough in some cases of course.

However, varnish and HAProxy can be used together effectively. HAProxy can distribute incoming traffic across multiple varnish instances.
This combination leverages the strengths of both: load balancing from HAProxy and fast content delivery from varnish's cache.

No matter any of this, other than the server I've gotten working, nothing else has worked so far. No article/documentation seems to cover the complete working process, there's always something missing.

[lewis]

The main problem I'm having right now is no matter what I do, I see https connections to the varnish server from haproxy.

I'm using a rule for the public ip to haproxy. acme is correctly handling the ssl cert.
In the hap front end, I have the address as the VIP with https offloading.
In the hap backend, I have the server list set to send to the varnish server on port 80.

No matter what I've done so far, I still see https getting to varnish.

[stephenw10]

Is varnish sending a redirect to https maybe?

But varnish can do load balancing so why not just use it for that and remove HAProxy from the setup?

[lewis]

@stephenw10
As mentioned, I've been doing a lot of searching and trial and error trying to solve this and what I've learned is that while varnish can do it, haproxy has a lot more functionality.

That said, I'm not sure I need all that functionality but I do like to keep things separated to avoid putting too much load on any one thing.
But it's a moot point because no matter which way I go, it's not working.
If I use a NAT rule to send to varnish or a rule to send to haproxy, the traffic to varnish is always https.
Even when I get rid of haproxy using NAT, the traffic is always https.

I've checked countless times and don't see why.