HAProxy won't connect to TLS backend

[ma0f97]

Hello,

Currently using HAProxy with SSL Offloading (so unencrypted backend). Now I want to add a backend that is using encryption (HTTPS Nginx), but the connection just wont work. Clients get a 503 Service Unavailable when using the following backend config:

Mode Name Forwardto Address Port Encrypt(SSL) SSL checks Weight Actions
active vaultwarden_nginx Address+Port: 10.20.1.21 443 yes no
(with no health checks)

When using health check "Basic" and disabling Encrypt(SSL) checkbox the client will get a 400 Bad Request from Nginx: "This plain HTTP request was send to HTTPS port". I confirmed using Packet Capture that plain HTTP is send on port 443.

When using the same setting and "HTTP" health check it will give a 503 again.

So my question basically is:
What combination of Encrypt(SSL) and Health Check do I have to choose so that HAProxy speaks actual TLS to the backend without giving me a 503?

[viragomann]

@ma0f97
Try basic health check.
Ensure that the backend is shown as online in the stats before you go on.

[ma0f97]

@viragomann Its green when using basic check so I guess its online. What now?

[viragomann]

@ma0f97
The frontend is configured for SSL offloading with a proper certificate assigned, and the backend with SSL encryption?
This should work straight forward. I have mine set up in the same way.

[ma0f97]

@viragomann Yep now it works. It was the backend all along. I thought that setting up then nginx and vaultwarden in the same docker compose with the same network attached will expose the port 80 of vaultwarden to nginx automatically, but apparently int didn't ;)
I was just confused about the 503, as the direct connection inside the LAN (so without HAProxy) worked fine without a problem so I thought HAProxy was the problem.

[johnpoz]

@ma0f97 glad you got it working, and this question could also be to @viragomann

What is your reasoning to run tls to your backend if your doing offloading on haproxy? I don't see a reason for this unless your backend network needs to be considered hostile - someone sniffing it to get details that they would not see if the connecting was encrypted between the proxy and the backend.

I would assume this is a very rare sort of scenario.

On the con side your now forcing the backend to do more work handing the https. You also have to now configure this backend with the cert, and everything else you might want to do for a good https connection. While If I do that at the front end - its all in one place, I can setup the what ciphers and algos I want to use right at the front end..

[viragomann]

@johnpoz said in HAProxy won't connect to TLS backend:

What is your reasoning to run tls to your backend if your doing offloading on haproxy?

The application, in my case.
It uses a script, which builds the url wrong if the backend is not on https, but the client though. I gave up the troubleshooting after hours and enabled the encryption in the backend.

[ma0f97]

@johnpoz Yeah I did the 'offloading only' approach for quite a while and it works great - actually it's how I do it for most other services I host publicly.

But in this case the backend server is Vaultwarden, an open source implementation of Bitwarden (password manager).
I am currently in the progress of strengthen my security posture and I came to the conclusion to treat every network that has a live connection to the internet under 'assume breach' and evaluate the risk based on that. Under this assumption it is really really important that no one ever sees decrypted traffic to that server (e.g. master vault password of a user etc.) under any circumstance (except of course if vaultwarden itself is compromised).

So for this specific scenario Internet > HAProxy > Vaultwarden the potential higher backend load is more than acceptable when compared to the security gain.