HAProxy backend to docker container not working

[CyBiS]

Hi,

I have bound HAProxy to a localhost VIP and configured a single front-end for SSL/TLS offloading (wildcard certificate via the ACME package) listening on port 443 and multiple backends to various hosts/services on my local network.

I get a proper https connection with a valid Let's Encrypt certificate to my dedicated hosts like the pfsense box itself (listening on port 443) or my NAS (listening on port 5001).

However, I'm getting a 503 error for any of my docker containers/services that are running on my NAS, i.e. the backend pointing to https://NAS_IP:CONTAINER_PORT (encryption on and SSL checks off) and me requesting https://CONTAINER_NAME.mydomain.tld where CONTAINER_NAME is according to the frontend ACL. I can connect to https://NAS_IP:CONTAINER_PORT in the browser just fine (NAS and browser host being in the same subnet, i.e. no traffic via pfsense). Pinging the HAProxy VIP from within one of those containers is working fine as well. So I don't think it's some docker/container misconfiguration.

BR

[paulbr]

@cybis Can you provide screenshots of the HAProxy frontend / backed configuration

In the HAProxy stats page is the backend green ?

[CyBiS]

@paulbr I'm running the community edition 2.5.2 with HAProxy 2.2.14-a07ac36 (haproxy-devel package 0.62_4).

Here's the backend:

And here's the frontend:

The backend is not green:

[paulbr]

@cybis looks like unifi may have an SSL error. If you hit unifi directly from a browser and check the SSL certificate are there errors and is it also self signed ?.

Can you expose unifi via http only in the backed and then try that to rule out any app issue. You will still be secure if it's all internal as SSL will be done at the edge with pfsense

[CyBiS]

@paulbr As mentioned in my first post, the direct connection in the browser via https://192.168.15.5:8443, for example, works fine. The SSL certificate in this case is self-signed by the Unifi container. There is also no possibility to turn off SSL in this case. Furthermore the other two hosts in the example configuration above have both SSL turned on with self-signed certificates (when one connects to them directly). For those hosts the connection via HAProxy works fine, i.e. a valid Let's Encrypt certificate is presented.

However, I tested a grocy container as well which does not have SSL enabled. Here I have the same issue, a 503 error when I go via HAProxy, i.e. via the container's FQDN. And if I hit it directly via the browser, e.g http://192.168.15.5:9283, I get the grocy UI as expected.

The HTTP health check for the grocy container also shows an error:

There shouldn't be any difference between those hosts. The docker containers expose the proper ports and I can connect to them directly. I also tested all the ports for the non-working hosts from the pfsense UI and the HAProxy VIP as source IP. The tests are passing. And the HAProxy VIP can be reached from within the containers ... I don't get it

[CyBiS]

So I deleted all the HAProxy settings, uninstalled the package, rebootet my pfsense box, installed the HAProxy 1.8.30 package (haproxy package 0.61_3) and reconfigured all backends and frontends again ... now none of the backends were working.

I repeated the process with the very same result. For a backend to work I had to enable the health check which shouldn't be necessary I guess.

So I repeated the whole process a third time and for some reason it works now as expected. All backends work without any issues so far. I hope this is still the case when I add further backends in the future

[Lars_LE]

@cybis did you foudn a solution, i have a similar thing...

i'm using qnap docker functionality, in case the docker container have there oqn network and some ports are mounted to the host (qnap nas) i can't reach them from haproxy. with conatiner directly bridged to host interface it works fine, but i need that virtual network in the middle...

any hints ?

[CyBiS]

@lars_le As mentioned in my last post, I switched to the non-devel package and had to reinstall HAProxy a couple of times. I do not know what the issue was unfortnately. But so far I had no further hiccups, the connection to all my hosts (both dedicated and docker hosts) via HAProxy is working fine.

[Lars_LE]

@cybis thanks a lot for responding