ACME/HAproxy settings for Domoticz client not working

[gschmidt]

Hi,

I have a problem to open Domoticz (domoticz webbased application) secure and encrypted outside the WAN with ACME/HAproxy. It gets a connection error.

First additional info:
I have Domoticz running for 3 years now on an RPi (OS: Raspbian Buster). By default it can be accessed with HTTP (8080) and HTTPS (443), but I only access it outside the WAN with HTTPS to control my domotica devices. I also have a paid service for Domoticz which also needs WAN access.

It is already encrypted for 3 years on the client side with letsencrypt. Letsencrypt-auto and certbot-auto are used to create and renew the certificate for my domain: xxxxxx.duckdns.org
In the Domoticz folder a PEM file is located.

What is my goal
My goal is to create a secured encrypted connection and handle the ssl offloading for Domoticz only at the frontend of HAproxy. If this is working, the certificate at the Client side can be removed (to ensure the encrypted connection during configuration).

What steps did I perform

1. I have created a letsencrypt certificate with ACME for xxxxxx.duckdns.org
   According to the log, the certificate is successfully issued/renewed (I double checked this with renewing the certificate, below is the last row of the log)

update cert![Wed Mar 10 21:26:01 CET 2021] Reload success

This certificate has a different CA and Key values then the PEM on the client side

2. Configured the HAproxy settings page:
   Maximum connections 1000 (pfSense runs on a mini-pc with i3Intel chip with 4 cores and 4gb internal memory)
   Internal Stats Port: 2200
   SSL DH Parameter is 2048

3. Configured Backend settings are quite default:
   
   All "advanced server settings" fields are left blank
   Health check set to None
   Rest of the fields are the default settings

4. Configured Frontend settings
   
   
   The rest of the fields are left default

Result:
In order to check if this works I needed to remove the PEM file in the Domoticz directory of the Rpi.

After I restarted Domoticz, I get the following error (in DuckDuckGo browser of my Phone not connected to the home network):
net::ERR_CONNECTION_REFUSED

I can see in the Stats page that the Frontend gets requests.

Note: when I move back the PEM file to the original location the encryption works normal again

What I am I doing wrong?

[PiBa]

@gschmidt
The Stats frontend gets 8 requests, the domoticz frontend got 0 in the screenshot above.. So that would mean haproxy didnt get any requests on :443 sofar.. Might be firewall rules or a old NAT rule diverting traffic elsewhere..

Also if the server is listening on :443 that probably uses a certificate, and as such haproxy must also re-encrypt the offloaded frontend connection before passing it further to that server. Or perhaps configure haproxy to connect to that 8080 port.? As for healthchecks they are nice to see if the server turns 'green' according to haproxy.. And if not if you have a possible L4 L6 or L7 issue there.. Though yes it can also hinder things if the request haproxy makes for the check isnt using the domain / method / sni the webserver expects. That can usually be added in the config though..

[gschmidt]

@piba

So several possibilities which may cause HAproxy not working properly. You did not mention anything about the settings/backend/frontend configuration I shared. Does this mean that my configuration seems to be Ok?

If it appears to be, let’s start with the first possible cause you mentioned...NAT rules.
In my old setup I obviously had to port forward 443 to the domoticz client. I have disabled this rule after I created the 80 and 443 rules for HAproxy.

Can disabled rules cause this issue?

[gschmidt]

@piba

I think I know where it goes wrong! but not solved yet

3 years ago, In my old (encrypted) situation I port fowarded 443 from WAN to the Domoticz host (I also port forwarded the HTTP port 9090 to 80 for renewal of the domoticz certificate, which I only enable for renewal )

Note: default domoticz http port 8080 I mentioned earlier, I also changed (3 years ago) to 9090

in Firewall/Rules/WAN these port forwards were added to the Rules

When I yesterday disabled this port forward 443 in Firewall/Rules/WAN, I didn't realised that in Firewall/NAT/Port Forward this still was enabled!
After disabeling this in Firewall/NAT/Port Forward, the Rule was also disabled.

When I look now at the statspage I see some action:

However when I open my xxxxx.duckdns.org from outside WAN I get the following error: 502 Bad Gateway, the server returned an invalid or incomplete response.

What may cause this error?

When I followed the tutorial I had to create 2 firewall rules:
create firewall rule
Now create two firewall rules (Firewall / Rules /WAN). It is open TCP-ports 80 and 443 through WAN interface for opening our HAProxy to the external world.

Is this correct?

[Update]

If I set in the Backend health check to HTTP, I get this statspage:

And when I open the duckdns page it returns this Error:
503 No server is available to handle this request

[PiBa]

@gschmidt said in ACME/HAproxy settings for Domoticz client not working:

You did not mention anything about the settings/backend/frontend configuration I shared. Does this mean that my configuration seems to be Ok?

I did mention the server configuration which i think is wrong (in the backend config page). Either you should check the "Encrypt(SSL)" checkbox or change the 443 port there to 9090.

[gschmidt]

@piba

I changed 443 to 9090...I and now I got logged in!

Why is 443 not working then?

[update]
check the "Encrypt(SSL)" with 443 is alo working by the way

[PiBa]

@gschmidt said in ACME/HAproxy settings for Domoticz client not working:

Why is 443 not working then?

Haproxy 'offloads'/decrypts the https traffic on the frontend so it is 'plain http' when passing through haproxy's rules/memory.

Then it sends the traffic to the backend and to the configured server on the configured port. If you configure it to send to :443 it will happily do that, but it will send it as 'plain http' to the webserver. Which expects 'https' traffic on that port so denies the plain request..

So to make sure haproxy sends 'https' to the server you must check the 'encrypt(ssl)' checkbox on the backend/server config.

[gschmidt]

@PiBa

I get it now!

I already figured out that if I delete the (old) PEM file in the domoticz folder,
I can't use setting: check the "Encrypt(SSL)" with 443

So the 9090 setting is exactly what I had in mind
Now I can delete the PEM file on the domoticz host and ACME/HAproxy is dealing with the secured encryption of the Domoticz host

Thanx for the help man!

[gschmidt]

@piba

I have a question about the 503 error page.

If somebody is accessing my WAN IP adress (e.g. https://67.46.29.83:443) instead of my domain name, HAproxy shows a 503 error page, Is this normal behaviour of HAproxy?

If so, this is nice because I want to block access to WAN ip, but is it also possible to modify the header and content of the 503 page?