HAProxy not routing multiple internal hosts to one public IP

strongthany · Apr 9, 2020, 3:05 PM

Howdy! I had previously posted about my intent to host multiple publicly available via my one public IP utilizing a reverse proxy to achieve this. I have since moved from squid to HAProxy and enjoying it. However, I am running into an issue where I'm only able to get one of my servers to work, that being my rocket chat server. I used this guide as I've seen others in the past use. Here's what I have for status currently(I have covered the IP's of people using the chat server as well as the domain I'm using to host said services for privacy reasons)

login-to-view

If we go to my HAProxy config page, we can see what I have for front end and back end:

login-to-view

All the front ends are configured as the previously linked tutorial outlines. The following are the back ends:

login-to-view

The following is the config for the working proxy(rocket chat server). The rest of the page is configured, including the transparent ClientIP box to "Use Client-IP to connect to backend servers"

login-to-view

The following is the configuration for the blog server(ghost on ubuntu). I am unable to configure let's encrypt on it as I get an error when I try.

login-to-view

Here's what ghost gives me when I try to get a cert:

admin@blog:/var/www/ghost$ ghost setup ssl
? Enter your email (For SSL Certificate) email@domainname
+ sudo /etc/letsencrypt/acme.sh --issue --home /etc/letsencrypt --domain blog.domainname.tld --webroot /var/www/ghost/system/nginx-root --reloadcmd "nginx -s reload" --accountemail email@domainname
✖ Setting up SSL
One or more errors occurred.

1) ProcessError

Message: Command failed: /bin/sh -c sudo -S -p '#node-sudo-passwd#'  /etc/letsencrypt/acme.sh --issue --home /etc/letsencrypt --domain blog.domain.tld--webroot /var/www/ghost/system/nginx-root --reloadcmd "nginx -s reload" --accountemail email@domainname
[Wed Apr  8 13:25:48 CDT 2020] blog.domain.tld:Verify error:Invalid response from https://blog.domain.tld/.well-known/acme-challenge/lYvoILwkO33QDkZL4NWJ86QheQSKeuwL8Hi9drHHDsk [my.public.ip.address`]: 503
[Wed Apr  8 13:25:48 CDT 2020] Please add '--debug' or '--log' to check more details.
[Wed Apr  8 13:25:48 CDT 2020] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh

[Wed Apr  8 13:25:44 CDT 2020] Single domain='blog.domain.tld'
[Wed Apr  8 13:25:44 CDT 2020] Getting domain auth token for each domain
[Wed Apr  8 13:25:45 CDT 2020] Getting webroot for domain='blog.domain.tld'
[Wed Apr  8 13:25:45 CDT 2020] Verifying: blog.domain.tld

Exit code: 1


Debug Information:
    OS: Ubuntu, v18.04.4 LTS
    Node Version: v10.19.0
    Ghost Version: 3.12.1
    Ghost-CLI Version: 1.13.1
    Environment: production
    Command: 'ghost setup ssl'

Additional log info available in: /home/crabman/.ghost/logs/ghost-cli-debug-2020-04-08T18_25_48_512Z.log

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/api/ghost-cli/ for troubleshooting.
admin@blog:/var/www/ghost$

The other server in the config you saw earlier is for a Nextcloud server, of which I also get the same error when I try to get a certificate.

Firewall has 80 and 443 open, and I can confirm inbound connections work in a capacity as my chat server is publically routable.

Hopefully this was enough information to help you get an idea, but if there's anything else I can provide to help please let me know. Thank you !

PiBa · Apr 9, 2020, 9:29 PM

@strongthany
Is it about the two servers that Haproxy says are down.?. Then first fix that. Change the configured health-check a bit to make it report them as up. Try method:GET, try adding a Host header. Try check on the stats page why server is marked down. Perhaps it requires authentication so a 401 response should be allowed as valid response?

strongthany · Apr 10, 2020, 2:56 PM

@PiBa Thank you for the help, though that doesn't seem to have done it yet. I changed the method to GET, though I'm not sure where/how to change the Host Header. I should have included in my initial message that when I try to browse to those pages I get a 503.

PiBa · Apr 10, 2020, 3:56 PM

@strongthany said in HAProxy not routing multiple internal hosts to one public IP:

where/how to change the Host Header.

You can configure a "Http check version" for the healthcheck and that can include a host-header.. HTTP/1.1\r\nHost:\ www.yoursite.tld

@strongthany said in HAProxy not routing multiple internal hosts to one public IP:

I should have included in my initial message that when I try to browse to those pages I get a 503.

You did ;).. Invalid response ..... : 503 (b.t.w. a 503 is the expected response when all servers in a backend are down..)

Have you checked 'why' the server is marked as down? L4 L6 or L7 issue? And what additional info is there wrong status / timeout? https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_troubleshooting

strongthany · Apr 10, 2020, 4:24 PM

@PiBa Here's two things I notice when I try to get to the server in various ways:

If I browse to it over http via it's IP, I get the Nginx landing page. https fails but that's expected.
If I browse to it through the url, I get redirected to https and it fails.

I'm going to try re-doing the server(nothing's on it yet, not a big deal) because I previously had spun it up while trying to get it to work on squid. I'll assess it from there, but I wonder if the https redirect is messing with the connection?

I will report back later with my findings.

strongthany · Apr 10, 2020, 8:03 PM

@PiBa So I've done some troubleshooting but so far no dice. Would the https redirect be messing with a connection to the nginx service running on the blog server? It doesn't have ssl yet, and when it tries to connect it fails. The following is the error report ghost gives me when I try to get ssl set up:

One or more errors occurred.

1) ProcessError

Message: Command failed: /bin/sh -c sudo -S -p '#node-sudo-passwd#'  /etc/letsencrypt/acme.sh --issue --home /etc/letsencrypt --domain blog.mydomain.tld --webroot /var/www/ghost/system/nginx-root --reloadcmd "nginx -s reload" --accountemail dmarc@mydomain.tld
[Fri Apr 10 19:38:52 UTC 2020] blog.mydomain.tld:Verify error:Invalid response from https://blog.mydomain.tld/.well-known/acme-challenge/-4tlF7sPWL7xaqUAmSfRK7soTvyub5mS-Y2rLCN8qmE [216.126.212.192]: 503
[Fri Apr 10 19:38:52 UTC 2020] Please add '--debug' or '--log' to check more details.
[Fri Apr 10 19:38:52 UTC 2020] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh

[Fri Apr 10 19:38:48 UTC 2020] Create account key ok.
[Fri Apr 10 19:38:48 UTC 2020] Registering account
[Fri Apr 10 19:38:48 UTC 2020] Registered
[Fri Apr 10 19:38:49 UTC 2020] ACCOUNT_THUMBPRINT='rvit2sVFUOdgyrcEu9azhghKQUwYiQbVUtyxoCp4J5Q'
[Fri Apr 10 19:38:49 UTC 2020] Creating domain key
[Fri Apr 10 19:38:49 UTC 2020] The domain key is here: /etc/letsencrypt/blog.mydomain.tld/blog.mydomain.tld.key
[Fri Apr 10 19:38:49 UTC 2020] Single domain='blog.mydomain.tld'
[Fri Apr 10 19:38:49 UTC 2020] Getting domain auth token for each domain
[Fri Apr 10 19:38:49 UTC 2020] Getting webroot for domain='blog.mydomain.tld'
[Fri Apr 10 19:38:49 UTC 2020] Verifying: blog.mydomain.tld

Exit code: 1


Debug Information:
    OS: Ubuntu, v18.04.4 LTS
    Node Version: v10.20.0
    Ghost Version: 3.13.1
    Ghost-CLI Version: 1.13.1
    Environment: production
    Command: 'ghost install'

Additional log info available in: /home/crabman/.ghost/logs/ghost-cli-debug-2020-04-10T19_39_41_113Z.log

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/api/ghost-cli/ for troubleshooting.

PiBa · Apr 11, 2020, 2:38 PM

@strongthany
So to sum it up, what is 'listening' on the wherever your blog.mydomain.tld is pointing to?

http://blog.mydomain.tld
If i understand correctly hapoxy is listening on :80 and you have a redirect to https there (which LetsEncrypt would follow.)
https://blog.mydomain.tld
Haproxy is listening here as well, but has has no way to connect to any 'working' webserver behind it..? As you have likely only configured the backend to connect to a server over :443, which it doesn't listen on yet..?

So how do you expect traffic / domain-validation to get handled?

I think an option would be to configure a self-signed cert on the webserver so at least it will be 'reachable' to reply with the acme token placed in its webroot.. Or perhaps temporarily direct the haproxy to :80 on the webserver.?.

strongthany · Apr 20, 2020, 9:38 PM

@PiBa Good news, I got it to work! I did as you suggested and got a self signed certificate on the server using this guide. After that HAProxy is able to route traffic to the host. It even works with the Let's Encrypt wildcard cert I have through the ACME package, so there's no cert errors getting to the site. Thank you for the help again.