HAProxy is running, but backend is down in stats and cannot access server

TGill · Apr 8, 2021, 1:33 AM

Re: [HAProxy is running](but backend is down in stats and cannot access server)
Hello @PiBa, I am running into a similar issue and this is what I am seeing on my PfSense UI under stats:

L7STS/404 : HTTP Status Check returned code <404>

Contents of my haproxy.cfg file is as follows:

global
        maxconn                 1000
        stats socket /tmp/haproxy.socket level admin  expose-fd listeners
        gid                     80
        nbproc                  1
        nbthread                        1
        hard-stop-after         15m
        chroot                          /tmp/haproxy_chroot
        daemon
        tune.ssl.default-dh-param       2048
        log-send-hostname               TGpfSense-proxy
        server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
        bind 127.0.0.1:2200 name localstats
        mode http
        stats enable
        stats admin if TRUE
        stats show-legends
        stats uri /haproxy/haproxy_stats.php?haproxystats=1
        timeout client 5000
        timeout connect 5000
        timeout server 5000

frontend shared-frontend-merged
        bind                    1.2.3.4:443 name 1.2.3.4:443   ssl crt-list /var/etc/haproxy/shared-frontend.crt_list
        mode                    http
        log                     global
        option                  http-keep-alive
        option                  forwardfor
        acl https ssl_fc
        http-request set-header         X-Forwarded-Proto http if !https
        http-request set-header         X-Forwarded-Proto https if https
        timeout client          30000
        acl                     aclcrt_shared-frontend  var(txn.txnhost) -m reg -i ^([^\.]*)\.domain\.com(:([0-9]){1,5})?$
        acl                     ACL1    var(txn.txnhost) -m str -i wiki.domain.com
        http-request set-var(txn.txnhost) hdr(host)
        use_backend wiki.domain.com_ipv4  if  ACL1

frontend http-to-https
        bind                    1.2.3.4:80 name 1.2.3.4:80
        mode                    http
        log                     global
        option                  http-keep-alive
        timeout client          30000
        http-request redirect scheme https

backend wiki.domain.com_ipv4
        mode                    http
        id                      10100
        log                     global
        timeout connect         30000
        timeout server          30000
        retries                 3
        source ipv4@ usesrc clientip
        option                  httpchk OPTIONS /
        server                  wikijs 10.10.10.30:3000 id 10101 check inter 1000

haproxy -vv result:

HA-Proxy version 1.8.27-493ce0b 2020/11/06
Copyright 2000-2020 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = freebsd
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Kindly advise how to address this issue. I would like to avoid having the need to run another system for this purpose when such a feature is available in PFSense.

Note: Public IP and domain names have been modified in the .cfg file for this post.

PiBa · Apr 13, 2021, 7:23 PM

@tgill
The webserver replies with a '404' response.. why?

Do you need to supply a 'Host: www' header perhaps with the request.? The healthcheck backend configuration page 'hints' at how that can be done.

TGill · Apr 15, 2021, 3:16 AM

@piba
Thank you for checking out this post. I was able to resolve the issue by adding http-check expect status 404 under advanced settings. TBH, I didn't quite understand how this line would make a difference because while accessing the site through standard NAT rules, I didn't see any 404 errors. Unless the page requested was sending a 404 response prior to redirecting to a valid page. Please feel free to shed some knowledge if you have deeper explanation for this.

Thank you once again for your time.

PiBa · Apr 17, 2021, 6:21 AM

@tgill
Well i cant be sure, and using "http-check expect status 404" can be a decent workaround, though i don't think it is required here. (for a '401 authentication required' i think its acceptable ;) as haproxy doesn't know credentials to login..)
For the 404 i expect that the webserver 'needs' a Host header. Do you know if the webserver is configured to use 'virtual-hosts'.?. If you visit the webserver locally by its IP does it respond with the proper page? What happens if you use curl to request the page.?

TGill · Apr 17, 2021, 6:21 AM

@piba

Do you know if the webserver is configured to use 'virtual-hosts'.?

I believe the built in NGINX server may be the cause of that

 If you visit the webserver locally by its IP does it respond with the proper page?

Yes I do get routed to the correct page and when I ran the cURL command I did see it was redirecting to the sign-in page but with 302 status code. Please see the command output below.

curl 10.10.10.30:10080

<html><body>You are being <a href="http://10.10.10.30:10080/users/sign_in">redirected</a>.</body></html>

curl -I 10.10.10.30:10080

HTTP/1.1 302 Found
Server: nginx
Date: Sat, 17 Apr 2021 05:57:58 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: no-cache
Location: http://10.10.10.30:10080/users/sign_in
Set-Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqWmhPVFV5WmpKbUxXRTNPVFV0TkdVNVppMWhZMlpqTFRSalpqRmhaV1JqTmpaak1TST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--1e52d67fefb6115c51916e512ee89e72134ea6ab; path=/; expires=Wed, 17 Apr 2041 05:57:58 GMT; HttpOnly
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: DENY
X-Gitlab-Feature-Category: projects
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 01F3F6GGKSQNK1DQZBJV8FJZNA
X-Runtime: 0.090303
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000
Referrer-Policy: strict-origin-when-cross-origin

I realize that my setup maybe not be an ideal one but I think this workaround will have to do since this is a home-lab environment of Gitlab running on a docker container. I had just wanted to get some understanding of what might have been happening in the background. I really appreciate your time @PiBa.

PiBa · Apr 21, 2021, 9:55 PM

@tgill
Okay, a 302 is not a 404 though, so curl's request must be different from where haproxy's healthcheck ends up.. Perhaps just change the check method from OPTIONS to GET and see if that makes a difference.? Curl would be using GET by default.

Heracles31 · Apr 24, 2021, 4:05 AM

@TGill,

HAProxy is testing over HTTP/1.0 while your curl is using HTTP/1.1. That may very well be the difference between the two tests and the two results.

You can try something like
HTTP/1.1\r\nHost:\ hostname.domain.lan

in the "Http check version" box in your backend's configuration.