HAProxy / Rancher config
-
New to both HAproxy and Rancher. Trying to use haproxy package on pfsense to load balance 3 rancher nodes. General config works fine - I can reach the rancher console without a problem. When I deploy a workload in rancher (such as longhorn), I create a CNAME for the IP I'm load balancing via HAProxy. When I attempt to go to that CNAME I get a 503 Service Unavailable. I'm guessing something is wrong with my HAProxy config and it isn't passing through the right headers. Rancher docs say I need to pass through:
- Host
- X-Forwarded-Proto
- X-Forwarded-Port
- X-Forwarded-For
Haproxy docs say they don't strip the Host header out, so I believe I just need to have my backend look like this:
backend rancher_http_ipvANY mode http log global balance roundrobin timeout connect 30000 timeout server 30000 retries 3 option httpchk GET /healthz option forwardfor http-request set-header X-Forwarded-Host %[req.hdr(Host)] http-request set-header X-Forwarded-Port %[dst_port] server swarth-dok-001 10.168.12.247:80 check inter 1000 server swarth-dok-003 10.168.12.248:80 check inter 1000 server swarth-dok-004 10.168.12.245:80 check inter 1000Could certainly be something on the rancher side that I'm doing wrong as well. Would appreciate someone gut checking my haproxy config...
-
Backend 'i think' looks fine, what does the frontend look like though.? Are you sure the 503 is generated by the rancher.? Or could it be haproxy doesnt even try and send the request there.? If it doesn't find a usable backend it would also reply 503.
p.s. X-Forwarded-Proto is not send with that config.? If its really required that could be the issue.?
-
Trying again - I am getting a No server is available to handle this request. That would fit with your thought that rancher is opting to ignore the request. I am using a custom hostname rather than generating an xip.io name.
Frontend here:
frontend rancher bind 10.168.12.20:443 name 10.168.12.20:443 ssl crt /var/etc/haproxy/rancher.pem mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 http-request set-header X-Forwarded-Host %[req.hdr(Host)] acl aclcrt_rancher hdr_reg(host) -i ^rancher\.home\.swth(:([0-9]){1,5})?$ use_backend rancher_http_ipvANY if aclcrt_rancherI am able to hit the rancher node that has the container running directly on the backend port.
If I create a test CNAME that points directly to that rancher node, I am able to bring up the page on port 80.Could this have something to do with SSL offloading somehow?
-
Noticed the ACL line on the frontend. That looks promising. I tried adding this to the frontend, but it doesn't appear to have helped:
acl aclcrt_longhorn hdr_beg(host) -i longhorn -
Perhaps disable the 'Add ACL for certificate CommonName.' so the backend could be used as a 'default_backend' , does that help? Looking at stats page of haproxy does it register that the webserver is sending 5xx responses.?
-
@jarush
Just defining a new acl called aclcrt_longhorn wont affect anything if it isn't used for some 'action' like also a use_backend that was checking the "if aclcrt_rancher" -
Yep, that was it, needed to do the action. Thanks for the help!
frontend rancher bind 10.168.12.20:443 name 10.168.12.20:443 ssl crt /var/etc/haproxy/rancher.pem mode http log global option http-keep-alive option forwardfor acl https ssl_fc http-request set-header X-Forwarded-Proto http if !https http-request set-header X-Forwarded-Proto https if https timeout client 30000 acl aclcrt_longhorn hdr_reg(host) -i ^longhorn\.home\.swth(:([0-9]){1,5})?$ acl aclcrt_rancher hdr_reg(host) -i ^rancher\.home\.swth(:([0-9]){1,5})?$ use_backend rancher_http_ipvANY if aclcrt_longhorn or aclcrt_rancher aclcrt_rancher