HAProxy / Rancher config



  • New to both HAproxy and Rancher. Trying to use haproxy package on pfsense to load balance 3 rancher nodes. General config works fine - I can reach the rancher console without a problem. When I deploy a workload in rancher (such as longhorn), I create a CNAME for the IP I'm load balancing via HAProxy. When I attempt to go to that CNAME I get a 503 Service Unavailable. I'm guessing something is wrong with my HAProxy config and it isn't passing through the right headers. Rancher docs say I need to pass through:

    • Host
    • X-Forwarded-Proto
    • X-Forwarded-Port
    • X-Forwarded-For

    Haproxy docs say they don't strip the Host header out, so I believe I just need to have my backend look like this:

    backend rancher_http_ipvANY
    	mode			http
    	log			global
    	balance			roundrobin
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk GET /healthz
    	option			forwardfor
    	http-request set-header X-Forwarded-Host %[req.hdr(Host)]
    	http-request set-header X-Forwarded-Port %[dst_port]
    	server			swarth-dok-001 10.168.12.247:80 check inter 1000
    	server			swarth-dok-003 10.168.12.248:80 check inter 1000
    	server			swarth-dok-004 10.168.12.245:80 check inter 1000
    
    

    Could certainly be something on the rancher side that I'm doing wrong as well. Would appreciate someone gut checking my haproxy config...



  • Backend 'i think' looks fine, what does the frontend look like though.? Are you sure the 503 is generated by the rancher.? Or could it be haproxy doesnt even try and send the request there.? If it doesn't find a usable backend it would also reply 503.

    p.s. X-Forwarded-Proto is not send with that config.? If its really required that could be the issue.?



  • Trying again - I am getting a No server is available to handle this request. That would fit with your thought that rancher is opting to ignore the request. I am using a custom hostname rather than generating an xip.io name.

    Frontend here:

    frontend rancher
    	bind			10.168.12.20:443 name 10.168.12.20:443 ssl  crt /var/etc/haproxy/rancher.pem
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	http-request set-header X-Forwarded-Host %[req.hdr(Host)]
    	acl			aclcrt_rancher	hdr_reg(host) -i ^rancher\.home\.swth(:([0-9]){1,5})?$
    	use_backend rancher_http_ipvANY  if   aclcrt_rancher
    
    

    I am able to hit the rancher node that has the container running directly on the backend port.
    If I create a test CNAME that points directly to that rancher node, I am able to bring up the page on port 80.

    Could this have something to do with SSL offloading somehow?



  • Noticed the ACL line on the frontend. That looks promising. I tried adding this to the frontend, but it doesn't appear to have helped:

    acl			aclcrt_longhorn	hdr_beg(host) -i longhorn
    


  • Perhaps disable the 'Add ACL for certificate CommonName.' so the backend could be used as a 'default_backend' , does that help? Looking at stats page of haproxy does it register that the webserver is sending 5xx responses.?



  • @jarush
    Just defining a new acl called aclcrt_longhorn wont affect anything if it isn't used for some 'action' like also a use_backend that was checking the "if aclcrt_rancher"



  • Yep, that was it, needed to do the action. Thanks for the help!

    frontend rancher
    	bind			10.168.12.20:443 name 10.168.12.20:443 ssl  crt /var/etc/haproxy/rancher.pem
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	acl			aclcrt_longhorn	hdr_reg(host) -i ^longhorn\.home\.swth(:([0-9]){1,5})?$
    	acl			aclcrt_rancher	hdr_reg(host) -i ^rancher\.home\.swth(:([0-9]){1,5})?$
    	use_backend rancher_http_ipvANY  if  aclcrt_longhorn or aclcrt_rancher aclcrt_rancher
    
    

 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy