HAProxy config fails to operate properly after 2.4.4 upgrade

[Leon Straathof]

My working HAProxy config fails to operate properly after 2.4.4 upgrade, i also switched to the HAproxy development branch to see if that would fix it again but it did not, upgrade of pfSense to 2.4.4_1 does not fix it either.
I also did a fresh install but that also did not fix the problem.

Chrome says ERR_CONNECTION_CLOSED when trying to connect trough HAproxy and IE11 says "Can’t connect securely to this page".

My configuration of HAproxy is similar to the config described here: http://loredo.me/post/116633549315/geeking-out-with-haproxy-on-pfsense-the-ultimate

[PiBa]

@leon-straathof
Can you please share the actual haproxy.cfg you have. (bottom of settings tab) (perhaps obfuscate the IP's and domains a little..)

[Leon Straathof]

Sure it is quite big, and after some more testing i found that only the HTTPS parts are not working any more after the upgrade. The OpenVPN, RDP ,SSH and HTTP redirector are all stil working. I changed all ip adresses and names to fictional information. And i cut out large repetative parts (If one part is fixed i can apply the same on everything). And don't judge me on the silly RDP implementation i use it to be able to rdp into all my machines (i don't know how to select machines in tcp on subdomain so i select the machine on unique username per machine) and hey it works for me so i am happy. So the only open port is 443 for everything i need.

Automaticaly generated, dont edit manually.

Generated on: 2018-12-07 16:46

global
maxconn 100
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
server-state-file /tmp/haproxy_server_state
# Modern browser compatibility only as mentioned here:
# https://wiki.mozilla.org/Security/Server_Side_TLS
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
tune.ssl.default-dh-param 2048

# Time-to-first-Byte (TTFB) value needs to be optimized based on
# the actual public certificate chain see
# https://www.igvita.com/2013/10/24
# /optimizing-tls-record-size-and-buffering-latency/
tune.ssl.maxrecord 1370

listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend WAN_443-merged
bind 192.168.1.10:443 name 192.168.1.10:443
mode tcp
log global
timeout client 7200000
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 } or !{ req.ssl_hello_type 1 }
acl ovpn1 req.len 0
acl ovpn2 req.ssl_hello_type 1
acl ovpn3 req.rdp_cookie_cnt 0
acl https1 req.ssl_hello_type 1
acl https2 req.ssl_sni -m end -i vpn.acme.com
acl https3 req.ssl_sni -m end -i ssh.acme.com
acl vpn1 req.ssl_hello_type 1
acl vpn2 req.ssl_sni -m end -i .vpn.acme.com
acl ssh1 req.ssl_hello_type 1
acl ssh2 req.ssl_sni -m end -i .ssh.acme.com
acl rdp0 req.rdp_cookie_cnt 0
acl rdp1 rdp_cookie(mstshash) -m sub -i deskuser
acl rdp2 rdp_cookie(mstshash) -m sub -i servuser
use_backend OpenVPN_ipvANY if !ovpn1 !ovpn2 ovpn3
use_backend WAN_HTTPS_ipvANY if https1 !https2 !https3
use_backend WAN_HTTPS_auth_ipvANY if vpn1 vpn2
use_backend WAN_SSLH_ipvANY if ssh1 ssh2
use_backend RDP_w10_1_ipvANY if !rdp0 rdp1
use_backend RDP_w2019s1_ipvANY if !rdp0 rdp2

frontend WAN_HTTPS-merged
bind 127.0.0.1:2043 name 127.0.0.1:2043 no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS.crt_list accept-proxy npn http/1.1
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 7200000
errorfile 503 /var/etc/haproxy/errorfile_WAN_HTTPS_503_ExampleErrorfile
# Remove headers that expose security-sensitive information.
rspidel ^Server:.$
rspidel ^X-Powered-By:.$
rspidel ^X-AspNet-Version:.*$
rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains
acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
acl guacamole var(txn.txnhost) -m str -i guacamole.acme.com
acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
acl nuget var(txn.txnhost) -m str -i nuget.acme.com
acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
acl sinusbot var(txn.txnhost) -m str -i sinusbot.acme.com
acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend Guacamole_ipvANY if guacamole aclcrt_WAN_HTTPS_guacamole
use_backend Nuget_ipvANY if nuget aclcrt_WAN_HTTPS_nuget
use_backend SinusBot_ipvANY if sinusbot aclcrt_WAN_HTTPS_sinusbot

frontend WAN_HTTPS_auth-merged
bind 127.0.0.1:2044 name 127.0.0.1:2044 no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS_auth.crt_list ca-file /var/etc/haproxy/clientca_WAN_HTTPS_auth.pem verify required crt-ignore-err all accept-proxy npn http/1.1
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 7200000
# Remove headers that expose security-sensitive information.
rspidel ^Server:.$
rspidel ^X-Powered-By:.$
rspidel ^X-AspNet-Version:.*$
acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
acl pfsense var(txn.txnhost) -m str -i pfsense.vpn.acme.com
acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
acl R7800 var(txn.txnhost) -m str -i r7800.vpn.acme.com
acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
acl modem var(txn.txnhost) -m str -i modem.vpn.acme.com
acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend pfSense_ipvANY if pfsense aclcrt_WAN_HTTPS_AUTH_pfsense
use_backend R7800_ipvANY if R7800 aclcrt_WAN_HTTPS_AUTH_dd-wrt
use_backend modem_ipvANY if modem aclcrt_WAN_HTTPS_AUTH_modem

frontend WAN_SSLH-merged
bind 127.0.0.1:2022 name 127.0.0.1:2022 no-sslv3 ssl crt-list /var/etc/haproxy/WAN_SSLH.crt_list ca-file /var/etc/haproxy/clientca_WAN_SSLH.pem verify required accept-proxy npn ssh/2.0
mode tcp
log global
timeout client 7200000
acl ssh_pfsense1 ssl_fc_npn -i ssh/2.0
acl ssh_pfsense2 ssl_fc_sni_reg pfsense.ssh.acme.com
acl ssh_guacamole1 ssl_fc_npn -i ssh/2.0
acl ssh_guacamole2 ssl_fc_sni_reg guacamole.ssh.acme.com
acl ssh_w2016s1_1 ssl_fc_npn -i ssh/2.0
acl ssh_w2016s1_2 ssl_fc_sni_reg guacamole.ssh.acme.com
use_backend ssh_pfsense_ipvANY if ssh_pfsense1 ssh_pfsense2
use_backend ssh_guacamole_ipvANY if ssh_guacamole1 ssh_guacamole2
use_backend ssh_w2016s1_ipvANY if ssh_w2016s1_1 ssh_w2016s1_2

frontend WAN_HTTP
bind 192.168.1.10:80 name 192.168.1.10:80
mode http
log global
option http-keep-alive
timeout client 30000
acl acme-challenge var(txn.txnpath) -m beg -i /.well-known/acme-challenge/
http-request set-var(txn.txnpath) path
use_backend pfSense_ipvANY if acme-challenge
default_backend ssl-redirect_ipvANY

backend OpenVPN_ipvANY
mode tcp
id 118
log global
timeout connect 30000
timeout server 7200000
retries 2
server OpenVPN 127.0.0.1:1194 id 119 weight 1

backend WAN_HTTPS_ipvANY
mode tcp
id 130
log global
timeout connect 30000
timeout server 7200000
retries 3
server WAN_HHTPS 127.0.0.1:2043 id 131 ssl verify none send-proxy

backend WAN_HTTPS_auth_ipvANY
mode tcp
id 132
log global
timeout connect 30000
timeout server 7200000
retries 3
server WAN_HTTPS_auth 127.0.0.1:2044 id 133 ssl verify none send-proxy

backend WAN_SSLH_ipvANY
mode tcp
id 134
log global
timeout connect 30000
timeout server 7200000
retries 3
server WAN_SSLH 127.0.0.1:2022 id 135 ssl verify none send-proxy

backend RDP_w10_1_ipvANY
mode tcp
id 102
log global
timeout connect 30000
timeout server 30000
retries 3
server w10_1 192.168.2.32:3389 id 103 check inter 1000

backend RDP_w2019s1_ipvANY
mode tcp
id 148
log global
timeout connect 30000
timeout server 30000
retries 3
server w2019s1 192.168.5.19:3389 id 103 check inter 1000

backend Guacamole_ipvANY
mode http
id 110
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server Guacamole 192.168.2.18:8080 id 111 check inter 1000 weight 1

backend Nuget_ipvANY
mode http
id 116
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server Nuget 192.168.5.19:80 id 117 check inter 1000 weight 1

backend SinusBot_ipvANY
mode http
id 122
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk GET /
server SinusBot 192.168.5.19:8087 id 123 check inter 1000 weight 1

backend pfSense_ipvANY
mode http
id 120
log global
timeout connect 30000
timeout server 30000
retries 3
server pfSense 127.0.0.1:80 id 121 weight 1

backend R7800_ipvANY
mode http
id 104
log global
timeout connect 30000
timeout server 30000
retries 3
server R7800 192.168.2.2:80 id 105 check inter 1000 weight 1

backend modem_ipvANY
mode http
id 136
log global
timeout connect 30000
timeout server 30000
retries 3
server modem 192.168.1.1:80 id 137 check inter 5000 weight 1

backend ssh_pfsense_ipvANY
mode tcp
id 126
log global
timeout connect 3000
timeout server 7200000
retries 2
server ssh_pfsense 192.168.2.3:22 id 127

backend ssh_guacamole_ipvANY
mode tcp
id 124
log global
timeout connect 3000
timeout server 7200000
retries 2
server ssh_guacamole 192.168.2.18:22 id 125 check inter 1000

backend ssh_w2016s1_ipvANY
mode tcp
id 140
log global
timeout connect 3000
timeout server 7200000
retries 2
server ssh_w2016s1 192.168.5.16:22 id 141 check inter 1000

backend ssl-redirect_ipvANY
mode http
id 128
log global
timeout connect 30000
timeout server 30000
retries 3
redirect scheme https code 301

[PiBa]

@leon-straathof
Can you try and disable the 'SSL Encryption' checkbox in the "WAN_HTTPS" backend.?

(Also for the WAN_HTTPS_auth and WAN_SSLH backend servers)

[Leon Straathof]

PiBa, that worked. Can you explain why because i am stumped that a internal only part cannot be encrypted anymore. Remember this used to work before i upgraded. Is this a change in pfSense or in HAproxy (both are upgraded at the same time when i upgraded to 2.4.4 the upgrade of Haproxy was mandetory.

[PiBa]

@leon-straathof
Its a change in the haproxy-package that caused this, and the upgrade code didn't take all possible scenario's into account properly. There used to be 1 checkbox behind a server, which auto-magically determined if it was required to do ssl-checks or to ssl-encrypt the traffic. Now there are 2 checkboxes that specifically instructs haproxy to do either one.. This gives the user more control..

In your case the main frontend does not do any decryption of traffic, so the backend should not add a second layer of ssl-encryption. Which is then passed to the second frontend or webserver which decrypts traffic once, and then doesn't know how to interpret the still ssl encrypted traffic as http..

[Leon Straathof]

@piba, thanks for the very clear explanation, funny that the checkbox also gets filled wrong when building a new pfSense and using my old config file. By any chance you know how i could improve my RDP solution. When i try to find examples for what i want i cannot find any. Only find rds-farm frontend and how to do sticky on that. But that is simple. I don't have a rds farm i just want to be able to connect to my machines at home (without a Remote Desktop Gateway) i want to use HAproxy to do the job of Remote Desktop Gateway. Currently i select the machine to connect internaly on the username that is at the front door.

[PiBa]

@leon-straathof
I do not know of a better way for RDP, most ive read about selecting the desired RDP server is that it couldn't be done at all.. Youve already been creative to use a username for specific target which apparently shows up as part of the rdp_cookie(mstshash). I don't think there are much better ways, besides using the "Remote Desktop Gateway", or there would need to be a way to mimic RDG's behavior with some extended lua scripting or something, probably tricky to do..

[Leon Straathof]

@piba I might try creating a lua script in the future, i guess i start reading about the inner working of Remote Desktop Gateway or try a attempt to decompile some MS binaries :-) to get a idea what they are doing. Thanks for the help so far.