HAProxy config fails to operate properly after 2.4.4 upgrade



  • My working HAProxy config fails to operate properly after 2.4.4 upgrade, i also switched to the HAproxy development branch to see if that would fix it again but it did not, upgrade of pfSense to 2.4.4_1 does not fix it either.
    I also did a fresh install but that also did not fix the problem.

    Chrome says ERR_CONNECTION_CLOSED when trying to connect trough HAproxy and IE11 says "Can’t connect securely to this page".

    My configuration of HAproxy is similar to the config described here: http://loredo.me/post/116633549315/geeking-out-with-haproxy-on-pfsense-the-ultimate



  • @leon-straathof
    Can you please share the actual haproxy.cfg you have. (bottom of settings tab) (perhaps obfuscate the IP's and domains a little..)



  • Sure it is quite big, and after some more testing i found that only the HTTPS parts are not working any more after the upgrade. The OpenVPN, RDP ,SSH and HTTP redirector are all stil working. I changed all ip adresses and names to fictional information. And i cut out large repetative parts (If one part is fixed i can apply the same on everything). And don't judge me on the silly RDP implementation i use it to be able to rdp into all my machines (i don't know how to select machines in tcp on subdomain so i select the machine on unique username per machine) and hey it works for me so i am happy. So the only open port is 443 for everything i need.

    Automaticaly generated, dont edit manually.

    Generated on: 2018-12-07 16:46

    global
    maxconn 100
    stats socket /tmp/haproxy.socket level admin
    uid 80
    gid 80
    nbproc 1
    hard-stop-after 15m
    chroot /tmp/haproxy_chroot
    daemon
    server-state-file /tmp/haproxy_server_state
    # Modern browser compatibility only as mentioned here:
    # https://wiki.mozilla.org/Security/Server_Side_TLS
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
    tune.ssl.default-dh-param 2048

    # Time-to-first-Byte (TTFB) value needs to be optimized based on
    # the actual public certificate chain see
    # https://www.igvita.com/2013/10/24
    # /optimizing-tls-record-size-and-buffering-latency/
    tune.ssl.maxrecord 1370
    

    listen HAProxyLocalStats
    bind 127.0.0.1:2200 name localstats
    mode http
    stats enable
    stats admin if TRUE
    stats show-legends
    stats uri /haproxy/haproxy_stats.php?haproxystats=1
    timeout client 5000
    timeout connect 5000
    timeout server 5000

    frontend WAN_443-merged
    bind 192.168.1.10:443 name 192.168.1.10:443
    mode tcp
    log global
    timeout client 7200000
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 } or !{ req.ssl_hello_type 1 }
    acl ovpn1 req.len 0
    acl ovpn2 req.ssl_hello_type 1
    acl ovpn3 req.rdp_cookie_cnt 0
    acl https1 req.ssl_hello_type 1
    acl https2 req.ssl_sni -m end -i vpn.acme.com
    acl https3 req.ssl_sni -m end -i ssh.acme.com
    acl vpn1 req.ssl_hello_type 1
    acl vpn2 req.ssl_sni -m end -i .vpn.acme.com
    acl ssh1 req.ssl_hello_type 1
    acl ssh2 req.ssl_sni -m end -i .ssh.acme.com
    acl rdp0 req.rdp_cookie_cnt 0
    acl rdp1 rdp_cookie(mstshash) -m sub -i deskuser
    acl rdp2 rdp_cookie(mstshash) -m sub -i servuser
    use_backend OpenVPN_ipvANY if !ovpn1 !ovpn2 ovpn3
    use_backend WAN_HTTPS_ipvANY if https1 !https2 !https3
    use_backend WAN_HTTPS_auth_ipvANY if vpn1 vpn2
    use_backend WAN_SSLH_ipvANY if ssh1 ssh2
    use_backend RDP_w10_1_ipvANY if !rdp0 rdp1
    use_backend RDP_w2019s1_ipvANY if !rdp0 rdp2

    frontend WAN_HTTPS-merged
    bind 127.0.0.1:2043 name 127.0.0.1:2043 no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS.crt_list accept-proxy npn http/1.1
    mode http
    log global
    option http-keep-alive
    option forwardfor
    acl https ssl_fc
    http-request set-header X-Forwarded-Proto http if !https
    http-request set-header X-Forwarded-Proto https if https
    timeout client 7200000
    errorfile 503 /var/etc/haproxy/errorfile_WAN_HTTPS_503_ExampleErrorfile
    # Remove headers that expose security-sensitive information.
    rspidel ^Server:.$
    rspidel ^X-Powered-By:.
    $
    rspidel ^X-AspNet-Version:.*$
    rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains
    acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
    acl guacamole var(txn.txnhost) -m str -i guacamole.acme.com
    acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_guacamole var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
    acl nuget var(txn.txnhost) -m str -i nuget.acme.com
    acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_nuget var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
    acl sinusbot var(txn.txnhost) -m str -i sinusbot.acme.com
    acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^guacamole.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^guacamole.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^nuget.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^nuget.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^sinusbot.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_sinusbot var(txn.txnhost) -m reg -i ^sinusbot.vpn.acme.com(:([0-9]){1,5})?$
    http-request set-var(txn.txnhost) hdr(host)
    use_backend Guacamole_ipvANY if guacamole aclcrt_WAN_HTTPS_guacamole
    use_backend Nuget_ipvANY if nuget aclcrt_WAN_HTTPS_nuget
    use_backend SinusBot_ipvANY if sinusbot aclcrt_WAN_HTTPS_sinusbot

    frontend WAN_HTTPS_auth-merged
    bind 127.0.0.1:2044 name 127.0.0.1:2044 no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS_auth.crt_list ca-file /var/etc/haproxy/clientca_WAN_HTTPS_auth.pem verify required crt-ignore-err all accept-proxy npn http/1.1
    mode http
    log global
    option http-keep-alive
    option forwardfor
    acl https ssl_fc
    http-request set-header X-Forwarded-Proto http if !https
    http-request set-header X-Forwarded-Proto https if https
    timeout client 7200000
    # Remove headers that expose security-sensitive information.
    rspidel ^Server:.$
    rspidel ^X-Powered-By:.
    $
    rspidel ^X-AspNet-Version:.*$
    acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_auth var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
    acl pfsense var(txn.txnhost) -m str -i pfsense.vpn.acme.com
    acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_pfsense var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
    acl R7800 var(txn.txnhost) -m str -i r7800.vpn.acme.com
    acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_dd-wrt var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
    acl modem var(txn.txnhost) -m str -i modem.vpn.acme.com
    acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^modem.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^modem.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^pfsense.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^pfsense.vpn.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^r7800.acme.com(:([0-9]){1,5})?$
    acl aclcrt_WAN_HTTPS_AUTH_modem var(txn.txnhost) -m reg -i ^r7800.vpn.acme.com(:([0-9]){1,5})?$
    http-request set-var(txn.txnhost) hdr(host)
    use_backend pfSense_ipvANY if pfsense aclcrt_WAN_HTTPS_AUTH_pfsense
    use_backend R7800_ipvANY if R7800 aclcrt_WAN_HTTPS_AUTH_dd-wrt
    use_backend modem_ipvANY if modem aclcrt_WAN_HTTPS_AUTH_modem

    frontend WAN_SSLH-merged
    bind 127.0.0.1:2022 name 127.0.0.1:2022 no-sslv3 ssl crt-list /var/etc/haproxy/WAN_SSLH.crt_list ca-file /var/etc/haproxy/clientca_WAN_SSLH.pem verify required accept-proxy npn ssh/2.0
    mode tcp
    log global
    timeout client 7200000
    acl ssh_pfsense1 ssl_fc_npn -i ssh/2.0
    acl ssh_pfsense2 ssl_fc_sni_reg pfsense.ssh.acme.com
    acl ssh_guacamole1 ssl_fc_npn -i ssh/2.0
    acl ssh_guacamole2 ssl_fc_sni_reg guacamole.ssh.acme.com
    acl ssh_w2016s1_1 ssl_fc_npn -i ssh/2.0
    acl ssh_w2016s1_2 ssl_fc_sni_reg guacamole.ssh.acme.com
    use_backend ssh_pfsense_ipvANY if ssh_pfsense1 ssh_pfsense2
    use_backend ssh_guacamole_ipvANY if ssh_guacamole1 ssh_guacamole2
    use_backend ssh_w2016s1_ipvANY if ssh_w2016s1_1 ssh_w2016s1_2

    frontend WAN_HTTP
    bind 192.168.1.10:80 name 192.168.1.10:80
    mode http
    log global
    option http-keep-alive
    timeout client 30000
    acl acme-challenge var(txn.txnpath) -m beg -i /.well-known/acme-challenge/
    http-request set-var(txn.txnpath) path
    use_backend pfSense_ipvANY if acme-challenge
    default_backend ssl-redirect_ipvANY

    backend OpenVPN_ipvANY
    mode tcp
    id 118
    log global
    timeout connect 30000
    timeout server 7200000
    retries 2
    server OpenVPN 127.0.0.1:1194 id 119 weight 1

    backend WAN_HTTPS_ipvANY
    mode tcp
    id 130
    log global
    timeout connect 30000
    timeout server 7200000
    retries 3
    server WAN_HHTPS 127.0.0.1:2043 id 131 ssl verify none send-proxy

    backend WAN_HTTPS_auth_ipvANY
    mode tcp
    id 132
    log global
    timeout connect 30000
    timeout server 7200000
    retries 3
    server WAN_HTTPS_auth 127.0.0.1:2044 id 133 ssl verify none send-proxy

    backend WAN_SSLH_ipvANY
    mode tcp
    id 134
    log global
    timeout connect 30000
    timeout server 7200000
    retries 3
    server WAN_SSLH 127.0.0.1:2022 id 135 ssl verify none send-proxy

    backend RDP_w10_1_ipvANY
    mode tcp
    id 102
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server w10_1 192.168.2.32:3389 id 103 check inter 1000

    backend RDP_w2019s1_ipvANY
    mode tcp
    id 148
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server w2019s1 192.168.5.19:3389 id 103 check inter 1000

    backend Guacamole_ipvANY
    mode http
    id 110
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server Guacamole 192.168.2.18:8080 id 111 check inter 1000 weight 1

    backend Nuget_ipvANY
    mode http
    id 116
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server Nuget 192.168.5.19:80 id 117 check inter 1000 weight 1

    backend SinusBot_ipvANY
    mode http
    id 122
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk GET /
    server SinusBot 192.168.5.19:8087 id 123 check inter 1000 weight 1

    backend pfSense_ipvANY
    mode http
    id 120
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server pfSense 127.0.0.1:80 id 121 weight 1

    backend R7800_ipvANY
    mode http
    id 104
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server R7800 192.168.2.2:80 id 105 check inter 1000 weight 1

    backend modem_ipvANY
    mode http
    id 136
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    server modem 192.168.1.1:80 id 137 check inter 5000 weight 1

    backend ssh_pfsense_ipvANY
    mode tcp
    id 126
    log global
    timeout connect 3000
    timeout server 7200000
    retries 2
    server ssh_pfsense 192.168.2.3:22 id 127

    backend ssh_guacamole_ipvANY
    mode tcp
    id 124
    log global
    timeout connect 3000
    timeout server 7200000
    retries 2
    server ssh_guacamole 192.168.2.18:22 id 125 check inter 1000

    backend ssh_w2016s1_ipvANY
    mode tcp
    id 140
    log global
    timeout connect 3000
    timeout server 7200000
    retries 2
    server ssh_w2016s1 192.168.5.16:22 id 141 check inter 1000

    backend ssl-redirect_ipvANY
    mode http
    id 128
    log global
    timeout connect 30000
    timeout server 30000
    retries 3
    redirect scheme https code 301



  • @leon-straathof
    Can you try and disable the 'SSL Encryption' checkbox in the "WAN_HTTPS" backend.?

    (Also for the WAN_HTTPS_auth and WAN_SSLH backend servers)



  • PiBa, that worked. Can you explain why because i am stumped that a internal only part cannot be encrypted anymore. Remember this used to work before i upgraded. Is this a change in pfSense or in HAproxy (both are upgraded at the same time when i upgraded to 2.4.4 the upgrade of Haproxy was mandetory.



  • @leon-straathof
    Its a change in the haproxy-package that caused this, and the upgrade code didn't take all possible scenario's into account properly. There used to be 1 checkbox behind a server, which auto-magically determined if it was required to do ssl-checks or to ssl-encrypt the traffic. Now there are 2 checkboxes that specifically instructs haproxy to do either one.. This gives the user more control..

    In your case the main frontend does not do any decryption of traffic, so the backend should not add a second layer of ssl-encryption. Which is then passed to the second frontend or webserver which decrypts traffic once, and then doesn't know how to interpret the still ssl encrypted traffic as http..



  • @piba, thanks for the very clear explanation, funny that the checkbox also gets filled wrong when building a new pfSense and using my old config file. By any chance you know how i could improve my RDP solution. When i try to find examples for what i want i cannot find any. Only find rds-farm frontend and how to do sticky on that. But that is simple. I don't have a rds farm i just want to be able to connect to my machines at home (without a Remote Desktop Gateway) i want to use HAproxy to do the job of Remote Desktop Gateway. Currently i select the machine to connect internaly on the username that is at the front door.



  • @leon-straathof
    I do not know of a better way for RDP, most ive read about selecting the desired RDP server is that it couldn't be done at all.. Youve already been creative to use a username for specific target which apparently shows up as part of the rdp_cookie(mstshash). I don't think there are much better ways, besides using the "Remote Desktop Gateway", or there would need to be a way to mimic RDG's behavior with some extended lua scripting or something, probably tricky to do..



  • @piba I might try creating a lua script in the future, i guess i start reading about the inner working of Remote Desktop Gateway or try a attempt to decompile some MS binaries :-) to get a idea what they are doing. Thanks for the help so far.