Problem with new update to HaProxy



  • Problem with new update to HaProxy, My SSL offloading sites are no longer working I get the error:

    An error occurred during a connection to geneabujold.accra.ca. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG

    I use Haproxy in conjunction with PFsense Acme for the let's Encrypt certificates. verified certificates and they are valid. I presume it is something in my configuration but do not see my error.

    Please help, here is an example of one of the sites.

    Thanks

    Automaticaly generated, dont edit manually.

    Generated on: 2019-06-08 14:43

    global
    maxconn 10000
    log /var/run/log local0 alert
    stats socket /tmp/haproxy.socket level admin
    uid 80
    gid 80
    nbproc 1
    hard-stop-after 15m
    chroot /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param 2048
    log-send-hostname HaproxyMasterNode
    server-state-file /tmp/haproxy_server_state
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM
    ssl-default-bind-options no-sslv3 force-tlsv12

    listen HAProxyLocalStats
    bind 127.0.0.1:2200 name localstats
    mode http
    stats enable
    stats refresh 10
    stats admin if TRUE
    stats show-legends
    stats uri /haproxy/haproxy_stats.php?haproxystats=1
    timeout client 5000
    timeout connect 5000
    timeout server 5000

    mailers globalmailers
    mailer XXXXXXXXXX

    frontend Secure-offloading-3
    bind 127.0.0.1:1443 name 127.0.0.1:1443 ssl crt-list /var/etc/haproxy/Secure-offloading-3.crt_list
    bind /tmp/haproxy_chroot/Secure-offloading-3.socket name unixsocket uid 80 accept-proxy ssl crt-list /var/etc/haproxy/Secure-offloading-3.crt_list
    mode http
    log global
    option http-keep-alive
    option forwardfor
    acl https ssl_fc
    http-request set-header X-Forwarded-Proto http if !https
    http-request set-header X-Forwarded-Proto https if https
    timeout client 30000
    acl genealogie var(txn.txnhost) -m str -i geneabujold.accra.ca
    acl aclcrt_Secure-offloading-3 var(txn.txnhost) -m reg -i ^geneabujold.accra.ca(:([0-9]){1,5})?$
    http-request set-var(txn.txnhost) hdr(host)
    use_backend WebServer214_ipvANY if genealogie aclcrt_Secure-offloading-3
    use_backend WEBServer214_ipvANY if aclcrt_Secure-offloading-3

    backend NasWEBServer4_ipvANY
    mode http
    id 105
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to cjbujold@accra.ca
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server NasWEBServer4 192.168.20.4:80 id 106 check inter 1000

    backend WebServer214_ipvANY
    mode http
    id 117
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXX.XXX.com
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server WebServer214 192.168.20.14:80 id 118 check inter 1000



  • @cjbujold
    Somehow 'http://geneabujold.accra.ca:443/' gives a better response than when using https:// ... Anyhow i don't see where or how the :443 port is actually being listened on in the config you attached.. It seems the :443 is using plain un-encrypted connection and already speaking 'http'..? I guess the important part of the config isn't here.?



  • The config is for Haproxy to manage the SSL and the fontend offload the ssl (secure offloading config from the PFSense Haproxy doc) and then connect internally to port 80 of the internal server.

    There does not need to be a redirect to a 443 since PFSense does the ssl and not the server.


  • LAYER 8 Global Moderator

    @cjbujold said in Problem with new update to HaProxy:

    Problem with new update to HaProxy

    Which exact version are you running? And what did you update from?
    I am running
    haproxy-devel 0.59_19
    Package Dependencies:
    haproxy-1.8.17

    And I do ssl offload and not having any issues - but I do not recall any recent updates to haproxy? And just looked and not seeing any update available?



  • currently using 0.59_19, what can make the SSL_ERROR_RX_RECORD_TOO_LONG?

    I also noticed that when editing Actions list I no longer see the up and down arrows to move an action up or down in the list. Tried re-installing haproxy but they still do not show. Is HAProxy Gui the cause of my issue?



  • @cjbujold
    I'm not talking about a redirect to :443, i'm asking where is the client browser connecting to? There surely must be a 443 port listening somewhere?? I suspect on a frontend that is not in the config above but you actually do have?

    As for the up/down arrows those have been removed, and are replaced by the checkbox+anchor click options everywhere to make it more generic.. There used to be some lists with up/down arrows and others only with anchors, now everywhere the anchor icons are used to move rules around.


  • LAYER 8 Global Moderator

    Yeah there is no port forwarding or redirection to setup on pfsense if you using haproxy.. Just the rule to allow access to your wan IP on 443.

    Im even port sharing with openvpn on my setup.. And use acl in haproxy based on sni so get sent to 2 different backends depending, etc.



  • @cjbujold said in Problem with new update to HaProxy:

    what can make the SSL_ERROR_RX_RECORD_TOO_LONG?

    Try visiting this in firefox: https://google.com:80/ it will show that same message.. Browser expects https/ssl, but google is replying 'plain http'..



  • @PiBa I have 3 frontends:

    fontend 1-http port 80 web sites, works no problems

    SecureServers SNI-2 (stright HTTPS) which uses 443 uses a nat connection to 127.0.0.1 port 4443 which is the SNI-2 connection,

    anything that is not handled by SecureServer SNI-2 is forwarded to the default backend "Frontend3-Offloading" which is listening on port 1443 which is the third front-end that handles SSL -offloading and is the frontend that stopped working.



  • @cjbujold said in Problem with new update to HaProxy:

    SecureServers SNI-2 (stright HTTPS) which uses 443

    Can you share the config of that?



  • @PiBa here is an updated config (complete) did some changes to see if I could get it working (mostly cleanup) - no success. :

    Automaticaly generated, dont edit manually.

    Generated on: 2019-06-10 12:52

    global
    maxconn 10000
    log /var/run/log local0 alert
    stats socket /tmp/haproxy.socket level admin
    uid 80
    gid 80
    nbproc 1
    hard-stop-after 15m
    chroot /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param 2048
    log-send-hostname HaproxyMasterNode
    server-state-file /tmp/haproxy_server_state
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM
    ssl-default-bind-options no-sslv3 force-tlsv12

    listen HAProxyLocalStats
    bind 127.0.0.1:2200 name localstats
    mode http
    stats enable
    stats refresh 10
    stats admin if TRUE
    stats show-legends
    stats uri /haproxy/haproxy_stats.php?haproxystats=1
    timeout client 5000
    timeout connect 5000
    timeout server 5000

    mailers globalmailers
    mailer zeus.canspace.ca zeus.canspace.ca:26

    frontend Frontend1-http
    bind 156.34.233.202:80 name 156.34.233.202:80
    mode http
    log global
    option socket-stats
    option dontlog-normal
    option log-separate-errors
    option httplog
    option http-keep-alive
    option forwardfor
    acl https ssl_fc
    http-request set-header X-Forwarded-Proto http if !https
    http-request set-header X-Forwarded-Proto https if https
    timeout client 30000
    #remove header that expose security-sensitive information
    rspidel ^Server:.*S
    rspidel ^X-Powered-By:.*S
    rspidel ^X-AspNet-Version:.*S
    acl nas_acl var(txn.txnhost) -m beg -i famille
    acl syncbox_acl var(txn.txnhost) -m beg -i syncbox
    acl filoptoweb var(txn.txnhost) -m str -i www.filopto.com
    acl support var(txn.txnhost) -m str -i support.accra.ca
    acl remotehelp_acl var(txn.txnhost) -m str -i remotehelp.accra.ca
    acl filoptoreg_acl var(txn.txnhost) -m str -i reg.filopto.com
    acl helpfilopto var(txn.txnhost) -m str -i help10.filopto.com
    acl helpfilopto var(txn.txnhost) -m str -i help.filopto.com
    acl helpbackup var(txn.txnhost) -m str -i help.accrabackup.accra.ca
    acl backuphelp var(txn.txnhost) -m str -i backuphelp.accra.ca
    acl genealogy var(txn.txnhost) -m beg -i geneabujold.accra.ca
    acl supportfilopto var(txn.txnhost) -m str -i support.filopto.com
    acl dragondreams_acl var(txn.txnhost) -m end -i dragondreams.ca
    acl medicalcoderaccra var(txn.txnhost) -m str -i medicalcoder.accra.ca
    acl medicalcoderfilopto var(txn.txnhost) -m str -i medicalcoder.filopto.com
    acl filopto_acl var(txn.txnhost) -m end -i filopto.com
    acl home var(txn.txnhost) -m beg -i home.accra.ca
    acl genealogie var(txn.txnhost) -m beg -i genealogie.bujold.ca
    acl geneatng var(txn.txnhost) -m beg -i genea.bujold.ca
    acl syncbox_acl var(txn.txnhost) -m str -i secure.accra.ca
    acl accraphp19 var(txn.txnhost) -m str -i accraphp19.accra.ca
    acl accra_acl var(txn.txnhost) -m end -i accra.ca
    acl securebackup var(txn.txnhost) -m beg -i securebackup.accra.ca
    http-request set-var(txn.txnhost) hdr(host)
    http-request redirect scheme https if filoptoweb
    http-request redirect scheme https if support
    http-request redirect scheme https if supportfilopto
    http-request redirect scheme https if helpfilopto
    http-request redirect scheme https if backuphelp
    http-request redirect scheme https if helpbackup
    http-request redirect scheme https if accraphp19
    use_backend NasWEBServer4_ipvANY if nas_acl
    use_backend frontend3-offloading-redirect_ipvANY if syncbox_acl
    use_backend RemoteHelp25_ipvANY if remotehelp_acl
    use_backend WEBServer14_ipvANY if filoptoreg_acl
    use_backend WEBServer14_ipvANY if filopto_acl
    use_backend WEBServer14_ipvANY if dragondreams_acl
    use_backend WEBServer14_ipvANY if medicalcoderaccra
    use_backend WEBServer14_ipvANY if medicalcoderfilopto
    use_backend WEBServer14_ipvANY if accra_acl
    use_backend WEBServer14_ipvANY if genealogie
    use_backend WEBServer14_ipvANY if geneatng
    use_backend WEBServer14_ipvANY if genealogy
    use_backend WEBServer14_ipvANY if home
    use_backend frontend3-offloading-redirect_ipvANY if securebackup
    default_backend WEBServer14_ipvANY

    frontend SecureServers-SNI-2
    bind 156.34.233.202:443 name 156.34.233.202:443
    mode tcp
    log global
    option socket-stats
    option log-separate-errors
    option tcplog
    timeout client 30000
    tcp-request inspect-delay 5s
    acl ftpweb_acl req.ssl_sni -i ftpweb.accra.ca
    acl wwwfilopto req.ssl_sni -i www.filopto.com
    acl updatefilopto req.ssl_sni -i update.filopto.com
    acl securebackup req.ssl_sni -i securebackup.accra.ca
    tcp-request content accept if { req.ssl_hello_type 1 }
    use_backend SecureFTPWEB214_ipvANY if ftpweb_acl
    use_backend ssl14backend_ipvANY if wwwfilopto
    use_backend ssl14backend_ipvANY if updatefilopto
    use_backend ssl14backend_ipvANY if securebackup
    default_backend frontend3-offloading-redirect_ipvANY

    frontend Https-offloading-3
    bind 127.0.0.1:1443 name 127.0.0.1:1443 ssl crt-list /var/etc/haproxy/Https-offloading-3.crt_list
    bind /tmp/haproxy_chroot/Https-offloading-3.socket name unixsocket uid 80 accept-proxy ssl crt-list /var/etc/haproxy/Https-offloading-3.crt_list
    mode http
    log global
    option http-keep-alive
    option forwardfor
    acl https ssl_fc
    http-request set-header X-Forwarded-Proto http if !https
    http-request set-header X-Forwarded-Proto https if https
    timeout client 30000
    acl filoptoreg var(txn.txnhost) -m str -i reg.filopto.com
    acl remotehelp var(txn.txnhost) -m str -i remotehelp.accra.ca
    acl familleNas var(txn.txnhost) -m str -i famille.accra.ca
    acl genealogie var(txn.txnhost) -m str -i geneabujold.accra.ca
    acl support var(txn.txnhost) -m str -i support.accra.ca
    acl updatefilopto var(txn.txnhost) -m str -i update.filopto.com
    acl supportfilopto var(txn.txnhost) -m str -i support.filopto.com
    acl supportaccra var(txn.txnhost) -m str -i support.accra.ca
    acl clientbackup var(txn.txnhost) -m str -i secure.accra.ca
    acl accraphp19 var(txn.txnhost) -m str -i accraphp19.accra.ca
    acl helpfilopto var(txn.txnhost) -m str -i help10.filopto.com
    acl backuphelp var(txn.txnhost) -m beg -i backuphelp
    acl helpbackup var(txn.txnhost) -m beg -i help.accrabackup.accra.ca
    acl updateaccra var(txn.txnhost) -m str -i update.accra.ca
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^accraphp19.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^backuphelp.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^famille.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^ftpweb.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^geneabujold.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^help.accrabackup.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^medicalcoder.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^protector.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^remotehelp.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^secure.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^support.accra.ca(:([0-9]){1,5})?$
    acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^update.accra.ca(:([0-9]){1,5})?$
    http-request set-var(txn.txnhost) hdr(host)
    use_backend WEBServer14_ipvANY if filoptoreg aclcrt_Https-offloading-3
    use_backend RemoteHelp25_ipvANY if remotehelp aclcrt_Https-offloading-3
    use_backend SecureNAS4_ipvANY if familleNas aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if support aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if updatefilopto aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if supportfilopto aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if genealogie aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if accraphp19 aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if helpfilopto aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if backuphelp aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if helpbackup aclcrt_Https-offloading-3
    use_backend WEBServer14_ipvANY if updateaccra aclcrt_Https-offloading-3

    backend NasWEBServer4_ipvANY
    mode http
    id 105
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXXXXXXXXXXXXX
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server NasWEBServer4 192.168.20.4:80 id 106 check inter 1000

    backend frontend3-offloading-redirect_ipvANY
    mode http
    id 103
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXXXXXXXXXXXXX
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    server frontend3-offloading /Https-offloading-3.socket send-proxy-v2-ssl-cn id 101 ssl check inter 5000 verify none

    backend RemoteHelp25_ipvANY
    mode http
    id 107
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXXXXXXXXXXXXX
    email-alert myhostname protector.accra.ca
    option log-health-checks
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server Remotehelp 192.168.20.25:80 id 108 check inter 1000

    backend WEBServer14_ipvANY
    mode http
    id 115
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXXXXXXXXXXXXX
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    option httpchk OPTIONS /
    server AccraWEB14 192.168.20.14:80 id 116 check inter 1000

    backend SecureFTPWEB214_ipvANY
    mode tcp
    id 111
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXXXXXXXXXXXXX
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    server secureFTPweb 192.168.20.14:3443 id 112 check inter 1000

    backend ssl14backend_ipvANY
    mode tcp
    id 119
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXXXXXXXXXXXXX
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    server ssl14server 192.168.20.14:443 id 120 check inter 1000

    backend SecureNAS4_ipvANY
    mode http
    id 113
    log global
    # use mailers
    # level alert
    email-alert mailers globalmailers
    email-alert level alert
    email-alert from protector.accra.ca
    email-alert to XXXXXXXXXXXXXXXXXX
    email-alert myhostname protector.accra.ca
    timeout connect 30000
    timeout server 30000
    retries 3
    server SecureNas4 192.168.20.4:6240 id 114 check inter 1000



  • @cjbujold
    Can you disable the SSL-Encryption checkbox on the server 'frontend3-offloading' of the the backend 'frontend3-offloading-redirect' ? It causes the already encrypted request by the browser to be encrypted again.



  • Un checked Encrypt(SSL) field in the 'frontend3-offloading-redirect' backend, I don't see any changes



  • @cjbujold
    Can you clone the 'frontend3-offloading-redirect' backend specifically for SSL and use that new cloned backend in the 'SecureServers-SNI-2' frontend?

    I see its using 'mode http' while as being a backend used for the SNI frontend it should still be using 'mode tcp' in the configuration.



  • Clone and named frontend3-offloading-redirect-2 applied the change

    it seems to work now , but I do not understand my error or what cause it.
    I am getting a 503 error on one web site, I have to look to find out why?

    Thank you would not have been able to find the error with out you. Much appreciated.

    Charles


Log in to reply