Problem with new update to HaProxy

cjbujold · Jun 8, 2019, 5:58 PM

Problem with new update to HaProxy, My SSL offloading sites are no longer working I get the error:

An error occurred during a connection to geneabujold.accra.ca. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG

I use Haproxy in conjunction with PFsense Acme for the let's Encrypt certificates. verified certificates and they are valid. I presume it is something in my configuration but do not see my error.

Please help, here is an example of one of the sites.

Thanks

Automaticaly generated, dont edit manually.

Generated on: 2019-06-08 14:43

global
maxconn 10000
log /var/run/log local0 alert
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
log-send-hostname HaproxyMasterNode
server-state-file /tmp/haproxy_server_state
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM
ssl-default-bind-options no-sslv3 force-tlsv12

listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats refresh 10
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

mailers globalmailers
mailer XXXXXXXXXX

frontend Secure-offloading-3
bind 127.0.0.1:1443 name 127.0.0.1:1443 ssl crt-list /var/etc/haproxy/Secure-offloading-3.crt_list
bind /tmp/haproxy_chroot/Secure-offloading-3.socket name unixsocket uid 80 accept-proxy ssl crt-list /var/etc/haproxy/Secure-offloading-3.crt_list
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl genealogie var(txn.txnhost) -m str -i geneabujold.accra.ca
acl aclcrt_Secure-offloading-3 var(txn.txnhost) -m reg -i ^geneabujold.accra.ca(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend WebServer214_ipvANY if genealogie aclcrt_Secure-offloading-3
use_backend WEBServer214_ipvANY if aclcrt_Secure-offloading-3

backend NasWEBServer4_ipvANY
mode http
id 105
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to cjbujold@accra.ca
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server NasWEBServer4 192.168.20.4:80 id 106 check inter 1000

backend WebServer214_ipvANY
mode http
id 117
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXX.XXX.com
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server WebServer214 192.168.20.14:80 id 118 check inter 1000

PiBa · Jun 8, 2019, 6:57 PM

@cjbujold
Somehow 'http://geneabujold.accra.ca:443/' gives a better response than when using https:// ... Anyhow i don't see where or how the :443 port is actually being listened on in the config you attached.. It seems the :443 is using plain un-encrypted connection and already speaking 'http'..? I guess the important part of the config isn't here.?

cjbujold · Jun 9, 2019, 1:58 PM

The config is for Haproxy to manage the SSL and the fontend offload the ssl (secure offloading config from the PFSense Haproxy doc) and then connect internally to port 80 of the internal server.

There does not need to be a redirect to a 443 since PFSense does the ssl and not the server.

johnpoz · Jun 8, 2019, 11:50 PM

@cjbujold said in Problem with new update to HaProxy:

Problem with new update to HaProxy

Which exact version are you running? And what did you update from?
I am running
haproxy-devel 0.59_19
Package Dependencies:
haproxy-1.8.17

And I do ssl offload and not having any issues - but I do not recall any recent updates to haproxy? And just looked and not seeing any update available?

cjbujold · Jun 9, 2019, 2:44 PM

currently using 0.59_19, what can make the SSL_ERROR_RX_RECORD_TOO_LONG?

I also noticed that when editing Actions list I no longer see the up and down arrows to move an action up or down in the list. Tried re-installing haproxy but they still do not show. Is HAProxy Gui the cause of my issue?

PiBa · Jun 9, 2019, 1:58 PM

@cjbujold
I'm not talking about a redirect to :443, i'm asking where is the client browser connecting to? There surely must be a 443 port listening somewhere?? I suspect on a frontend that is not in the config above but you actually do have?

As for the up/down arrows those have been removed, and are replaced by the checkbox+anchor click options everywhere to make it more generic.. There used to be some lists with up/down arrows and others only with anchors, now everywhere the anchor icons are used to move rules around.

johnpoz · Jun 9, 2019, 2:03 PM

Yeah there is no port forwarding or redirection to setup on pfsense if you using haproxy.. Just the rule to allow access to your wan IP on 443.

Im even port sharing with openvpn on my setup.. And use acl in haproxy based on sni so get sent to 2 different backends depending, etc.

PiBa · Jun 9, 2019, 2:44 PM

@cjbujold said in Problem with new update to HaProxy:

what can make the SSL_ERROR_RX_RECORD_TOO_LONG?

Try visiting this in firefox: https://google.com:80/ it will show that same message.. Browser expects https/ssl, but google is replying 'plain http'..

cjbujold · Jun 10, 2019, 11:23 AM

@PiBa I have 3 frontends:

fontend 1-http port 80 web sites, works no problems

SecureServers SNI-2 (stright HTTPS) which uses 443 uses a nat connection to 127.0.0.1 port 4443 which is the SNI-2 connection,

anything that is not handled by SecureServer SNI-2 is forwarded to the default backend "Frontend3-Offloading" which is listening on port 1443 which is the third front-end that handles SSL -offloading and is the frontend that stopped working.

PiBa · Jun 10, 2019, 3:56 PM

@cjbujold said in Problem with new update to HaProxy:

SecureServers SNI-2 (stright HTTPS) which uses 443

Can you share the config of that?

cjbujold · Jun 10, 2019, 3:56 PM

@PiBa here is an updated config (complete) did some changes to see if I could get it working (mostly cleanup) - no success. :

Automaticaly generated, dont edit manually.

Generated on: 2019-06-10 12:52

global
maxconn 10000
log /var/run/log local0 alert
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
log-send-hostname HaproxyMasterNode
server-state-file /tmp/haproxy_server_state
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM
ssl-default-bind-options no-sslv3 force-tlsv12

listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats refresh 10
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

mailers globalmailers
mailer zeus.canspace.ca zeus.canspace.ca:26

frontend Frontend1-http
bind 156.34.233.202:80 name 156.34.233.202:80
mode http
log global
option socket-stats
option dontlog-normal
option log-separate-errors
option httplog
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
#remove header that expose security-sensitive information
rspidel ^Server:.*S
rspidel ^X-Powered-By:.*S
rspidel ^X-AspNet-Version:.*S
acl nas_acl var(txn.txnhost) -m beg -i famille
acl syncbox_acl var(txn.txnhost) -m beg -i syncbox
acl filoptoweb var(txn.txnhost) -m str -i www.filopto.com
acl support var(txn.txnhost) -m str -i support.accra.ca
acl remotehelp_acl var(txn.txnhost) -m str -i remotehelp.accra.ca
acl filoptoreg_acl var(txn.txnhost) -m str -i reg.filopto.com
acl helpfilopto var(txn.txnhost) -m str -i help10.filopto.com
acl helpfilopto var(txn.txnhost) -m str -i help.filopto.com
acl helpbackup var(txn.txnhost) -m str -i help.accrabackup.accra.ca
acl backuphelp var(txn.txnhost) -m str -i backuphelp.accra.ca
acl genealogy var(txn.txnhost) -m beg -i geneabujold.accra.ca
acl supportfilopto var(txn.txnhost) -m str -i support.filopto.com
acl dragondreams_acl var(txn.txnhost) -m end -i dragondreams.ca
acl medicalcoderaccra var(txn.txnhost) -m str -i medicalcoder.accra.ca
acl medicalcoderfilopto var(txn.txnhost) -m str -i medicalcoder.filopto.com
acl filopto_acl var(txn.txnhost) -m end -i filopto.com
acl home var(txn.txnhost) -m beg -i home.accra.ca
acl genealogie var(txn.txnhost) -m beg -i genealogie.bujold.ca
acl geneatng var(txn.txnhost) -m beg -i genea.bujold.ca
acl syncbox_acl var(txn.txnhost) -m str -i secure.accra.ca
acl accraphp19 var(txn.txnhost) -m str -i accraphp19.accra.ca
acl accra_acl var(txn.txnhost) -m end -i accra.ca
acl securebackup var(txn.txnhost) -m beg -i securebackup.accra.ca
http-request set-var(txn.txnhost) hdr(host)
http-request redirect scheme https if filoptoweb
http-request redirect scheme https if support
http-request redirect scheme https if supportfilopto
http-request redirect scheme https if helpfilopto
http-request redirect scheme https if backuphelp
http-request redirect scheme https if helpbackup
http-request redirect scheme https if accraphp19
use_backend NasWEBServer4_ipvANY if nas_acl
use_backend frontend3-offloading-redirect_ipvANY if syncbox_acl
use_backend RemoteHelp25_ipvANY if remotehelp_acl
use_backend WEBServer14_ipvANY if filoptoreg_acl
use_backend WEBServer14_ipvANY if filopto_acl
use_backend WEBServer14_ipvANY if dragondreams_acl
use_backend WEBServer14_ipvANY if medicalcoderaccra
use_backend WEBServer14_ipvANY if medicalcoderfilopto
use_backend WEBServer14_ipvANY if accra_acl
use_backend WEBServer14_ipvANY if genealogie
use_backend WEBServer14_ipvANY if geneatng
use_backend WEBServer14_ipvANY if genealogy
use_backend WEBServer14_ipvANY if home
use_backend frontend3-offloading-redirect_ipvANY if securebackup
default_backend WEBServer14_ipvANY

frontend SecureServers-SNI-2
bind 156.34.233.202:443 name 156.34.233.202:443
mode tcp
log global
option socket-stats
option log-separate-errors
option tcplog
timeout client 30000
tcp-request inspect-delay 5s
acl ftpweb_acl req.ssl_sni -i ftpweb.accra.ca
acl wwwfilopto req.ssl_sni -i www.filopto.com
acl updatefilopto req.ssl_sni -i update.filopto.com
acl securebackup req.ssl_sni -i securebackup.accra.ca
tcp-request content accept if { req.ssl_hello_type 1 }
use_backend SecureFTPWEB214_ipvANY if ftpweb_acl
use_backend ssl14backend_ipvANY if wwwfilopto
use_backend ssl14backend_ipvANY if updatefilopto
use_backend ssl14backend_ipvANY if securebackup
default_backend frontend3-offloading-redirect_ipvANY

frontend Https-offloading-3
bind 127.0.0.1:1443 name 127.0.0.1:1443 ssl crt-list /var/etc/haproxy/Https-offloading-3.crt_list
bind /tmp/haproxy_chroot/Https-offloading-3.socket name unixsocket uid 80 accept-proxy ssl crt-list /var/etc/haproxy/Https-offloading-3.crt_list
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl filoptoreg var(txn.txnhost) -m str -i reg.filopto.com
acl remotehelp var(txn.txnhost) -m str -i remotehelp.accra.ca
acl familleNas var(txn.txnhost) -m str -i famille.accra.ca
acl genealogie var(txn.txnhost) -m str -i geneabujold.accra.ca
acl support var(txn.txnhost) -m str -i support.accra.ca
acl updatefilopto var(txn.txnhost) -m str -i update.filopto.com
acl supportfilopto var(txn.txnhost) -m str -i support.filopto.com
acl supportaccra var(txn.txnhost) -m str -i support.accra.ca
acl clientbackup var(txn.txnhost) -m str -i secure.accra.ca
acl accraphp19 var(txn.txnhost) -m str -i accraphp19.accra.ca
acl helpfilopto var(txn.txnhost) -m str -i help10.filopto.com
acl backuphelp var(txn.txnhost) -m beg -i backuphelp
acl helpbackup var(txn.txnhost) -m beg -i help.accrabackup.accra.ca
acl updateaccra var(txn.txnhost) -m str -i update.accra.ca
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^accraphp19.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^backuphelp.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^famille.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^ftpweb.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^geneabujold.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^help.accrabackup.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^medicalcoder.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^protector.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^remotehelp.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^secure.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^support.accra.ca(:([0-9]){1,5})?$
acl aclcrt_Https-offloading-3 var(txn.txnhost) -m reg -i ^update.accra.ca(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend WEBServer14_ipvANY if filoptoreg aclcrt_Https-offloading-3
use_backend RemoteHelp25_ipvANY if remotehelp aclcrt_Https-offloading-3
use_backend SecureNAS4_ipvANY if familleNas aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if support aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if updatefilopto aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if supportfilopto aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if genealogie aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if accraphp19 aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if helpfilopto aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if backuphelp aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if helpbackup aclcrt_Https-offloading-3
use_backend WEBServer14_ipvANY if updateaccra aclcrt_Https-offloading-3

backend NasWEBServer4_ipvANY
mode http
id 105
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXXXXXXXXXXXXX
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server NasWEBServer4 192.168.20.4:80 id 106 check inter 1000

backend frontend3-offloading-redirect_ipvANY
mode http
id 103
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXXXXXXXXXXXXX
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
server frontend3-offloading /Https-offloading-3.socket send-proxy-v2-ssl-cn id 101 ssl check inter 5000 verify none

backend RemoteHelp25_ipvANY
mode http
id 107
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXXXXXXXXXXXXX
email-alert myhostname protector.accra.ca
option log-health-checks
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server Remotehelp 192.168.20.25:80 id 108 check inter 1000

backend WEBServer14_ipvANY
mode http
id 115
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXXXXXXXXXXXXX
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server AccraWEB14 192.168.20.14:80 id 116 check inter 1000

backend SecureFTPWEB214_ipvANY
mode tcp
id 111
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXXXXXXXXXXXXX
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
server secureFTPweb 192.168.20.14:3443 id 112 check inter 1000

backend ssl14backend_ipvANY
mode tcp
id 119
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXXXXXXXXXXXXX
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
server ssl14server 192.168.20.14:443 id 120 check inter 1000

backend SecureNAS4_ipvANY
mode http
id 113
log global
# use mailers
# level alert
email-alert mailers globalmailers
email-alert level alert
email-alert from protector.accra.ca
email-alert to XXXXXXXXXXXXXXXXXX
email-alert myhostname protector.accra.ca
timeout connect 30000
timeout server 30000
retries 3
server SecureNas4 192.168.20.4:6240 id 114 check inter 1000

PiBa · Jun 10, 2019, 6:05 PM

@cjbujold
Can you disable the SSL-Encryption checkbox on the server 'frontend3-offloading' of the the backend 'frontend3-offloading-redirect' ? It causes the already encrypted request by the browser to be encrypted again.

cjbujold · Jun 10, 2019, 6:28 PM

Un checked Encrypt(SSL) field in the 'frontend3-offloading-redirect' backend, I don't see any changes

PiBa · Jun 10, 2019, 6:44 PM

@cjbujold
Can you clone the 'frontend3-offloading-redirect' backend specifically for SSL and use that new cloned backend in the 'SecureServers-SNI-2' frontend?

I see its using 'mode http' while as being a backend used for the SNI frontend it should still be using 'mode tcp' in the configuration.

cjbujold · Jun 10, 2019, 7:07 PM

Clone and named frontend3-offloading-redirect-2 applied the change

it seems to work now , but I do not understand my error or what cause it.
I am getting a 503 error on one web site, I have to look to find out why?

Thank you would not have been able to find the error with out you. Much appreciated.

Charles