HAProxy https no longer works

[ivaped]

Hi,

After the latest HAProxy upgrade (No sure from what version I upgraded from though..) my HTTPS sties no longer works. I've got a setup inspired by: geeking-out-with-haproxy-on-pfsense-the-ultimate with some HTTP sites, some HTTPS and some with client auth certificates checking enabled as well.

I've used this successfully together with ACME and Let's Encrypt for quite some time and it's been working perfectly, didn't seem to hard and I at least thought that I understood what was going on, but I guess not so much anymore..

Using curl on my Mac i can see that the redirect frontend works as expected, but then I get into some ssl issues..

curl -vL http://gitlab.itard.se
* Rebuilt URL to: http://gitlab.itard.se/
*   Trying 85.230.78.103...
* TCP_NODELAY set
* Connected to gitlab.itard.se (85.230.78.103) port 80 (#0)
> GET / HTTP/1.1
> Host: gitlab.itard.se
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-length: 0
< Location: https://gitlab.itard.se/
< 
* Connection #0 to host gitlab.itard.se left intact
* Issue another request to this URL: 'https://gitlab.itard.se/'
*   Trying 85.230.78.103...
* TCP_NODELAY set
* Connected to gitlab.itard.se (85.230.78.103) port 443 (#1)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/etc/openssl/cert.pem
  CApath: /usr/local/etc/openssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Closing connection 1
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

I've tried to play around with the config but I obviously don't really know what I'm doing and I've unable to get this working again.

The HAProxy stats page is all green btw.

I've also tried to get to the site using curl directly on the pfSense box, but this also fails.

curl -v -H 'Host: gitlab.itard.se' https://127.0.0.1:2043
* Rebuilt URL to: https://127.0.0.1:2043/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 2043 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:2043 
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:2043

Generated HAProxy config below:

# Automaticaly generated, dont edit manually.
# Generated on: 2018-10-06 22:33
global
	maxconn			2000
	log			/var/run/log	syslog	debug
	stats socket /tmp/haproxy.socket level admin 
	uid			80
	gid			80
	nbproc			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		HAproxyGWOC
	server-state-file /tmp/haproxy_server_state
	stats socket /var/run/haproxy.sock mode 600 level admin
	stats timeout 2m #Wait up to 2 minutes for input
	
	# Modern browser compatibility only as mentioned here:
	# https://wiki.mozilla.org/Security/Server_Side_TLS
	
	tune.ssl.default-dh-param 2048
	
	# Time-to-first-Byte (TTFB) value needs to be optimized based on
	# the actual public certificate chain see
	# https://www.igvita.com/2013/10/24
	# /optimizing-tls-record-size-and-buffering-latency/
	
	tune.ssl.maxrecord 1370

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats refresh 10
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

mailers globalmailers
	mailer smtp smtp.bredband.net:25

frontend WAN_HTTPS-merged
	bind			127.0.0.1:2043 name 127.0.0.1:2043  no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS.crt_list  accept-proxy npn http/1.1
	mode			http
	log			global
	option			httplog
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		7200000
	# Remove headers that expose security-sensitive information.
	rspidel ^Server:.*$
	rspidel ^X-Powered-By:.*$
	rspidel ^X-AspNet-Version:.*$
	
	# add some security related headers
	rspadd Content-Security-Policy:\ default-src\ https:\ data:\ \‘unsafe-inline\\’\ \\'unsafe-eval\'
	
	rspadd X-Frame-Options:\ SAMEORIGIN
	rspadd X-Content-Type-Options:\ nosniff
	rspadd X-Xss-Protection:\ 1;\ mode=block
	acl			smplex_acl	var(txn.txnhost) -m str -i smplex.itard.se
	acl			aclcrt_smplex	var(txn.txnhost) -m reg -i ^smplex\.itard\.se(:([0-9]){1,5})?$
	acl			gitlab_acl	var(txn.txnhost) -m str -i gitlab.itard.se
	acl			aclcrt_gitlab	var(txn.txnhost) -m reg -i ^gitlab\.itard\.se(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend smplex_ipvANY  if  smplex_acl aclcrt_smplex
	use_backend gitlab_ipvANY  if  gitlab_acl aclcrt_gitlab

frontend WAN_HTTPS_auth
	bind			127.0.0.1:2044 name 127.0.0.1:2044  no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS_auth.crt_list ca-file /var/etc/haproxy/clientca_WAN_HTTPS_auth.pem verify required  accept-proxy npn http/1.1
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		7200000
	# Remove headers that expose security-sensitive information.
	rspidel ^Server:.*$
	rspidel ^X-Powered-By:.*$
	rspidel ^X-AspNet-Version:.*$
	
	# add some security related headers
	rspadd Content-Security-Policy:\ default-src\ https:\ data:\ \‘unsafe-inline\\’\ \\'unsafe-eval\'
	
	rspadd X-Frame-Options:\ SAMEORIGIN
	rspadd X-Content-Type-Options:\ nosniff
	rspadd X-Xss-Protection:\ 1;\ mode=block
	acl			aclcrt_WAN_HTTPS_auth	var(txn.txnhost) -m reg -i ^([^\.]*)\.itard\.se(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)

frontend WAN_HTTP-merged
	bind			85.230.78.103:80 name 85.230.78.103:80   
	bind			10.1.1.2:80 name 10.1.1.2:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			not_redirect_acl	var(txn.txnpath) -m beg -i  /.well-known/acme-challenge/
	acl			not_redirect_acl	var(txn.txnpath) -m beg -i /haproxy/haproxy_stats.php
	acl			acme_acl	var(txn.txnpath) -m beg -i  /.well-known/acme-challenge/
	acl			haproxy_csv_acl	var(txn.txnpath) -m beg -i /haproxy/haproxy_stats.php
	http-request set-var(txn.txnpath) path
	use_backend ssl-redirect_ipvANY  if  !not_redirect_acl 
	use_backend gwoc_ipvANY  if  acme_acl 
	use_backend haproxy_csv_ipvANY  if  haproxy_csv_acl 

frontend WAN_443-merged
	bind			85.230.78.103:443 name 85.230.78.103:443   
	bind			10.1.1.2:443 name 10.1.1.2:443   
	mode			tcp
	log			global
	option			tcplog
	timeout client		30000
	tcp-request inspect-delay 5s
	tcp-request content accept if { req.ssl_hello_type 1 } or !{ req.ssl_hello_type 1 }
	acl			acl	req.ssl_sni -m str -i smplex.itard.se
	acl			acl	req.ssl_sni -m str -i plexrequests.itard.se
	acl			acl	req.ssl_sni -m str -i gitlab.itard.se
	acl			acl_req.ssl_hello_type	req.ssl_hello_type 1
	acl			acl	req.ssl_sni -m str -i gwoc.itard.se
	use_backend WAN_HTTPS_ipvANY  if  acl 
	default_backend WAN_HTTPS_ipvANY
	default_backend WAN_HTTPS_auth_ipvANY

backend smplex_ipvANY
	mode			http
	id			119
	log			global
	# use mailers
	# level  notice 
	email-alert mailers			globalmailers
	email-alert level			notice
	email-alert from			haproxy@gwoc.itard.se
	email-alert to			ivan@alpe.se
	email-alert myhostname			gwoc.itard.se
	timeout connect		30000
	timeout server		30000
	retries			3
	server			smplex 10.1.1.100:32400 id 120 check inter 5000  weight 1 

backend gitlab_ipvANY
	mode			http
	id			135
	log			global
	# use mailers
	# level  notice 
	email-alert mailers			globalmailers
	email-alert level			notice
	email-alert from			haproxy@gwoc.itard.se
	email-alert to			ivan@alpe.se
	email-alert myhostname			gwoc.itard.se
	timeout connect		30000
	timeout server		30000
	retries			3
	server			gitlab 10.1.1.108:80 id 136 check inter 5000  weight 1 

backend ssl-redirect_ipvANY
	mode			http
	id			121
	log			global
	# use mailers
	# level  notice 
	email-alert mailers			globalmailers
	email-alert level			notice
	email-alert from			haproxy@gwoc.itard.se
	email-alert to			ivan@alpe.se
	email-alert myhostname			gwoc.itard.se
	timeout connect		30000
	timeout server		30000
	retries			3
	redirect scheme https code 301

backend gwoc_ipvANY
	mode			http
	id			103
	log			global
	# use mailers
	# level  notice 
	email-alert mailers			globalmailers
	email-alert level			notice
	email-alert from			haproxy@gwoc.itard.se
	email-alert to			ivan@alpe.se
	email-alert myhostname			gwoc.itard.se
	http-response set-header Strict-Transport-Security max-age=60;
	rspirep ^(Set-Cookie:((?!;\ secure).)*)$ \1;\ secure if { ssl_fc }
	timeout connect		30000
	timeout server		30000
	retries			3
	server			gwoc 10.1.1.2:8080 id 104 check inter 5000  

backend haproxy_csv_ipvANY
	mode			http
	id			105
	log			global
	# use mailers
	# level  notice 
	email-alert mailers			globalmailers
	email-alert level			notice
	email-alert from			haproxy@gwoc.itard.se
	email-alert to			ivan@alpe.se
	email-alert myhostname			gwoc.itard.se
	timeout connect		30000
	timeout server		7200000
	retries			3
	server			haproxy_csv 127.0.0.1:2200 id 106 check inter 1000  

backend WAN_HTTPS_ipvANY
	mode			tcp
	id			125
	log			global
	# use mailers
	# level  notice 
	email-alert mailers			globalmailers
	email-alert level			notice
	email-alert from			haproxy@gwoc.itard.se
	email-alert to			ivan@alpe.se
	email-alert myhostname			gwoc.itard.se
	timeout connect		30000
	timeout server		7200000
	retries			3
	server			WAN_HTTPS 127.0.0.1:2043 id 126 ssl  verify none send-proxy 

backend WAN_HTTPS_auth_ipvANY
	mode			tcp
	id			127
	log			global
	# use mailers
	# level  notice 
	email-alert mailers			globalmailers
	email-alert level			notice
	email-alert from			haproxy@gwoc.itard.se
	email-alert to			ivan@alpe.se
	email-alert myhostname			gwoc.itard.se
	timeout connect		30000
	timeout server		7200000
	retries			3
	server			WAN_HTTPS_auth 127.0.0.1:2044 id 128 ssl  verify none send-proxy

Can anyone see anything incorrect with my setup?

/Ivan

[PiBa]

@ivaped
For the WAN_HTTPS and WAN_HTTPS_auth server, disable SSL-Encyption checkbox on the server, enable the SSL-Checks on the server?

[ivaped]

Hum, you're on to something here! The gitlab server does not use SSL in the background, so I shouldn't have it there but I disabled SSL encryption and enabled SSL checks on the WAN_HTTPS backend and that seems to have done the trick!

Sweet bananas, huge thanks!