HAProxy https no longer works



  • Hi,

    After the latest HAProxy upgrade (No sure from what version I upgraded from though..) my HTTPS sties no longer works. I've got a setup inspired by: geeking-out-with-haproxy-on-pfsense-the-ultimate with some HTTP sites, some HTTPS and some with client auth certificates checking enabled as well.

    I've used this successfully together with ACME and Let's Encrypt for quite some time and it's been working perfectly, didn't seem to hard and I at least thought that I understood what was going on, but I guess not so much anymore..

    Using curl on my Mac i can see that the redirect frontend works as expected, but then I get into some ssl issues..

    curl -vL http://gitlab.itard.se
    * Rebuilt URL to: http://gitlab.itard.se/
    *   Trying 85.230.78.103...
    * TCP_NODELAY set
    * Connected to gitlab.itard.se (85.230.78.103) port 80 (#0)
    > GET / HTTP/1.1
    > Host: gitlab.itard.se
    > User-Agent: curl/7.61.1
    > Accept: */*
    > 
    < HTTP/1.1 301 Moved Permanently
    < Content-length: 0
    < Location: https://gitlab.itard.se/
    < 
    * Connection #0 to host gitlab.itard.se left intact
    * Issue another request to this URL: 'https://gitlab.itard.se/'
    *   Trying 85.230.78.103...
    * TCP_NODELAY set
    * Connected to gitlab.itard.se (85.230.78.103) port 443 (#1)
    * ALPN, offering http/1.1
    * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
    * successfully set certificate verify locations:
    *   CAfile: /usr/local/etc/openssl/cert.pem
      CApath: /usr/local/etc/openssl/certs
    * TLSv1.2 (OUT), TLS header, Certificate Status (22):
    * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    * error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
    * Closing connection 1
    curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
    

    I've tried to play around with the config but I obviously don't really know what I'm doing and I've unable to get this working again.

    The HAProxy stats page is all green btw.

    I've also tried to get to the site using curl directly on the pfSense box, but this also fails.

    curl -v -H 'Host: gitlab.itard.se' https://127.0.0.1:2043
    * Rebuilt URL to: https://127.0.0.1:2043/
    *   Trying 127.0.0.1...
    * TCP_NODELAY set
    * Connected to 127.0.0.1 (127.0.0.1) port 2043 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
    * successfully set certificate verify locations:
    *   CAfile: /usr/local/share/certs/ca-root-nss.crt
      CApath: none
    * TLSv1.2 (OUT), TLS header, Certificate Status (22):
    * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:2043 
    curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:2043
    

    Generated HAProxy config below:

    # Automaticaly generated, dont edit manually.
    # Generated on: 2018-10-06 22:33
    global
    	maxconn			2000
    	log			/var/run/log	syslog	debug
    	stats socket /tmp/haproxy.socket level admin 
    	uid			80
    	gid			80
    	nbproc			1
    	hard-stop-after		15m
    	chroot				/tmp/haproxy_chroot
    	daemon
    	tune.ssl.default-dh-param	2048
    	log-send-hostname		HAproxyGWOC
    	server-state-file /tmp/haproxy_server_state
    	stats socket /var/run/haproxy.sock mode 600 level admin
    	stats timeout 2m #Wait up to 2 minutes for input
    	
    	# Modern browser compatibility only as mentioned here:
    	# https://wiki.mozilla.org/Security/Server_Side_TLS
    	
    	tune.ssl.default-dh-param 2048
    	
    	# Time-to-first-Byte (TTFB) value needs to be optimized based on
    	# the actual public certificate chain see
    	# https://www.igvita.com/2013/10/24
    	# /optimizing-tls-record-size-and-buffering-latency/
    	
    	tune.ssl.maxrecord 1370
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:2200 name localstats
    	mode http
    	stats enable
    	stats refresh 10
    	stats admin if TRUE
    	stats show-legends
    	stats uri /haproxy/haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    mailers globalmailers
    	mailer smtp smtp.bredband.net:25
    
    frontend WAN_HTTPS-merged
    	bind			127.0.0.1:2043 name 127.0.0.1:2043  no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS.crt_list  accept-proxy npn http/1.1
    	mode			http
    	log			global
    	option			httplog
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		7200000
    	# Remove headers that expose security-sensitive information.
    	rspidel ^Server:.*$
    	rspidel ^X-Powered-By:.*$
    	rspidel ^X-AspNet-Version:.*$
    	
    	# add some security related headers
    	rspadd Content-Security-Policy:\ default-src\ https:\ data:\ \‘unsafe-inline\\’\ \\'unsafe-eval\'
    	
    	rspadd X-Frame-Options:\ SAMEORIGIN
    	rspadd X-Content-Type-Options:\ nosniff
    	rspadd X-Xss-Protection:\ 1;\ mode=block
    	acl			smplex_acl	var(txn.txnhost) -m str -i smplex.itard.se
    	acl			aclcrt_smplex	var(txn.txnhost) -m reg -i ^smplex\.itard\.se(:([0-9]){1,5})?$
    	acl			gitlab_acl	var(txn.txnhost) -m str -i gitlab.itard.se
    	acl			aclcrt_gitlab	var(txn.txnhost) -m reg -i ^gitlab\.itard\.se(:([0-9]){1,5})?$
    	http-request set-var(txn.txnhost) hdr(host)
    	use_backend smplex_ipvANY  if  smplex_acl aclcrt_smplex
    	use_backend gitlab_ipvANY  if  gitlab_acl aclcrt_gitlab
    
    frontend WAN_HTTPS_auth
    	bind			127.0.0.1:2044 name 127.0.0.1:2044  no-sslv3 ssl crt-list /var/etc/haproxy/WAN_HTTPS_auth.crt_list ca-file /var/etc/haproxy/clientca_WAN_HTTPS_auth.pem verify required  accept-proxy npn http/1.1
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		7200000
    	# Remove headers that expose security-sensitive information.
    	rspidel ^Server:.*$
    	rspidel ^X-Powered-By:.*$
    	rspidel ^X-AspNet-Version:.*$
    	
    	# add some security related headers
    	rspadd Content-Security-Policy:\ default-src\ https:\ data:\ \‘unsafe-inline\\’\ \\'unsafe-eval\'
    	
    	rspadd X-Frame-Options:\ SAMEORIGIN
    	rspadd X-Content-Type-Options:\ nosniff
    	rspadd X-Xss-Protection:\ 1;\ mode=block
    	acl			aclcrt_WAN_HTTPS_auth	var(txn.txnhost) -m reg -i ^([^\.]*)\.itard\.se(:([0-9]){1,5})?$
    	http-request set-var(txn.txnhost) hdr(host)
    
    frontend WAN_HTTP-merged
    	bind			85.230.78.103:80 name 85.230.78.103:80   
    	bind			10.1.1.2:80 name 10.1.1.2:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	timeout client		30000
    	acl			not_redirect_acl	var(txn.txnpath) -m beg -i  /.well-known/acme-challenge/
    	acl			not_redirect_acl	var(txn.txnpath) -m beg -i /haproxy/haproxy_stats.php
    	acl			acme_acl	var(txn.txnpath) -m beg -i  /.well-known/acme-challenge/
    	acl			haproxy_csv_acl	var(txn.txnpath) -m beg -i /haproxy/haproxy_stats.php
    	http-request set-var(txn.txnpath) path
    	use_backend ssl-redirect_ipvANY  if  !not_redirect_acl 
    	use_backend gwoc_ipvANY  if  acme_acl 
    	use_backend haproxy_csv_ipvANY  if  haproxy_csv_acl 
    
    frontend WAN_443-merged
    	bind			85.230.78.103:443 name 85.230.78.103:443   
    	bind			10.1.1.2:443 name 10.1.1.2:443   
    	mode			tcp
    	log			global
    	option			tcplog
    	timeout client		30000
    	tcp-request inspect-delay 5s
    	tcp-request content accept if { req.ssl_hello_type 1 } or !{ req.ssl_hello_type 1 }
    	acl			acl	req.ssl_sni -m str -i smplex.itard.se
    	acl			acl	req.ssl_sni -m str -i plexrequests.itard.se
    	acl			acl	req.ssl_sni -m str -i gitlab.itard.se
    	acl			acl_req.ssl_hello_type	req.ssl_hello_type 1
    	acl			acl	req.ssl_sni -m str -i gwoc.itard.se
    	use_backend WAN_HTTPS_ipvANY  if  acl 
    	default_backend WAN_HTTPS_ipvANY
    	default_backend WAN_HTTPS_auth_ipvANY
    
    backend smplex_ipvANY
    	mode			http
    	id			119
    	log			global
    	# use mailers
    	# level  notice 
    	email-alert mailers			globalmailers
    	email-alert level			notice
    	email-alert from			haproxy@gwoc.itard.se
    	email-alert to			ivan@alpe.se
    	email-alert myhostname			gwoc.itard.se
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			smplex 10.1.1.100:32400 id 120 check inter 5000  weight 1 
    
    backend gitlab_ipvANY
    	mode			http
    	id			135
    	log			global
    	# use mailers
    	# level  notice 
    	email-alert mailers			globalmailers
    	email-alert level			notice
    	email-alert from			haproxy@gwoc.itard.se
    	email-alert to			ivan@alpe.se
    	email-alert myhostname			gwoc.itard.se
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			gitlab 10.1.1.108:80 id 136 check inter 5000  weight 1 
    
    backend ssl-redirect_ipvANY
    	mode			http
    	id			121
    	log			global
    	# use mailers
    	# level  notice 
    	email-alert mailers			globalmailers
    	email-alert level			notice
    	email-alert from			haproxy@gwoc.itard.se
    	email-alert to			ivan@alpe.se
    	email-alert myhostname			gwoc.itard.se
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	redirect scheme https code 301
    
    backend gwoc_ipvANY
    	mode			http
    	id			103
    	log			global
    	# use mailers
    	# level  notice 
    	email-alert mailers			globalmailers
    	email-alert level			notice
    	email-alert from			haproxy@gwoc.itard.se
    	email-alert to			ivan@alpe.se
    	email-alert myhostname			gwoc.itard.se
    	http-response set-header Strict-Transport-Security max-age=60;
    	rspirep ^(Set-Cookie:((?!;\ secure).)*)$ \1;\ secure if { ssl_fc }
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			gwoc 10.1.1.2:8080 id 104 check inter 5000  
    
    backend haproxy_csv_ipvANY
    	mode			http
    	id			105
    	log			global
    	# use mailers
    	# level  notice 
    	email-alert mailers			globalmailers
    	email-alert level			notice
    	email-alert from			haproxy@gwoc.itard.se
    	email-alert to			ivan@alpe.se
    	email-alert myhostname			gwoc.itard.se
    	timeout connect		30000
    	timeout server		7200000
    	retries			3
    	server			haproxy_csv 127.0.0.1:2200 id 106 check inter 1000  
    
    backend WAN_HTTPS_ipvANY
    	mode			tcp
    	id			125
    	log			global
    	# use mailers
    	# level  notice 
    	email-alert mailers			globalmailers
    	email-alert level			notice
    	email-alert from			haproxy@gwoc.itard.se
    	email-alert to			ivan@alpe.se
    	email-alert myhostname			gwoc.itard.se
    	timeout connect		30000
    	timeout server		7200000
    	retries			3
    	server			WAN_HTTPS 127.0.0.1:2043 id 126 ssl  verify none send-proxy 
    
    backend WAN_HTTPS_auth_ipvANY
    	mode			tcp
    	id			127
    	log			global
    	# use mailers
    	# level  notice 
    	email-alert mailers			globalmailers
    	email-alert level			notice
    	email-alert from			haproxy@gwoc.itard.se
    	email-alert to			ivan@alpe.se
    	email-alert myhostname			gwoc.itard.se
    	timeout connect		30000
    	timeout server		7200000
    	retries			3
    	server			WAN_HTTPS_auth 127.0.0.1:2044 id 128 ssl  verify none send-proxy
    

    Can anyone see anything incorrect with my setup?

    /Ivan



  • @ivaped
    For the WAN_HTTPS and WAN_HTTPS_auth server, disable SSL-Encyption checkbox on the server, enable the SSL-Checks on the server?



  • Hum, you're on to something here! The gitlab server does not use SSL in the background, so I shouldn't have it there but I disabled SSL encryption and enabled SSL checks on the WAN_HTTPS backend and that seems to have done the trick!

    Sweet bananas, huge thanks!