HAproxy HTTPS + Openvpn

Actionhenk · Oct 11, 2019, 5:00 PM

Hi,

Im trying to set up haproxy in a way so I can use openvpn and https on 1 public ip on port 443. Im a bit stuck at the point where im at now.

I am using a tcp mode main frontend with a default backend which goes to the openvpn server. This works without issues, but my website on https is no longer working.. I found a few articles on google using "req.ssl_sni -i ..." and with this im able to get my website working again over https ... to get it working I needed to turn on SSL offloading on the main frontend which I dont want because then open vpn wont work anymore. So im stuck at a working website, no vpn or a working vpn and no website.

I was thinking of setting it up next using a tcp mode frontend -> to a backend if a sni header matches -> from the backend back to a http frontend listening on 127.0.0.1:<randomport> -> from the http frontend to the website. This didnt work either.

Below is my current configuration (all my backend websites are running on port 443 with a valid cert, maybe I need it to be non-ssl port 80? )

Im hoping someone can point me in the right direction

# Automaticaly generated, dont edit manually.
# Generated on: 2019-10-11 18:26
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin 
	gid			80
	nbproc			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	4096
	log-send-hostname		haproxy-node
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

## trying to set the mode to HTTP
frontend Frontend-loopback
	bind			127.0.0.1:5555 name 127.0.0.1:5555   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	http-request del-header X-Forwarded-Proto 
	http-request set-header X-Forwarded-Proto "\ https if { ssl_fc }" 
	default_backend bw-test_ipvANY

## main incoming frontend for openvpn and HTTPS
frontend Main
	bind			192.168.2.252:443 name 192.168.2.252:443   
	mode			tcp
	log			global
	timeout client		30000
	tcp-request content accept if HTTP
	tcp-request content accept if { req.ssl_hello_type 1 }
	acl			Acl1	req.ssl_sni -i .domain.com
	use_backend Backend-to-front-loopback_ipvANY  if  Acl1 
	default_backend v.domain.com_ipvANY

## the website im trying to go to from the "Frontend-loopback"
backend bw-test_ipvANY
	mode			http
	id			110
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			bw 172.16.252.9:443 id 101 ssl  verify none crt /var/etc/haproxy/server_clientcert_5d9e36a600d17.pem 

## backend to go back to HTTP frontend
backend Backend-to-front-loopback_ipvANY
	mode			tcp
	id			111
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			Bk-to-front 127.0.0.1:5555 id 112  

## open vpn server on pfsense
backend v.domain.com_ipvANY
	mode			tcp
	id			108
	log			global
	timeout connect		30000
	timeout server		30000
	retries			2
	server			v.domain.com 192.168.2.252:6785 id 109

Thanks!

PiBa · Oct 11, 2019, 8:12 PM

@Actionhenk said in HAproxy HTTPS + Openvpn:

frontend Frontend-loopback
bind 127.0.0.1:5555 name 127.0.0.1:5555

Can you enable ssl-offloading on this loopback frontend?

Actionhenk · Oct 11, 2019, 8:40 PM

Thanks for your reply. I had already tried that but unfortunately it wasnt working. I changed a few things and ended up with a working solution.

I made a change in the main front end, I was using req.ssl_sni in the front end to filter domain where I should have been using req.ssl_hello_type 1

I then set the action to go to a backend and put all my sites in a single backend using req.ssl_sni on the backend to filter for sites. Then forwarding it back to individual frontends for each site. Not sure if it is best practice but for now its OK :))

# Automaticaly generated, dont edit manually.
# Generated on: 2019-10-11 22:33
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin 
	gid			80
	nbproc			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	4096
	log-send-hostname		haproxy-node
	server-state-file /tmp/haproxy_server_state
	ssl-default-bind-options no-sslv3 no-tls-tickets no-tlsv10 no-tlsv11 no-tlsv12
	ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
	

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend frontend-ssl
	bind			192.168.2.252:443 name 192.168.2.252:443   
	mode			tcp
	log			global
	timeout client		30000
	tcp-request inspect-delay 5s
	tcp-request content accept if HTTP
	acl			acl1	req.ssl_hello_type 1
	use_backend main-ssl_ipvANY  if  acl1 
	default_backend openvpn_ipvANY

frontend main-https-zim-mail
	bind			127.0.0.1:4646 name 127.0.0.1:4646   ssl crt-list /var/etc/haproxy/main-https-zim-mail.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			zim-mail	var(txn.txnhost) -m str -i zim-mail.domain.com
	acl			aclcrt_main-https-zim-mail	var(txn.txnhost) -m reg -i ^zim-mail\.domain\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend backend-zim-mail.domain.com_ipvANY  if  zim-mail aclcrt_main-https-zim-mail

frontend main-https-eas
	bind			127.0.0.1:5050 name 127.0.0.1:5050   ssl crt-list /var/etc/haproxy/main-https-eas.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			eas	var(txn.txnhost) -m str -i eas.domain.com
	acl			aclcrt_main-https-eas	var(txn.txnhost) -m reg -i ^eas\.domain\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend backend-eas.domain.com_ipvANY  if  eas aclcrt_main-https-eas

frontend main-https-bw
	bind			127.0.0.1:4747 name 127.0.0.1:4747   ssl crt-list /var/etc/haproxy/main-https-bw.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			bw	var(txn.txnhost) -m str -i bw.domain.com
	acl			aclcrt_main-https-bw	var(txn.txnhost) -m reg -i ^bw\.domain\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend backend-bw.domain.com_ipvANY  if  bw aclcrt_main-https-bw

frontend main-https-domain
	bind			127.0.0.1:4848 name 127.0.0.1:4848   ssl crt-list /var/etc/haproxy/main-https-domain.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			domain	var(txn.txnhost) -m str -i domain.com
	acl			aclcrt_main-https-domain	var(txn.txnhost) -m reg -i ^domain\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend backend-domain.com_ipvANY  if  domain aclcrt_main-https-domain

frontend main-https-www
	bind			127.0.0.1:4949 name 127.0.0.1:4949   ssl crt-list /var/etc/haproxy/main-https-www.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			eas	var(txn.txnhost) -m str -i eas.domain.com
	acl			aclcrt_main-https-www	var(txn.txnhost) -m reg -i ^www\.domain\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)
	use_backend backend-eas.domain.com_ipvANY  if  eas aclcrt_main-https-www

backend main-ssl_ipvANY
	mode			tcp
	id			110
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	acl			zim-mail	req.ssl_sni -i zim-mail.domain.com
	acl			bw	req.ssl_sni -i bw.domain.com
	acl			domain	req.ssl_sni -i domain.com
	acl			www	req.ssl_sni -i www.domain.com
	acl			eas	req.ssl_sni -i eas.domain.com
	use-server zim-mail  if  zim-mail 
	use-server bw  if  bw 
	use-server domain  if  domain 
	use-server www  if  www 
	use-server eas  if  eas 
	server			zim-mail 127.0.0.1:4646 id 123  
	server			bw 127.0.0.1:4747 id 124  
	server			domain 127.0.0.1:4848 id 125  
	server			www 127.0.0.1:4949 id 126  
	server			eas 127.0.0.1:5050 id 111  

backend openvpn_ipvANY
	mode			tcp
	id			112
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			openvpn 192.168.2.252:6785 id 113  

backend backend-zim-mail.domain.com_ipvANY
	mode			http
	id			116
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			backend-zim-mail.domain.com 172.16.252.7:443 id 117 ssl  verify none 

backend backend-eas.domain.com_ipvANY
	mode			http
	id			114
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			backend-eas.domain.com 172.16.252.8:443 id 115 ssl  verify none 

backend backend-bw.domain.com_ipvANY
	mode			http
	id			118
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			backend-bw.domain.com 172.16.252.9:443 id 119 ssl  verify none 

backend backend-domain.com_ipvANY
	mode			http
	id			122
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			backend-domain.com 172.16.252.12:443 id 121

PiBa · Oct 11, 2019, 10:09 PM

@Actionhenk
That looks overly complicated.. I do see that the 'loopback' kinda frontends now are using offloading as suggested. Also i see that the bw server is no longer using a client-certificate.. Perhaps that is the part that made the original configuration fail while you where testing/changing it.?.

Also you are using SNI to switch between the 5 webservers, in general its better to do that on the Host header. Are there 5 different certificates used for the 5 websites? Otherwise switching to a different domain could fail if the browser decides it can keep the connection alive with the same certificate..

Actionhenk · Oct 12, 2019, 7:58 AM

@Actionhenk said in HAproxy HTTPS + Openvpn:

req.ssl_sni -i zim-mail.domain.com

Thanks for the suggestions (didnt notice bw was missing a certificate, checked in the gui and it is there), I have tried to change it to match host headers but haproxy shows me an error saving, " error detected while parsing switching rule : no such ACL : 'acl-eas'" The error somehow doesnt show if I have a "req.ssl_sni -i www.domain.com" in the acl.

Maybe something different, do you know if it is possible to deny connections from an external source ip in tcp mode ? There are options using "Source IP matches IP or Alias" but its not matching the external ip's, send-proxy is only picking up internal ips. Is it possible to send to, or get originating IP through to the backends from a tcp frontend ?

Thanks!

PiBa · Oct 15, 2019, 11:23 PM

Hard to tell what go's wrong without the haproxy.cfg content that you might have. When using proxy-protocol between backend and next frontend the 'src' should still match the external client ip's afaik.. as for acl's some automatically disaprear depending on the mode chosen.. But when traffic is offloaded the 'ssl_fc_sni' returns the sni value. Not the 'ssl_sni' which is only used with tcp-passthrough.. Such acl's are also automatically hidden in the webgui on a frontend which knows the mode its going to run with better..