HAProxy SSL mode help needed

veldthui

Having trouble getting SSl to work. Have the port 80 working fine and redirecting to ACME on pfSense or my Synology okay.

At present I am just trying to get it to forward to a web server (test bench) to get it working. eventually I want it to forward on to my exchange server but want the basics working first. Have watched and read quite a few tutorials but most of them use SSL offloading and turning on the Transparent Client IP which causes me grief.

For testing I have set up a virtual IP of 10.101.101.1:443 and have a NAT port forward rule to forward port 8843 to that so I don't have to lose my mail while testing.
This parts works as when I try to get to the webserver I get the 503 error message set up in HAProxy for the 443 front end.

Can anyone point me to a simple tutorial on a basic set up for SSL through HAProxy and pfSense.

veldthui

I have made the webserver the default and only have it in there.

The error message I am getting back is "The site cant provide a secure connection." ERR_SSL_PROTCOL_ERROR

I am only using a self signed cert for now on the web server as it is just a test bed to be deleted later.

PiBa

@veldthui
What does haproxy.cfg look like now?

veldthui

Config is below. Reading up on the error it is pointing at the web server but using Chrome on the local network it displays okay except for the certificate warning due to being self signed.
The 10.101.101.1 is a virtual IP which I forward 8843 to so my mail still works on 443 while testing. I will change it once working.

# Automaticaly generated, dont edit manually.
# Generated on: 2019-04-21 15:33
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:4444 name localstats
	mode http
	stats enable
	stats refresh 10
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend ACME-JV-NET-NZ-PROD
	bind			Ext IP:80 name Ext IP:80   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	errorfile			503 /var/etc/haproxy/errorfile_ACME-JV-NET-NZ-PROD_503_ExampleErrorfile
	acl			JVNAS1	var(txn.txnhost) -m beg -i jvnas1
	acl			ACME	var(txn.txnpath) -m beg -i /.well-known/acme-challenge/
	http-request set-var(txn.txnhost) hdr(host)
	http-request set-var(txn.txnpath) path
	use_backend JVNAS1-LE_ipvANY  if  JVNAS1 
	use_backend ACME-JV-NET-NZ-PROD_ipvANY  if  ACME 

frontend HTTPS_FRONTEND
	bind			10.101.101.1:443 name 10.101.101.1:443   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	errorfile			503 /var/etc/haproxy/errorfile_HTTPS_FRONTEND_503_ExampleErrorfile
	acl			WEBSERVER	var(txn.txnhost) -m beg -i webserver
	http-request set-var(txn.txnhost) hdr(host)
	use_backend WEBSERVER_ipvANY  if  WEBSERVER 
	default_backend WEBSERVER_ipvANY

backend JVNAS1-LE_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			JVNAS1-BE 192.168.0.30:80 id 103 check inter 1000  

backend ACME-JV-NET-NZ-PROD_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			ACME-BACKEND 127.0.0.1:4002 id 101  

backend WEBSERVER_ipvANY
	mode			http
	id			108
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			WEBSERVER 192.168.0.6:443 id 109 check inter 1000

PiBa

@veldthui said in HAProxy SSL mode help needed:

frontend HTTPS_FRONTEND
bind 10.101.101.1:443 name 10.101.101.1:443
mode http

Its not possible to handle SSL traffic without offloading with 'mode http'.. Haproxy will try to 'understand' the http request, while a ssl handshake is being performed.. So this wont work..

Either add certificates and offloading to the haproxy frontend, or use ssl/tcp mode and use SNI for the webserver selection.. As the Host-header is not available when passing SSL along as-is.

veldthui

Okay, If I use the offloading do I need a SAN cert that will cover each machine that is going to passed through? Could I use a wildcard cert instead? Really need the path for the Exchange server bits so I guess offloading will be it?

I tried the ssl/tcp mode and for my web server it now replies okay but my exchange server still replies saying it can't provide a secure connection.

PiBa

@veldthui
If you need the 'path' for the haproxy acl's, then you must use offloading for the ssl traffic.

Certificates
As for the options with the certificates to be used on haproxy there are several options:

Wildcard-certificate
Certficate with multiple SAN's
Multiple seperate Certificates

These can also be combined like having both multiple certificates each for a subset of the domain-names that clients will use.

Exchange
As for the 'exchange secure connection' i am not sure what that one means exactly.. Note though that SMTP traffic usually does not use regular SSL connection but uses 'STARTTLS' which is something that haproxy doesnt understand either. In which case regular tcp mode is the only option if SMTP traffic has to pass through haproxy.
Are you trying to send a email with smtp? Or trying to visit the outlook-web-access website with a browser?

veldthui

@PiBa SMTP goes through port 25 which is NAT forwarded to the server and is only used for that so doesn't need to go through HAProxy.

No what I am after is web stuff. Just trying Outlook Web Access (OWA) for a start but will need activesync, ECP, OWA, and a couple of others. They are all on 443 through IIS on the exchange server so are basically just web pages but aren't co-operating as yet.

PiBa

@veldthui
So whats the current config like?

Frontend on :443 that does offloading with a certificate (and shows as such in the haproxy.conf)
And a Backend server that also uses :443 and re-encrypts the traffic between haproxy and IIS?

Something like this?: (i wrote it by hand.. so probably several mistakes there but the 'crt' and 'ssl' should be there..):

frontend HTTPS_FRONTEND
  bind :443 crt /var/haproxy/HTTPS_FRONTEND.crt
  mode http
  default_backend exchange 
backend exchange
  mode http
  server exchange 192.168.x.y:443 ssl verify none

If it aint working and it looks similar to above, what does it look like exactly?

veldthui

@PiBa At present using the ssl/tcp mode as I do not have any certs setup on HAProxy except for the one for pfsense itself using ACME.

PiBa

@veldthui Can you share the current config?

veldthui

Looks like this at present. webserver which is a CentOS apache webserver works. Exchange server which is on Windows 2016 IIS does not.

# Automaticaly generated, dont edit manually.
# Generated on: 2019-04-23 17:52
global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:4444 name localstats
	mode http
	stats enable
	stats refresh 10
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend ACME-JV-NET-NZ-PROD
	bind			Ext IP:80 name Ext IP:80   
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	errorfile			503 /var/etc/haproxy/errorfile_ACME-JV-NET-NZ-PROD_503_ExampleErrorfile
	acl			JVNAS1	var(txn.txnhost) -m beg -i jvnas1
	acl			ACME	var(txn.txnpath) -m beg -i /.well-known/acme-challenge/
	http-request set-var(txn.txnhost) hdr(host)
	http-request set-var(txn.txnpath) path
	use_backend JVNAS1-LE_ipvANY  if  JVNAS1 
	use_backend ACME-JV-NET-NZ-PROD_ipvANY  if  ACME 

frontend HTTPS_FRONTEND
	bind			10.101.101.1:443 name 10.101.101.1:443   
	mode			tcp
	log			global
	timeout client		30000
	errorfile			503 /var/etc/haproxy/errorfile_HTTPS_FRONTEND_503_ExampleErrorfile
	tcp-request inspect-delay	5s
	acl			WEBSERVER	req.ssl_sni -m beg -i webserver
	acl			MAILSERVER	req.ssl_sni -m beg -i jvnet
	tcp-request content accept if { req.ssl_hello_type 1 }
	use_backend WEBSERVER_ipvANY  if  WEBSERVER 
	use_backend MAILSERVER_ipvANY  if  MAILSERVER 

backend JVNAS1-LE_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			JVNAS1-BE 192.168.0.30:80 id 103 check inter 1000  

backend ACME-JV-NET-NZ-PROD_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			ACME-BACKEND 127.0.0.1:4002 id 101  

backend WEBSERVER_ipvANY
	mode			tcp
	id			108
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			WEBSERVER 192.168.0.6:443 id 109 check inter 1000  

backend MAILSERVER_ipvANY
	mode			tcp
	id			106
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3

PiBa

@veldthui
I presume the "backend MAILSERVER_ipvANY" also has a server similar to the 'server WEBSERVER' with the only difference being a different ip & id ? If so then i would think it 'should work'..

veldthui

@PiBa said in HAProxy SSL mode help needed:

@veldthui
I presume the "backend MAILSERVER_ipvANY" also has a server similar to the 'server WEBSERVER' with the only difference being a different ip & id ? If so then i would think it 'should work'..

It is not the same. One is an Apache webserver. The one on MAILSERVER is IIS running on Server 2016. It should still just present a web page and if I go via a port redirect to it, it responds okay and gives me the outlook web mail.
Using HAProxy I either get a cannot provide a secure connection or web site did not respond. To get to the mail via HAProxy setup I have at present I use. https://jvnet.xxx.net:8843/owa. This gives me the error.
The mail server has a self signed cert which has a SAN cert for the supplied FQDN.

I might do some more google searches for exchange/haproxy issues and see what comes up. May even set up a seperate Server 2016 IIS with just a web site on it to see if that works or has issues.

veldthui

Well I am not sure what has happened but I came home from work and got another IIS web server up and running on a VM and was going to test it but thought I would test the mail server one more time before changing it and suddenly it appears to be working. Will need to test it from work to be 100% sure as using my phone to get outside my local network and come in through pfSense WAN rather than the LAN.

If it is working I can now test the activesync for my phone through HAProxy to the server and if that works it is all go.

Fingers crossed but not back at work until tomorrow night.

veldthui

Righto tonight I switched the port forward for 443 off and pointed 443 at HAProxy.

Tested Outlook Web Mail from a remote computer and worked perfectly. Tried my iPhone using the mail app and activesync and that also worked perfectly.

Tried my two web sites with IE and apart from the certificate errors due to be self signed that worked as well.

Tried with Chrome and getting an error of ERR_SLL_CERT_BAD_FORMAT.

Now this is strange because Chrome works fine if I redirect through port 8843 and a Virtual IP.

Any ideas? Maybe a proper signed cert will fix this but not sure why it works one way and not the other.

Certainly getting there though

PiBa

@veldthui
Nice that is working (mostly), i dont have any good idea about the reason for the ERR_SLL_CERT_BAD_FORMAT to appear in chrome.. Maybe the cert was created with duplicate serialnumbers of a other self created cert, or signed with a to weak fingerprint or something... Try clearing chrome's cache, and search for some details perhaps on the developer-window security and console tabs.. maybe either of them tells something more detailed? If other browsers work there is little reason to assume the problem is with the haproxy config itself.

veldthui

Okay have put in the real certs and all is working 100%. Very nice.

Now question. One site will have a wordpress site on which requires FTP for the updates. Can HAProxy be used to redirect port 21 the same was or am I stuck with a NAT Port redirection and limited to 1 FTP server on one machine?

PiBa

@veldthui
Haproxy does not 'understand' FTP protocol..
But you might be able to do something with 'FTPS' where the ftp connection is wrapped inside SSL, and haproxy might be able to use a SNI header if the ftp client sets that... Really guessing/hoping there bigtime though.. If thats not gonna fly then i don't think haproxy will be able to help you out here. For sure its not intended for this that is for sure.

Good the http/https part works nicely now :).