HAProxy SSL mode help needed



  • Having trouble getting SSl to work. Have the port 80 working fine and redirecting to ACME on pfSense or my Synology okay.

    At present I am just trying to get it to forward to a web server (test bench) to get it working. eventually I want it to forward on to my exchange server but want the basics working first. Have watched and read quite a few tutorials but most of them use SSL offloading and turning on the Transparent Client IP which causes me grief.

    For testing I have set up a virtual IP of 10.101.101.1:443 and have a NAT port forward rule to forward port 8843 to that so I don't have to lose my mail while testing.
    This parts works as when I try to get to the webserver I get the 503 error message set up in HAProxy for the 443 front end.

    Can anyone point me to a simple tutorial on a basic set up for SSL through HAProxy and pfSense.



  • I have made the webserver the default and only have it in there.

    The error message I am getting back is "The site cant provide a secure connection." ERR_SSL_PROTCOL_ERROR

    I am only using a self signed cert for now on the web server as it is just a test bed to be deleted later.



  • @veldthui
    What does haproxy.cfg look like now?



  • Config is below. Reading up on the error it is pointing at the web server but using Chrome on the local network it displays okay except for the certificate warning due to being self signed.
    The 10.101.101.1 is a virtual IP which I forward 8843 to so my mail still works on 443 while testing. I will change it once working.

    # Automaticaly generated, dont edit manually.
    # Generated on: 2019-04-21 15:33
    global
    	maxconn			1000
    	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
    	uid			80
    	gid			80
    	nbproc			1
    	nbthread			1
    	hard-stop-after		15m
    	chroot				/tmp/haproxy_chroot
    	daemon
    	tune.ssl.default-dh-param	2048
    	server-state-file /tmp/haproxy_server_state
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:4444 name localstats
    	mode http
    	stats enable
    	stats refresh 10
    	stats admin if TRUE
    	stats show-legends
    	stats uri /haproxy/haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend ACME-JV-NET-NZ-PROD
    	bind			Ext IP:80 name Ext IP:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	errorfile			503 /var/etc/haproxy/errorfile_ACME-JV-NET-NZ-PROD_503_ExampleErrorfile
    	acl			JVNAS1	var(txn.txnhost) -m beg -i jvnas1
    	acl			ACME	var(txn.txnpath) -m beg -i /.well-known/acme-challenge/
    	http-request set-var(txn.txnhost) hdr(host)
    	http-request set-var(txn.txnpath) path
    	use_backend JVNAS1-LE_ipvANY  if  JVNAS1 
    	use_backend ACME-JV-NET-NZ-PROD_ipvANY  if  ACME 
    
    frontend HTTPS_FRONTEND
    	bind			10.101.101.1:443 name 10.101.101.1:443   
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	errorfile			503 /var/etc/haproxy/errorfile_HTTPS_FRONTEND_503_ExampleErrorfile
    	acl			WEBSERVER	var(txn.txnhost) -m beg -i webserver
    	http-request set-var(txn.txnhost) hdr(host)
    	use_backend WEBSERVER_ipvANY  if  WEBSERVER 
    	default_backend WEBSERVER_ipvANY
    
    backend JVNAS1-LE_ipvANY
    	mode			http
    	id			102
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			JVNAS1-BE 192.168.0.30:80 id 103 check inter 1000  
    
    backend ACME-JV-NET-NZ-PROD_ipvANY
    	mode			http
    	id			100
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			ACME-BACKEND 127.0.0.1:4002 id 101  
    
    backend WEBSERVER_ipvANY
    	mode			http
    	id			108
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			WEBSERVER 192.168.0.6:443 id 109 check inter 1000
    


  • @veldthui said in HAProxy SSL mode help needed:

    frontend HTTPS_FRONTEND
    bind 10.101.101.1:443 name 10.101.101.1:443
    mode http

    Its not possible to handle SSL traffic without offloading with 'mode http'.. Haproxy will try to 'understand' the http request, while a ssl handshake is being performed.. So this wont work..

    Either add certificates and offloading to the haproxy frontend, or use ssl/tcp mode and use SNI for the webserver selection.. As the Host-header is not available when passing SSL along as-is.



  • Okay, If I use the offloading do I need a SAN cert that will cover each machine that is going to passed through? Could I use a wildcard cert instead? Really need the path for the Exchange server bits so I guess offloading will be it?

    I tried the ssl/tcp mode and for my web server it now replies okay but my exchange server still replies saying it can't provide a secure connection.



  • @veldthui
    If you need the 'path' for the haproxy acl's, then you must use offloading for the ssl traffic.

    Certificates
    As for the options with the certificates to be used on haproxy there are several options:

    • Wildcard-certificate
    • Certficate with multiple SAN's
    • Multiple seperate Certificates

    These can also be combined like having both multiple certificates each for a subset of the domain-names that clients will use.

    Exchange
    As for the 'exchange secure connection' i am not sure what that one means exactly.. Note though that SMTP traffic usually does not use regular SSL connection but uses 'STARTTLS' which is something that haproxy doesnt understand either. In which case regular tcp mode is the only option if SMTP traffic has to pass through haproxy.
    Are you trying to send a email with smtp? Or trying to visit the outlook-web-access website with a browser?



  • @PiBa SMTP goes through port 25 which is NAT forwarded to the server and is only used for that so doesn't need to go through HAProxy.

    No what I am after is web stuff. Just trying Outlook Web Access (OWA) for a start but will need activesync, ECP, OWA, and a couple of others. They are all on 443 through IIS on the exchange server so are basically just web pages but aren't co-operating as yet.



  • @veldthui
    So whats the current config like?

    Frontend on :443 that does offloading with a certificate (and shows as such in the haproxy.conf)
    And a Backend server that also uses :443 and re-encrypts the traffic between haproxy and IIS?

    Something like this?: (i wrote it by hand.. so probably several mistakes there but the 'crt' and 'ssl' should be there..):

    frontend HTTPS_FRONTEND
      bind :443 crt /var/haproxy/HTTPS_FRONTEND.crt
      mode http
      default_backend exchange 
    backend exchange
      mode http
      server exchange 192.168.x.y:443 ssl verify none
    

    If it aint working and it looks similar to above, what does it look like exactly?



  • @PiBa At present using the ssl/tcp mode as I do not have any certs setup on HAProxy except for the one for pfsense itself using ACME.



  • @veldthui Can you share the current config?



  • Looks like this at present. webserver which is a CentOS apache webserver works. Exchange server which is on Windows 2016 IIS does not.

    # Automaticaly generated, dont edit manually.
    # Generated on: 2019-04-23 17:52
    global
    	maxconn			1000
    	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
    	uid			80
    	gid			80
    	nbproc			1
    	nbthread			1
    	hard-stop-after		15m
    	chroot				/tmp/haproxy_chroot
    	daemon
    	tune.ssl.default-dh-param	2048
    	server-state-file /tmp/haproxy_server_state
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:4444 name localstats
    	mode http
    	stats enable
    	stats refresh 10
    	stats admin if TRUE
    	stats show-legends
    	stats uri /haproxy/haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend ACME-JV-NET-NZ-PROD
    	bind			Ext IP:80 name Ext IP:80   
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	acl https ssl_fc
    	http-request set-header		X-Forwarded-Proto http if !https
    	http-request set-header		X-Forwarded-Proto https if https
    	timeout client		30000
    	errorfile			503 /var/etc/haproxy/errorfile_ACME-JV-NET-NZ-PROD_503_ExampleErrorfile
    	acl			JVNAS1	var(txn.txnhost) -m beg -i jvnas1
    	acl			ACME	var(txn.txnpath) -m beg -i /.well-known/acme-challenge/
    	http-request set-var(txn.txnhost) hdr(host)
    	http-request set-var(txn.txnpath) path
    	use_backend JVNAS1-LE_ipvANY  if  JVNAS1 
    	use_backend ACME-JV-NET-NZ-PROD_ipvANY  if  ACME 
    
    frontend HTTPS_FRONTEND
    	bind			10.101.101.1:443 name 10.101.101.1:443   
    	mode			tcp
    	log			global
    	timeout client		30000
    	errorfile			503 /var/etc/haproxy/errorfile_HTTPS_FRONTEND_503_ExampleErrorfile
    	tcp-request inspect-delay	5s
    	acl			WEBSERVER	req.ssl_sni -m beg -i webserver
    	acl			MAILSERVER	req.ssl_sni -m beg -i jvnet
    	tcp-request content accept if { req.ssl_hello_type 1 }
    	use_backend WEBSERVER_ipvANY  if  WEBSERVER 
    	use_backend MAILSERVER_ipvANY  if  MAILSERVER 
    
    backend JVNAS1-LE_ipvANY
    	mode			http
    	id			102
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			JVNAS1-BE 192.168.0.30:80 id 103 check inter 1000  
    
    backend ACME-JV-NET-NZ-PROD_ipvANY
    	mode			http
    	id			100
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			ACME-BACKEND 127.0.0.1:4002 id 101  
    
    backend WEBSERVER_ipvANY
    	mode			tcp
    	id			108
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	server			WEBSERVER 192.168.0.6:443 id 109 check inter 1000  
    
    backend MAILSERVER_ipvANY
    	mode			tcp
    	id			106
    	log			global
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    


  • @veldthui
    I presume the "backend MAILSERVER_ipvANY" also has a server similar to the 'server WEBSERVER' with the only difference being a different ip & id ? If so then i would think it 'should work'..



  • @PiBa said in HAProxy SSL mode help needed:

    @veldthui
    I presume the "backend MAILSERVER_ipvANY" also has a server similar to the 'server WEBSERVER' with the only difference being a different ip & id ? If so then i would think it 'should work'..

    It is not the same. One is an Apache webserver. The one on MAILSERVER is IIS running on Server 2016. It should still just present a web page and if I go via a port redirect to it, it responds okay and gives me the outlook web mail.
    Using HAProxy I either get a cannot provide a secure connection or web site did not respond. To get to the mail via HAProxy setup I have at present I use. https://jvnet.xxx.net:8843/owa. This gives me the error.
    The mail server has a self signed cert which has a SAN cert for the supplied FQDN.

    I might do some more google searches for exchange/haproxy issues and see what comes up. May even set up a seperate Server 2016 IIS with just a web site on it to see if that works or has issues.



  • Well I am not sure what has happened but I came home from work and got another IIS web server up and running on a VM and was going to test it but thought I would test the mail server one more time before changing it and suddenly it appears to be working. Will need to test it from work to be 100% sure as using my phone to get outside my local network and come in through pfSense WAN rather than the LAN.

    If it is working I can now test the activesync for my phone through HAProxy to the server and if that works it is all go.

    Fingers crossed but not back at work until tomorrow night.



  • Righto tonight I switched the port forward for 443 off and pointed 443 at HAProxy.

    Tested Outlook Web Mail from a remote computer and worked perfectly. Tried my iPhone using the mail app and activesync and that also worked perfectly.

    Tried my two web sites with IE and apart from the certificate errors due to be self signed that worked as well.

    Tried with Chrome and getting an error of ERR_SLL_CERT_BAD_FORMAT.

    Now this is strange because Chrome works fine if I redirect through port 8843 and a Virtual IP.

    Any ideas? Maybe a proper signed cert will fix this but not sure why it works one way and not the other.

    Certainly getting there though



  • @veldthui
    Nice that is working (mostly), i dont have any good idea about the reason for the ERR_SLL_CERT_BAD_FORMAT to appear in chrome.. Maybe the cert was created with duplicate serialnumbers of a other self created cert, or signed with a to weak fingerprint or something... Try clearing chrome's cache, and search for some details perhaps on the developer-window security and console tabs.. maybe either of them tells something more detailed? If other browsers work there is little reason to assume the problem is with the haproxy config itself.



  • Okay have put in the real certs and all is working 100%. Very nice.

    Now question. One site will have a wordpress site on which requires FTP for the updates. Can HAProxy be used to redirect port 21 the same was or am I stuck with a NAT Port redirection and limited to 1 FTP server on one machine?



  • @veldthui
    Haproxy does not 'understand' FTP protocol..
    But you might be able to do something with 'FTPS' where the ftp connection is wrapped inside SSL, and haproxy might be able to use a SNI header if the ftp client sets that... Really guessing/hoping there bigtime though.. If thats not gonna fly then i don't think haproxy will be able to help you out here. For sure its not intended for this that is for sure.

    Good the http/https part works nicely now :).


Log in to reply