HAProxy Single WAN IP -> Multiple Backends + SSL Offloading + Force 443

riaans · Jan 6, 2016, 11:53 AM

Hi, thanks in advance for helping!

We would like to setup HAProxy in the following way if possible:

–------------------------------------

1x WAN IP (HAProxy) accept port 80 and 443
<----------------->
SSL offloading and redirect 80 to 443 for WAN forcing SSL
<----------------->
Backend 1 (Single server port 80) hostname dev.example.com
Backend 2 (Sinlge server port 80) hostname tes.example.com
Backend 3 (Dual server port 80 round robbin) hostname production.example.com

Certificate is same for domain example.com (ie *.example.com)

Also would like to be able to block certain paths ( /inside1 // /inside2 ) from WAN access and only allow LAN access.

Even though I consider myself a Google Ninja, I have not been able to find a site that helps, and I also searched this forum before posting, so help would be greatly appreciated.

If someone can help with pics perhaps of the config in pfsense gui it woud be awesome. version 2.2.4

Thank you in advance!

PiBa · Jan 7, 2016, 11:07 PM

Hi Riaans,

You might want to checkout my new 'wiki' page:
https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/pfsense_2_3_haproxy_sni_plus_offloading_backends

I currently like my newest addition the 2.3 screenshot the most, it has almost the same as the package available on 2.2.x.

There is also a page there that shows the 2.2 screenshots, but that is of a older haproxy package. I think besides the color usage, the 2.3 actually does match better..

If you have read through those pages. And have questions left, let me know :).

Regards,
PiBa-NL

riaans · Jan 21, 2016, 7:58 PM

Thanks for the link and the info…

Problem is, I can get my 1st SSL Redirect to swap traffic from :80 to :443 ... but that is where the buck stops....

It doesn't redirect further down the pipe.... ie, http://dev.domain.tld goes to https://dev.domain.tld but doesnt load site...

I am doing this on a seperate IP so as not to screw with the current NAT / HAProxy setups for domains...

I have made screenshots of my 3 backends (4th is same as DEV one), and my 2 frontends.... Don't want to make this post 50pages long :)

XXX

Edit: Removed image inks

PiBa · Jan 19, 2016, 10:55 PM

Hi Riaans,

From screenshots it seems like it should work.
Perhaps you could post the haproxy config file thats generated? (On settings tab at the bottom is a link to show the config)

I would probably not use a backend just for the "redirect scheme https code 301" , but use a if statement in the frontend.

redirect scheme https code 301 if !{ ssl_fc }

Are you sure your not getting into a redirect loop or something? And the url requested is actually the proper dev.domain.tld that you are matching in the acl as well? That domain points to the haproxy ip?

Other than that, all backends show as 'up' in the stats page ?

Regards,
PiBa-NL

riaans · Jan 20, 2016, 9:02 AM

PiBa I really appreciate the help!

This is just the latest iteration in the madness that is setting up HAProxy, I've tried plenty of ideas and nothing worked, hence my post here… Will look into IF statement soon as I get a going solution first...

I'm not sure about the loop, I did network traces on the IP and opened from my side but nothing meaningful to see there..

URL requested is correct, acl matching domain name exactly... Outside DNS points to different IP {{we have a 24/7 uptime agreement}} but I have hijacked my local .hosts file to point directly to the IP for all domains...

{{I have noticed that the backend SSL-Redirect_http_ipvANY is shown twice in config, not sure why, only setup once}}

/var/etc/haproxy/haproxy.cfg file contents:
global
	maxconn			10000
	stats socket /tmp/haproxy.socket level admin
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048

frontend DOMAIN_SSL-merged
	bind			X.XXX.XXX.XX:80 name X.XXX.XXX.XX:80   
	bind			X.XXX.XXX.XX:443 name X.XXX.XXX.XX:443 ssl  crt /var/etc/haproxy/DOMAIN_SSL.pem crt /var/etc/haproxy/DOMAIN_HAProxy.pem  
	mode			http
	log			global
	option			socket-stats
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			aclcrt_DOMAIN_SSL	hdr_reg(host) -i ^([^\.]*)\.domain\.tld(:([0-9]){1,5})?$
	acl			aclcrt_DOMAIN_SSL	hdr_reg(host) -i ^domain\.tld(:([0-9]){1,5})?$
	acl			domaindev	hdr(host) -i dev.domain.tld
	acl			domaintes	hdr(host) -i tes.domain.tld
	acl			domainportal	hdr(host) -i portal.domain.tld
	acl			domainapp	hdr(host) -i app.domain.tld
	acl			domainos	hdr(host) -i os.domain.tld
	acl			aclcrt_DOMAIN_HAProxy	hdr_reg(host) -i ^([^\.]*)\.domain\.tld(:([0-9]){1,5})?$
	acl			aclcrt_DOMAIN_HAProxy	hdr_reg(host) -i ^domain\.tld(:([0-9]){1,5})?$
	use_backend Backend_DOMAIN_Dev_http_ipv4  if  domaindev aclcrt_DOMAIN_HAProxy
	use_backend Backend_DOMAIN_Tes_http_ipv4  if  domaintes aclcrt_DOMAIN_HAProxy
	use_backend Backend_DOMAIN_Prod_http_ipv4  if  domainportal aclcrt_DOMAIN_HAProxy
	use_backend Backend_DOMAIN_Prod_http_ipv4  if  domainapp aclcrt_DOMAIN_HAProxy
	use_backend Backend_DOMAIN_Prod_http_ipv4  if  domainos aclcrt_DOMAIN_HAProxy
	use_backend SSL-Redirect_http_ipvANY  if   aclcrt_DOMAIN_SSL
	use_backend SSL-Redirect_http_ipv4  if   aclcrt_DOMAIN_HAProxy

backend SSL-Redirect_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	redirect scheme https code 301
	server			nowhere 127.0.0.1:55234  

backend Backend_DOMAIN_Dev_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	option			httpchk GET /service/_ping.aspx 
	server			SERVER-DEV-01 10.10.20.30:80 check inter 1000  

backend Backend_DOMAIN_Tes_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	option			httpchk GET /service/_ping.aspx 
	server			SERVER-TES-01 10.10.20.20:80 check inter 1000  

backend Backend_DOMAIN_Prod_http_ipv4
	mode			http
	log			global
	balance			roundrobin
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	option			httpchk GET /service/_ping.aspx 
	server			SERVER-PRO-01 10.10.20.11:80 check inter 1000  
	server			SERVER-PRO-02 10.10.20.12:80 check inter 1000  

backend SSL-Redirect_http_ipv4
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	redirect scheme https code 301
	server			nowhere 127.0.0.1:55234

PiBa · Jan 20, 2016, 9:36 PM

Ive copied your config and ran it in haproxy. Changing only a few ip's..

It does not redirect dev.domain.tld to https and does send http traffic to the dev backend ( your stats screenshot does not seem to show a single connection towards dev which is a bit strange if you tried to connect to it after last restart, maybe the screenshot was taken after another restart.?.)

Anyway it seems to work properly regarding the part where my http request is forwarded to the server.. (When conditions are met..)

Could you try disabling the transparent-client-ip feature in the backend?

riaans · Jan 21, 2016, 4:30 PM

PiBa,

Thanks for all your help, I restarted without the SSL frontend, and then still could not connect even to the single frontend I set up until I figured out the true issue…

I did however resolve the problem and currently have a fully working multi-domain setup running... :o

Can I just ask (I'm going to set this up regardless) is it customary to set a default backend pointing to a blank ip/port for security? I noticed that even with a single ACL set, if I typed in the other domain names it would redirect to the first ACL set...

For those of you reading this in order to help yourselves, remember that when focusing so hard on the problem that you don't see beyond it, the problem might be beyond it... My issue ended up being as simple as not allowing traffic to my IP in the firewall.....

J. Daniel Atlas (Now you See Me (2013) : The closer you think you are, the less you'll actually see. ref: http://www.imdb.com/title/tt1670345/quotes?item=qt1944746

riaans · Feb 8, 2016, 3:51 PM

Hi,

So now a problem has jumped up because of this config…

We have 3 domains on a single IP, but now communications between the local servers are not working via the external IP/DNS names...

Obiously I can ping the IP's, tracert replies with 1line which is external IP...

I can connect directly to the servers, I can also connect if I add the domain name in .hosts file, but the moment I try and connect via the external DNS name, I get a 503 Service Unavailable...

Anyone else having this issue?

PiBa · Feb 8, 2016, 7:46 PM

Problems don't jump.. did you read the warning near the checkbox?

Client and server on same network-subnet? And Transparent-Client-IP set? If both are true that is likely the typical routing issue where server responds directly to client where client expects the response from the proxy.

There are a few options.
-splitdns, so client will always use the server directly, might be an issue for other networks trying to connect to it..
-don't use TransparentClientIP (but perhaps forwardfor header or proxyprotocol)
-move server to a different subnet
others?

riaans · Feb 8, 2016, 8:22 PM

No warning near checkbox? Am I missing something?

Client/Server on same subnet, yes. Only option set is use ForwardFor on Primary, see no options for Transparent-Client-IP…

SplitDNS - done by .hosts file?
-ForwardFor used in Primary frontend only
-Moving servers to different subset not possible, we have subnets per client...

Even though we have static IP's and setting the .hosts file once is not an issue to resolve this problem, I was hoping for a pfSense/HAProxy level solution... Don't understand why HAProxy would respond with 503 if the connection is made to server.domain.tld which points to external IP with ACL to point to correct server...

PiBa · Feb 8, 2016, 8:45 PM

In the backend edit page:

Transparent ClientIP WARNING Activating this option will load rules in IPFW and might interfere with CaptivePortal and possibly other services due to the way server return traffic must be 'captured' with a automatically created fwd rule. This also breaks directly accessing the (web)server on the ports configured above. Also a automatic sloppy pf rule is made to allow HAProxy to server traffic.

Maybe i should add that accessing haproxy-ip from the same network as where the server exists will be an issue to.. When that box is checked.

This problem is similar, method 2 should work with haproxy:
https://doc.pfsense.org/index.php/Why_can't_I_access_forwarded_ports_on_my_WAN_IP_from_my_LAN/OPTx_networks
But do check your not trading 1 problem for another again.. OPT1 clients might not like the splitdns accessing the server on LAN…