HAProxy + ACME [FIXED]

uwscia

I have to say I've tried multiple different tuts trying to get this working…
It seems that there is just 1 thing every tut is missing cause it doesn't matter which one I used nothing seems to work...
Currently getting a 503 which makes me think that haproxy is working... but... This is as close as I've come.

Installed

pfsense 2.4.2
haproxy package 0.54_2
acme package 0.1.23

Looking to setup multiple sub domains to pass through firewall

www.domain.com
cloud.domain.com
vpn.domain.com

And also setup multiple local domains

plex.local or plex.domain.local
http1.local
http2.local

And I would like to get ACME to server SSL certs for them all.

PiBa

A nice wish-list, only need 3 steps:

configure pfSense so it works
configure haproxy so it works
configure acme package so it works
And your done :o , besides what you 'want', it is important for me to know what you 'did'.

As currently there is just to little information here to tell what setting you might have missed that causes a 503.
Please share haproxy.cfg (from bottom of haproxy settings tab) and some additional screenshots that you have for acme configuration. Also check and tell what haproxy's stats page LastChk is telling about the servers, are they marked as 'down' ?

https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_troubleshooting

Other than that, acme cannot provide certificates for your http1.local's domain as it wont be able to check for its existence / ownership on pubic accessible http webserver or public dns txt records..

uwscia

I'm sitting with clean install, and the packages installed…

PiBa

Ok so your sitting, that seems like a safe position.. and now what do you expect will happen.?

Have you tried anything else?

uwscia

As stated, I've tried multiple tutorials…
and to clear my many attempts I cleared everything, Cause I was still seeing squid in the menu even though I had removed it.
I wanted a fresh install to start from.

uwscia

So things I was stuck with… that I thought could be issues.

Dynamic DNS as I only have a DHCP IP from my ISP a few tutorials utilized DDNS which I have not setup.

A (IPv4) record .domain.com to my IP
CNAME(s) set to subdomain.*

HAProxy listening on same port as pfsense
Or simple Rule not set correctly
My original issues were while playing with vlan that didn't work and then later finding out I need a switch that supported them, which is ordered and in route… but I figured I could still set the rest up while I waited.

PiBa

So you have a domain name www.domain.com , if you ping that domain it results in your wan-ip? DDNS is only 'required' when the ip changes and you want to update it dynamically.. A static A record should work fine while it doesnt.. (most dynamic ip's for residential connections ive worked with stay the same for weeks or even over a year..) so not using DDNS should not be a problem not in the short term / testing phase anyhow.

pfSense webgui and haproxy using the same port should 'technically' not be a problem, however i prefer to use different ports as from a security and simplicity standpoint it could be ambiguous which service is accepting the connection if either service is not running properly ..

You did have a firewall rule that allows TCP traffic on the WAN interface from any:any to wan-ip:80 ?
You did have haproxy configured with a frontend listening on wan-ip:80 ?

What happened when you try to browse the http://domain.com ? does it timeout? does it show a 503 error? Does stats page of haproxy show the backend is 'up'? Does it work internally?

If you want to properly use vlan's indeed you need a switch that supports vlans, and know/learn how to configure them on the switch.. trunk-port vs access-port with 1 selected native and/or several tagged vlans over a port.. will need a bit of practice to get it configured right..

Anyhow if you can configure things 90% like you think they should be, and show that configuration in screenshots, and explain what part of it doesn't work yet then i can probably help you get it working..

uwscia

Ok, so DDNS not so much a requirement which is what I thought, I've also held my IP for over a year before.

The pfSense port I'll leave as is for simplicity.

Yes, I originally had a WAN Rule ipv4 and ipv6 any:any to wan on 80 an 443

Trying to browse my domain would either send me to a 503 or a pfSense binding error screen, depending if the domain was set in HAProxy ACL
If I manually use the local IP it would obviously work.

I'll start configuring HAProxy

uwscia

Ok, I've gotten a little further… this is without ACME at this point... Starting to think the squid packages that I removed were causing background issues...

www.domain is getting pointed to server locally & externally

don't know how since I never set www. to do anything yet… is the default backend setting causing this?

sub1.domain, and .domain are getting pointed to server externally but not internally… so thinking loopback issue... or caching...

Will test a few more things after dinner...

PiBa

yes first make sure the basic's work for a simple :80 webservice and perhaps a selfsigned certificate for testing before going for complete acme configuration..

If haproxy returns a 503 usually that is because the server healthchecks are failing. Check on stats page why if that happens.

haproxy indeed sends any traffic it receives on a frontend to the default backend. If traffic arrives on the frontend depends on what ip the dns resolution of www.domain returns.

If it works externally and not internally then a likely cause is the transparent-client-ip feature in the backend.. try disabling that option. Or better make sure client and server are on different networks.. (vlans might help there when the switch arrives)

uwscia

Ok, That was going to be my next question… how to stop the loopback issue.

Transparent ClientIP not checked...

So if I have my servers on 192.168.1.1 and my devices on 192.168.2.1 ect... I'll be able to access the domains locally?
Or, Is there a way to redirect my domain internally to a local domain... sub1.domain.com to sub1.domain.local?
Which would be best practice?

PiBa

Internally you could go for a split-dns solution to reply the webserver 192.168.1.100 ip when a request for sub1.domain.com is made.. But i prefer to keep it simple and have all clients both externally and internally follow (more or less) the same route to the destination.

Sending a redirect could be done also, but well the users would be using 2 different domains for the same website / or a specific page on it, making a favorite to a page wont work when going outside anymore.. Or sending a link to a colleague thats outside..

There shouldn't be much of a 'loopback issue' when using haproxy (not like you would have with portforwards..). as the client resolves the pfSense wan-ip, haproxy accepts the connection, and makes another connection to the webserver.. That should 'just work'.. or is the wan-ip not actually on pfSense but on a upstream isp provided router.?.

So to check, the clients do resolve the wan-ip of pfSense when internally requesting the domain? You have made simple 'pass' rules? (no portforwards should be needed..) Does any error appear? Does stats page 'count' a new connection on frontend when you try to connect?

uwscia

Ok so I started working on getting HTTPS to work…

I've set HTTP redirect to HTTPS, but can't seem to disable it to just test basic http... not a big deal right now... but a pain for troubleshooting...

I'm having trouble finding "good practice" guides for setting up the backend SSL to talk with cert manager...

Here is my HAproxy settings ATM..


# Automaticaly generated, dont edit manually.
# Generated on: 2017-12-07 22:10
global
	maxconn			10
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		HAproxy
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend WAN_HTTP
	bind			wan_ip:80 name wan_ip:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	default_backend ssl-redirect_http_ipvANY

frontend WAN_HTTPS
	bind			wan_ip:443 name wan_ip:443   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			www-acl	hdr(host) -i www.domain.ca
	acl			cloud-acl	hdr(host) -i cloud.domain.ca
	acl			aclcrt_WAN_HTTPS	hdr_reg(host) -i ^www\.domain\.ca(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS	hdr_reg(host) -i ^cloud\.domain\.ca(:([0-9]){1,5})?$
	acl			aclcrt_WAN_HTTPS	hdr_reg(host) -i ^domain\.ca(:([0-9]){1,5})?$
	use_backend www_http_ipvANY  if  www-acl aclcrt_WAN_HTTPS 
	use_backend cloud_http_ipvANY  if  cloud-acl aclcrt_WAN_HTTPS 

backend ssl-redirect_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			redirect 127.0.0.1  

backend www_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			www-acl 192.168.1.100:8090 ssl  verify none crt /var/etc/haproxy/server_clientcert_5a22d36348213.pem 

backend cloud_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			cloud 192.168.1.100:8082

PiBa

For https does your webserver 'require' sending a client certificate? If not make the ssl related selection boxes there on the backend server empty..
As for the frontend youve got it configured with http/https(offloading) but have not configured the server certificate on the frontend. That is required to be able to read the host-header for the "hdr_reg(host)" acl's youve used.

As for the 'redirect' backend it seems your pointing it to the pfSense localhost webgui that is then sending a redirect.. Better configure the redirect in haproxy itself as a 'action' if desired, or point it to the actual webserver:80 for testing.?

Also it might be that the overall webgui redirect in pfSense advanced settings is confusing you.. Or a cached HSTS header that the webgui sends.. To remove that HSTS redirect from browser cache special steps are needed in the browser used..

uwscia

So the certs get set in the frontend or backend?

uwscia

Okay, I'm getting somewhere, I finally got working frontend and backends… I'm just getting an error when I try to use acme cert vs the default cert.


# Automaticaly generated, dont edit manually.
# Generated on: 2017-12-08 18:36
global
	maxconn			10
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		HAproxy
	server-state-file /tmp/haproxy_server_state
	ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend ssl-redirect
	bind			wan_ip:80 name wan_ip:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	default_backend ssl-redirect_http_ipvANY

frontend HTTPS_Frontend-merged
	bind			wan_ip:443 name wan_ip:443 ssl no-sslv3 crt /var/etc/haproxy/HTTPS_Frontend.pem  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		7200000
	# Remove headers that expose security-sensitive information.
	rspidel ^Server:.*$
	rspidel ^X-Powered-By:.*$
	rspidel ^X-AspNet-Version:.*$
	acl			aclcrt_HTTPS_Frontend	hdr_reg(host) -i ^pfSense-5a22d36348213(:([0-9]){1,5})?$
	acl			sub1	hdr(host) -i sub1.domain.ca
	use_backend dummy_backend_http_ipvANY  if   aclcrt_HTTPS_Frontend
	default_backend dummy_backend_http_ipvANY
	default_backend sub1_http_ipvANY

backend ssl-redirect_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	redirect scheme https code 301

backend dummy_backend_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			dummy_backend 127.0.0.1:8888 disabled 

backend sub1_http_ipvANY
	mode			http
	log			global
	stats			enable
	stats			uri /euc_haproxy?stats
	stats			realm .
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk GET /services/Heartbeat 
	server			sub1-svr 192.168.1.100:8090 check inter 1000

PiBa

Getting 'an error' is fixed by performing 'an solution'.
Does haproxy report an error while starting? Does the browser report an error while loading the page? Does the machine catch fire ::) Please specify 'what' error specifically you get and where/when it happens that really helps both you and me..

Anyhow..

	acl			aclcrt_HTTPS_Frontend	hdr_reg(host) -i ^pfSense-5a22d36348213(:([0-9]){1,5})?$
	acl			sub1	hdr(host) -i sub1.domain.ca
	use_backend dummy_backend_http_ipvANY  if   aclcrt_HTTPS_Frontend
	default_backend dummy_backend_http_ipvANY
	default_backend sub1_http_ipvANY

Having 2 default's and a unused sub1 acl probably creates a few issues.
Use the use-backend action to select a backend when the desired sub1 acl matches instead of specifying a default backend for each frontend without further certificate derived 'automatic' acl's.

As for certificates, those that are presented to a client/browser are indeed specified on the frontend. But seeing your config that seems good now.

If the webserver also uses https, and requires a client certificate or you just want to verify its a valid certificate, then haproxy can do that with the certificate options on the server.. But this is usually not needed to get 'things working', but could improve security if the network between haproxy and webserver is not 'trusted'.. For now lets leave this alone and focus on getting the basics working.

Locally http://192.168.1.100:8090/ works without https right.?

uwscia

Ok, the error I get if I move away from the webConfigurator default cert is…
[ALERT] 342/082804 (46521) : parsing [/var/etc/haproxy_test/haproxy.cfg:36] : 'bind WAN_IP:443' : unable to load SSL certificate from PEM file

Also when I try to add another shared front end with separate backend it will throw a 503 for all https requests

So I don't need seperate backend for each server?

I need plex.domain to go to 1 server
and www.domain to go to a pool of 4 servers
plus a few more domains…

and then for the cert(s) I was thinking I could use 1 for all domains but then I'll need a seperate 1 "vpn.domain" for openVPN

Yes http://192.168.1.100:8090/ works

uwscia


# Automaticaly generated, dont edit manually.
# Generated on: 2017-12-10 08:09
global
	maxconn			10
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		HAproxy
	server-state-file /tmp/haproxy_server_state
	ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend Frontend1-http
	bind			WAN_IP:80 name WAN_IP:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		60000
	acl			httpRedirectAcl	hdr(host) -i portal.domain.ca
	acl			httpRedirectAcl	hdr_beg(host) -i support.
	acl			httpRedirectAcl	hdr(host) -i piwik.domain.ca
	acl			httpRedirectAcl	hdr(host) -i clan.domain.ca
	acl			httpRedirectAcl	hdr(host) -i plex.domain.ca
	acl			httpRedirectAcl	hdr(host) -i mine.domain.ca
	acl			httpRedirectAcl	hdr(host) -i cloud.domain.ca
	http-request redirect scheme https  if  httpRedirectAcl 
	default_backend backend-www_http_ipvANY

frontend Frontend2-SNI
	bind			WAN_IP:443 name WAN_IP:443   
	mode			tcp
	log			global
	timeout client		60000
	tcp-request inspect-delay	5s
	acl			aclSupport	req.ssl_sni -i support.domain.ca
	acl			aclForum	req.ssl_sni -m beg -i forum.
	tcp-request content accept if { req.ssl_hello_type 1 }

	tcp-request content accept if { req.ssl_hello_type 1 }

	use_backend backend-support_https_ipvANY  if  aclSupport 
	use_backend backend-forum_https_ipvANY  if  aclForum 
	default_backend fronend3-offloading_https_ipvANY

frontend Frontend3-offloading
	bind			127.0.0.1:1443 name 127.0.0.1:1443 ssl no-sslv3 crt /var/etc/haproxy/Frontend3-offloading.pem  
	bind /tmp/haproxy_chroot/Frontend3-offloading.socket name unixsocket uid 80 accept-proxy ssl no-sslv3 crt /var/etc/haproxy/Frontend3-offloading.pem 
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		60000
	acl			aclPortal	hdr(host) -i portal.domain.ca
	acl			aclClan	hdr(host) -i clan.domain.ca
	acl			aclCloud	hdr(host) -i cloud.domain.ca
	acl			aclPiwik	hdr(host) -i piwik.domain.ca
	acl			aclPlex	hdr(host) -i plex.domain.ca
	acl			aclmineOS	hdr(host) -i mine.domain.ca
	use_backend backend-portal_http_ipvANY  if  aclPortal 
	use_backend backend-clan_http_ipvANY  if  aclClan 
	use_backend backend-cloud_http_ipvANY  if  aclCloud 
	use_backend backend-piwik_http_ipvANY  if  aclPiwik 
	use_backend backend-plex_http_ipvANY  if  aclPlex 
	use_backend backend-mineOS_http_ipvANY  if  aclmineOS 
	default_backend backend-www_http_ipvANY

backend backend-www_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			Default 192.168.1.100:19999 check inter 1000  

backend backend-support_https_ipvANY
	mode			tcp
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 

backend backend-forum_https_ipvANY
	mode			tcp
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 

backend fronend3-offloading_https_ipvANY
	mode			tcp
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			frontend3-srv /Frontend3-offloading.socket send-proxy-v2-ssl-cn check inter 5000  

backend backend-portal_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 

backend backend-clan_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			Clan 192.168.1.100:8090 check inter 1000  

backend backend-cloud_http_ipvANY
	mode			http
	log			global
	rspadd Strict-Transport-Security:\ max-age=15552000;
	timeout connect		30000
	timeout server		30000
	retries			3
	server			Cloud 192.168.1.100:8082 check inter 1000  

backend backend-piwik_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			Piwik 192.168.1.100:8190 check inter 1000  

backend backend-plex_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			Plex 192.168.1.100:32400 check inter 1000  

backend backend-mineOS_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS /

uwscia

COMPLETED… TY PiBa