HAProxy noob (SONARR NZBGET RADARR etc…)

PiBa

Currently it still reports 'webreverse/<nosrv>' so traffic was not forwarded to a webserver..

Can you add this in "Advanced pass thru" of the frontend:

capture request header Host len 100

And check again what the package logfile shows?</nosrv>

zanesavage

Hi,

This is what i got:

May 1 15:55:46 haproxy haproxy[8219]: 10.4.0.10:53016 [01/May/2018:15:55:46.399] webreverse~ webreverse/ <nosrv>-1/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 {mydomain.com} "GET / HTTP/1.1"
May 1 15:55:46 haproxy haproxy[8219]: 10.4.0.10:53014 [01/May/2018:15:55:46.304] webreverse webreverse/ <nosrv>0/-1/-1/-1/0 301 89 - - LR-- 1/1/0/0/0 0/0 {mydomain.com} "GET / HTTP/1.1"</nosrv></nosrv>

If I type in sonarr.mydomain.com in the browser the log doesn't show anything. If I just type mydomain.com I get the above log event.

PiBa

And your really really sure that mydomain.com and sonarr.mydomain.com point to the same IP ?

zanesavage

mydomain.com is definitely pointing to my WAN address. I've triple checked it.

sonarr is running on my NAS with an ip of 10.4.0.18 and sonarr being on port 32401. Which matches with the info I put on the backend server.

I must have not configured HAProxy correctly?

# Automaticaly generated, dont edit manually.
# Generated on: 2018-05-01 16:08
global
	maxconn			50
	log			/var/run/log	local0	info
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		haproxy
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			xx.xx.xx.xx:80 name xx.xx.xx.xx:80   
	bind			xx.xx.xx.xx:443 name xx.xx.xx.xx:443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			socket-stats
	option			httplog
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	maxconn			100
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }
	capture request header Host len 100
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:32401 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000

I feel like it would be easier to just configure the haproxy.cfg file. Which, I've tried to do but it just gets reset as soon as I launch HAProxy in the pfsense GUI.

PiBa

you want requests from a browser to sonarr to be handled by haproxy right? Then the dns record for sonarr must be pointing to the ip where haproxy is listening.. so sonarr.domain.com must be pointing to the wan-address as well.?

basically you would need (imho) would be that there 3 commands:
ping domain.com
ping sonarr.domain.com
ping radarr.domain.com
All would perform the same ping to the wan-ip.

If thats not the case, then in as i currently 'think' the desired state is you would need to reconfigure the DNS records for those names to point to the wanip..

zanesavage

HOLY MOTHER OF JESUS!! It worked…well at least radarr did. (see below) ;D

Your last email made me think and I went on my dns and I had not checked off wildcard for mydomain.com.

So sonarr.mydomain.com was not being recognized. So dumb!!!

Thank you so much for your time and especially your patience. I would buy you a beer if you were close by.

Now that I have ports 80 and 443 open…any suggestions on securing them better with pfsense?

PiBa

Well.. these services should be available from 'the internet' right? So the ports must be open.. nothing can be done about that part..

:80 doesn't need much securing as it is, as all requests are redirected to :443 anyhow.

You should move the pfSense webgui to a different port, 1443 or something perhaps, and disable the webgui-redirect as that would keep listening on :80 also otherwise.., so that if haproxy for some reason stops running external people wont end up on the webgui if they visit/scan your wan-ip..

You could try and use pfBlocker to limit the country's that can request the pages.. However geo-location aint a exact science. But maybe these items are only for a very limited set of known people.?. In that case you could add client-certificates to use for authentication on haproxy frontend ssl options if its only for yourself being 'on the road' then noone will be able to pass if they dont have the right client cert..

Other than that there aint much i can think of a.t.m. .. Basically you need to trust that the security of the website itself and the separated network segment / hardware its hosted from are secure.. Unless someone else has a great idea and is willing to share that :)

zanesavage

Thanks buddy.

I'm almost embarrassed to ask you this seeings you've already helped so much.

How do I get the SSL certificate to work for "In that case you could add client-certificates to use for authentication on haproxy frontend ssl options if its only for yourself being 'on the road' then noone will be able to pass if they dont have the right client cert.."

I have a wildcard certificate *.mydomain.com

Under SSL offloading in my Frontend my certificate shows up and I have it chosen. Underneath are some tick boxes for:

Add ACL for certificate CommonName. (host header matches the "CN" of the certificate)
Add ACL for certificate Subject Alternative Names.
Load certificate ocsp responses for easy certificate validation by the client.

Do I check off all of theses?

PiBa

To use client certificates you would first create a CA certificate in pfSense System/CertManager. Then also create a UserCert that is signed by that CA (just select the ca while creating the usercert it will sign it automatically)
(dont try use your real publicly signed cert for this.. it actually make thing less secure..)

Then in haproxy configure the "Client verification CA certificates" select the created CA. And on your client devices download and import the user-certificate into the certificate manager of the OS your using or the browser certificate store..

Now when visiting the website it should ask for the certificate and fail for users that don't have it..

Another thing i just though of that you should do is to specify the ciphers that can be used. use below to generate the cipher settings for haproxy
https://mozilla.github.io/server-side-tls/ssl-config-generator/
To keep it 'simple' i would stick with the 2 ssl-default-bind-ciphers and ssl-default-bind-options settings and put them on the global tab. That should allow a A rating on ssllabs test iirc..

OCSP can be enabled it usually makes little difference, but checking the box is easy so why not ;)..
the acl's for CN and Alternative names, well they dont really add much functionality as your already performing these actions also yourself..

zanesavage

Thanks man.

Followed what you said step by step but keep getting:

Errors found while starting haproxy
[ALERT] 122/084430 (51319) : parsing [/var/etc/haproxy_test/haproxy.cfg:29] : 'bind xx.xxx.xxx.xx:443' : unable to load SSL private key from PEM file '/var/etc/haproxy_test/webreverse.pem'. 
[ALERT] 122/084430 (51319) : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg 
[ALERT] 122/084430 (51319) : Fatal errors found in configuration.

no matter how many times i create a certificate and CA

PiBa

Not sure whats going wrong here..

Perhaps try to give either the CA or the *.domain.com a different name.?. The /webreverse.pem file itself was working previously.. And there really is no good reason for that to have changed.. Maybe something got broken somewhere.. Try and re-import or perhaps restore a config from before it broke.?. (last 30 configs are under diagnostics/backup/history in the gui)

zanesavage

Thanks for your response.

Unfortunately neither of those two options worked.

I created a few different CA's and user certificates to try with different names; also tried rebooting and then creating a CA/certificate - didn't work.

Reverted to an earlier version of HAProxy, but that didn't work.

I even tried uninstalling HAPROXY and installing HAProxy DEV but no good.

You would happen to have any other ideas?

# Automaticaly generated, dont edit manually.
# Generated on: 2018-05-03 16:16
global
	maxconn			20
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state
	# set default parameters to the modern configuration
	ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
	ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			xx.xx.xxx.xx:80 name xx.xx.xxx.xx:80   
	bind			xx.xx.xxx.xx:443 name xx.xx.xxx.xx:443  ssl no-sslv3 crt /var/etc/haproxy/webreverse.pem crt-list /var/etc/haproxy/webreverse.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	maxconn			100
	timeout client		7200000
	# Remove headers that expose security-sensitive information.
	rspidel ^Server:.*$
	rspidel ^X-Powered-By:.*$
	rspidel ^X-AspNet-Version:.*$

	# add some security related headers
	rspadd Content-Security-Policy:\ default-src\ https:\ data:\ \‘unsafe-inline\\’\ \\'unsafe-eval\'
	rspadd X-Frame-Options:\ SAMEORIGIN
	rspadd X-Content-Type-Options:\ nosniff
	rspadd X-Xss-Protection:\ 1;\ mode=block
	acl			radarr	var(txn.txnhost) -m str -i radarr.mydomain.com
	acl			ombi	var(txn.txnhost) -m str -i ombi.mydomain.com
	acl			sonarr	var(txn.txnhost) -m str -i sonarr.mydomain.com
	acl			nzbget	var(txn.txnhost) -m beg -i nzbget.mydomain.com
	acl			tautulli	var(txn.txnhost) -m beg -i tautulli.mydomain.com
	http-request set-var(txn.txnhost) hdr(host)
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 
	use_backend ombi_http_ipvANY  if  ombi 
	use_backend nzbget_http_ipvANY  if  nzbget 
	use_backend tautulli_http_ipvANY  if  tautulli 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:6787 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000  

backend ombi_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			ombi 10.4.0.18:8976 check inter 1000  

backend nzbget_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			nzbget 10.4.0.18:6789 check inter 1000  

backend tautulli_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			tautulli 10.4.0.18:8660 check inter 1000

Thanks in advance…

PiBa

This shouldnt be dependant on the version of haproxy used..

unable to load SSL private key from PEM file '/var/etc/haproxy_test/webreverse.pem'.

Can you check the content of that file, does it properly contain both the certificate and the (decrypted) psk ? Of your *.mydomain.com certificate ?

if you download the certificate in the certmanager, does it contain the correct cert and key there as well?

zanesavage

I used pfsense Diagnostic Edit file to go:

'/var/etc/haproxy_test/webreverse.pem'

File is empty?

When I checked the certificate it was empty too?

This what I'm doing to create a certificate:

ADD CA certificate
Method - create internal CA (fill out info)
common name - *.mydomain.com

Add certificate
Method - create internal certificate
Certificate authority - same as above from pull down list
fill out info
common name - *.mydomain.com

certificate type - user certificate
add

PiBa

Could it be that the selected "Certificate" on the frontend is not pointing to the 'official' signed cert anymore? Re-select and save that one?

(its stored by 'refid', so if you delete a cert and re-create it with the same name it wont be the same cert anymore and it might create a empty file because of that.?.)

Can you check inside the config.xml that the ID's used for the certificate in haproxy really do match those of a certificate?

For example below im using a certificate with refid '5a4004718858f'..

	 <cert><refid>5a4004718858f</refid>

		<type>server</type>
		<caref>57d3118d56766</caref>
		<crt>My-Cert-DATA DATA DATA....</crt>
		<prv>My-PSK-DATA DATA DATA....</prv></cert>

	 <installedpackages><haproxy><ha_backends><name>vhost1</name>
					<status>active</status>
					<secondary>yes</secondary>
					<primary_frontend>TEST-SNI</primary_frontend>
					<type>http</type>
					<httpclose>http-keep-alive</httpclose>
					<backend_serverpool>vhost1</backend_serverpool>
					<ssloffloadcert>5a4004718858f</ssloffloadcert></ha_backends></haproxy></installedpackages>

zanesavage

this is what it says

 <cert><refid>5aeb86ca0c986</refid>

		<type>user</type>
		<caref>5aeb8678bcd07</caref></cert>

 <haproxy><configversion>00.32</configversion>
			 <ha_backends><name>webreverse</name>
					<desc>site accessible tp public</desc>
					<status>active</status>
					<type>http</type>
					<httpclose>http-keep-alive</httpclose>
					<max_connections>100</max_connections>
					<client_timeout>7200000</client_timeout>
					<ssloffloadcert>5aeb86ca0c986</ssloffloadcert>
					<dcertadv>no-sslv3</dcertadv></ha_backends></haproxy>

I think the problem is that when I create an Internal certificate from the Internal CA the certificate is empty; I downloaded the certificate and it's zero bytes nothing inside?

Is the process that I'm following to create the CA and certificate seem right? See below:

ADD CA certificate
Method - create internal CA (fill out info)
common name - *.mydomain.com

Add certificate
Method - create internal certificate
Certificate authority - same as above from pull down list
fill out info
common name - *.mydomain.com

certificate type - user certificate
Alternative Names - I didn't put anything here
add

Should I be creating an intermediate certificate too?

zanesavage

Some progress… :o

If a create a CA and then take the export CA info and add it to "import an existing Certificate" when creating a certificate, I no longer get the HAProxy error:

Errors found while starting haproxy
[ALERT] 122/084430 (51319) : parsing [/var/etc/haproxy_test/haproxy.cfg:29] : 'bind xx.xxx.xxx.xx:443' : unable to load SSL private key from PEM file '/var/etc/haproxy_test/webreverse.pem'. 
[ALERT] 122/084430 (51319) : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg 
[ALERT] 122/084430 (51319) : Fatal errors found in configuration.

When i go to backend sonarr.mydomain.com I get, "safari can't open the page because Safari can't establish a connection to the server" however when I go to one of the other backends ombi.mydomain.com I was prompted to download the certificate. Although, once I downloaded the certificate I got the same safari warning, "safari can't open the page because Safari can't establish a connection to the server"

hmmm… :(

UPDATE*

I thought it was strange that some backends provided certificate while others didn't so I flushed my DNS Cache and now all the backends provided the certificate when I went to page, However after that still same Safari warning "safari can't open the page because Safari can't establish a connection to the server".

****** was going to delete the above, but decided not to in case it's useful to someone in the future. However, it's now irrelevant because of the below ******

UPDATE TWO

Well, after a very long night of having to setup pfsense from scratch (off the USB), I can now create user certificates that are populated with info from the Internal CA, and no longer blank. Looks like something was corrupting the ability to do that before.

So this is where I'm at today:

# Automaticaly generated, dont edit manually.
# Generated on: 2018-05-04 11:34
global
	maxconn			20
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state
	# set default parameters to the modern configuration

	    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
	    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
	    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
	    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats refresh 10
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			xx.xxx.xxx.xxx:80 name xx.xxx.xxx.xxx:80   
	bind			xx.xxx.xxx.xxx:443 name xx.xxx.xxx.xxx:443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			socket-stats
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		7200000
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	acl			ombi	hdr(host) -i ombi.mydomain.com
	acl			nzbget	hdr(host) -i nzbget.mydomain.com
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 
	use_backend ombi_http_ipvANY  if  ombi 
	use_backend nzbget_http_ipvANY  if  nzbget 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:6787 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000  

backend ombi_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			ombi 10.4.0.18:8976 check inter 1000  

backend nzbget_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			nzbget 10.4.0.18:6789 check inter 1000

When I have a 'user certificate' under 'SSL Offloading' and the CA under 'SSL Offloading - client certificates' and the user certificate loaded on my computer and on my iPhone the webpage just reloads constantly asking to confirm certificate.
When I have a 'server certificate' under 'SSL Offloading' and nothing in 'SSL Offloading - client certificates' then the webpage loads in https.

The only things I have altered from default in the HAProxy Frontend are:

Advanced settings - Client timeout - 7200000
SSL Offloading - OCSP - checked

PS. When creating the certificate for both user and server. I add 'Alternative Names' for each of my backends. For example:

sonarr.mydomain.com
radarr.mydomain.com

In case that makes a difference.

Any thoughts on why the webpage reloads constantly with a user certificate?

Cheer, and thanks in advance.

PiBa

When I have a 'user certificate' under 'SSL Offloading' and the CA under 'SSL Offloading - client certificates' and the user certificate loaded on my computer and on my iPhone the webpage just reloads constantly asking to confirm certificate.

The user certificate is only for the user, it should not be configure in haproxy

When I have a 'server certificate' under 'SSL Offloading' and nothing in 'SSL Offloading - client certificates' then the webpage loads in https.

This is a good starting point.. Now configure the CA that is used to generate the usercertificate under the client certificate options..

When creating the certificate for both user and server. I add 'Alternative Names' for each of my backends. For example:
sonarr.mydomain.com
radarr.mydomain.com

But you have a valid *.mydomain.com certificate right.? So no need to create a servercert yourself..

As for the user certificate, it doesnt need any alternative-names… just put in 'zanesavage' as the CN for the user-cert.. and download that to import into the client device OS or Browser certificate store..

rekd0514

What do you put in the Base URL on Ombi, Sonarr, etc. when using HAProxy?

EDIT: I think I found the answer. You don't need it unless you want /ombi for example after your domain name. (yourdomain.com/ombi)