HAProxy noob (SONARR NZBGET RADARR etc…)

[zanesavage]

FYI…

when I type in mydomain.com I get some action in the stats:

but still I get 503 "Service unavailable" "No server is available to handle this request.

[PiBa]

-Healthchecks are good as they are the L7OK message is what you want there.

-Transparent-client-ip can be nice in the end, but doesn't fix things.. It can break things if enabled though.. Leave it off for now.

-The 503 error is 'good'

The domain name mydomain.com itself is not 'matched' by the 2 acl's that check for the complete host headers.

mydomain.com v.s. webserver.media  is that indeed what happens? (or obfuscation typo?) If that really happens then we need to figure out where that redirect is coming from and fix that.? Could mean there was some response from the server.?.  As haproxy wouldn't just return a different domain.

[zanesavage]

webserver.media is a typo.

I'm using mydomain.com (I have an actual DNS address) as an example/obfuscation.

Anytime I refresh firefox using the example mydomain.com the front end bytes increase which means there is a flow of traffic going through correct?

[zanesavage]

BTW…

This a more than likely  a dumb question but do I need to do a pass though on the LAN as I did with the WAN side?

By the way this is the site I was initially using as an example:

https://www.edwork.org/2017/06/27/pfsense-with-haproxy/

[PiBa]

No extra firewall rules are needed on lan.

mydomain.com is supposed to give a 503 error as you dont have a acl that checks for that domainname. But firewallrule is okay as it did connect to haproxy.

If you visit sonarr.mydomain.com does the frontend also count some new connection / bytes transfered?

The backends as in the last stats screenshot does not show traffic going to the backends yet.. So your request is not matching the defined acl's yet.

Perhaps try and configure a 'default backend' in the frontend.? That would skip some of the acl's.. Youve got a good domainname in that certificate also.? Perhaps disable the automatically added certificate acl ? (disable checkboxes in cert section)

[zanesavage]

Ok disabled the, "Add ACL for certificate Subject Alternative Names".

The certificate is wildcard certificate *.mydomain.com

The only two options under Frontend "default backend" is sonarr or radarr.

i tried choosing one of them but no bytes/traffic went through in stat. :(

BTW in the Frontend advance section I have:

Use "forwardfor" option - checked

and

In the Advanced pass thru section I have:

redirect scheme https code 301 if !{ ssl_fc }

Not sure if that matters.

[PiBa]

The :443 frontend does show traffic arriving right? Can you post the current latest haproxy.conf?

If there are no acls on the default_backend than basically every request should end up on that default backend..
Unless perhaps if there is a different issue..

Can you fill in the Logging options on settings tab as follows?:
Syslog Host: /var/run/log
Syslog Facility: local0
Syslog Level: Informational
Log Hostname: haproxy

On the frontend enable 'detailed logging'
Save and apply settings..

Then try a request from the browser..

Check under status/packagelogs/haproxy what shows up there.?

[zanesavage]

I removed the default backend as it didn't do anything but here's the status page and log with the default backend I created:

Apr 30 21:04:08 haproxy haproxy[39695]: Proxy webreverse started.
Apr 30 21:04:08 haproxy haproxy[39695]: Proxy sonarr_http_ipvANY started.
Apr 30 21:04:08 haproxy haproxy[39695]: Proxy radarr_http_ipvANY started.
Apr 30 21:04:08 haproxy haproxy[39695]: Proxy default_backend_http_ipvANY started.
Apr 30 21:04:08 haproxy haproxy[93951]: Stopping frontend webreverse in 0 ms.
Apr 30 21:04:08 haproxy haproxy[93951]: Stopping backend sonarr_http_ipvANY in 0 ms.
Apr 30 21:04:08 haproxy haproxy[93951]: Stopping backend radarr_http_ipvANY in 0 ms.
Apr 30 21:04:08 haproxy haproxy[93951]: Proxy webreverse stopped (FE: 1 conns, BE: 1 conns).
Apr 30 21:04:08 haproxy haproxy[93951]: Proxy sonarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 21:04:08 haproxy haproxy[93951]: Proxy radarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 21:04:14 haproxy haproxy[40185]: 10.4.0.10:62273 [30/Apr/2018:21:04:14.559] webreverse~ default_backend_http_ipvANY/ <nosrv>0/-1/-1/-1/0 503 212 - - SC-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"</nosrv>

Below is traffic from me going to mydomain.com no traffic from radarr/sonarr without default backend enabled:

# Automaticaly generated, dont edit manually.
# Generated on: 2018-04-30 20:38
global
	maxconn			50
	log			/var/run/log	local0	info
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		haproxy
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			(my external WAN IP):80 name (my external WAN IP):80   
	bind			(my external WAN IP):443 name (my external WAN IP):443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			socket-stats
	option			httplog
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	maxconn			100
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:32401 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000

This is the output from the log:

Apr 30 20:38:44 haproxy haproxy[57627]: Proxy webreverse started.
Apr 30 20:38:44 haproxy haproxy[57627]: Proxy sonarr_http_ipvANY started.
Apr 30 20:38:44 haproxy haproxy[57627]: Proxy radarr_http_ipvANY started.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping frontend webreverse in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping backend sonarr_http_ipvANY in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping backend radarr_http_ipvANY in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping backend Default_backend_http_ipvANY in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy webreverse stopped (FE: 1 conns, BE: 0 conns).
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy sonarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy radarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy Default_backend_http_ipvANY stopped (FE: 0 conns, BE: 1 conns).
Apr 30 20:38:54 haproxy haproxy[65002]: Proxy webreverse started.
Apr 30 20:38:54 haproxy haproxy[65002]: Proxy sonarr_http_ipvANY started.
Apr 30 20:38:54 haproxy haproxy[65002]: Proxy radarr_http_ipvANY started.
Apr 30 20:38:54 haproxy haproxy[57878]: Stopping frontend webreverse in 0 ms.
Apr 30 20:38:54 haproxy haproxy[57878]: Stopping backend sonarr_http_ipvANY in 0 ms.
Apr 30 20:38:54 haproxy haproxy[57878]: Stopping backend radarr_http_ipvANY in 0 ms.
Apr 30 20:38:54 haproxy haproxy[57878]: Proxy webreverse stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:54 haproxy haproxy[57878]: Proxy sonarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:54 haproxy haproxy[57878]: Proxy radarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:39:12 haproxy haproxy[65416]: 10.4.0.10:61464 [30/Apr/2018:20:39:12.810] webreverse~ webreverse/ <nosrv>-1/-1/-1/-1/1 503 212 - - SC-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
Apr 30 20:39:50 haproxy haproxy[65416]: 185.227.153.226:53328 [30/Apr/2018:20:39:50.737] webreverse webreverse/ <nosrv>-1/-1/-1/-1/0 400 187 - - CR-- 0/0/0/0/0 0/0 "<badreq>"
Apr 30 20:39:50 haproxy haproxy[65416]: 185.227.153.226:53367 [30/Apr/2018:20:39:50.967] webreverse webreverse/ <nosrv>1/-1/-1/-1/1 301 102 - - LR-- 0/0/0/0/0 0/0 "PROPFIND / HTTP/1.1"
Apr 30 20:39:51 haproxy haproxy[65416]: 185.227.153.226:53449 [30/Apr/2018:20:39:51.411] webreverse webreverse/ <nosrv>0/-1/-1/-1/0 301 137 - - LR-- 0/0/0/0/0 0/0 "POST /wls-wsat/CoordinatorPortType HTTP/1.1"</nosrv></nosrv></badreq></nosrv></nosrv>

Thanks..hope this helps

[PiBa]

Currently it still reports 'webreverse/<nosrv>' so traffic was not forwarded to a webserver..

Can you add this in "Advanced pass thru" of the frontend:

capture request header Host len 100

And check again what the package logfile shows?</nosrv>

[zanesavage]

Hi,

This is what i got:

May 1 15:55:46 haproxy haproxy[8219]: 10.4.0.10:53016 [01/May/2018:15:55:46.399] webreverse~ webreverse/ <nosrv>-1/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 {mydomain.com} "GET / HTTP/1.1"
May 1 15:55:46 haproxy haproxy[8219]: 10.4.0.10:53014 [01/May/2018:15:55:46.304] webreverse webreverse/ <nosrv>0/-1/-1/-1/0 301 89 - - LR-- 1/1/0/0/0 0/0 {mydomain.com} "GET / HTTP/1.1"</nosrv></nosrv>

If I type in sonarr.mydomain.com in the browser the log doesn't show anything. If I just type mydomain.com I get the above log event.

[PiBa]

And your really really sure that mydomain.com and sonarr.mydomain.com point to the same IP ?

[zanesavage]

mydomain.com is definitely pointing to my WAN address. I've triple checked it.

sonarr is running on my NAS with an ip of 10.4.0.18 and sonarr being on port 32401. Which matches with the info I put on the backend server.

I must have not configured HAProxy correctly?

# Automaticaly generated, dont edit manually.
# Generated on: 2018-05-01 16:08
global
	maxconn			50
	log			/var/run/log	local0	info
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		haproxy
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			xx.xx.xx.xx:80 name xx.xx.xx.xx:80   
	bind			xx.xx.xx.xx:443 name xx.xx.xx.xx:443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			socket-stats
	option			httplog
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	maxconn			100
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }
	capture request header Host len 100
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:32401 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000

I feel like it would be easier to just configure the haproxy.cfg file. Which, I've tried to do but it just gets reset as soon as I launch HAProxy in the pfsense GUI.

[PiBa]

you want requests from a browser to sonarr to be handled by haproxy right? Then the dns record for sonarr must be pointing to the ip where haproxy is listening.. so sonarr.domain.com must be pointing to the wan-address as well.?

basically you would need (imho) would be that there 3 commands:
  ping domain.com
  ping sonarr.domain.com
  ping radarr.domain.com
All would perform the same ping to the wan-ip.

If thats not the case, then in as i currently 'think' the desired state is you would need to reconfigure the DNS records for those names to point to the wanip..

[zanesavage]

HOLY MOTHER OF JESUS!! It worked…well at least radarr did. (see below) ;D

Your last email made me think and I went on my dns and I had not checked off wildcard for mydomain.com.

So sonarr.mydomain.com was not being recognized. So dumb!!!

Thank you so much for your time and especially your patience. I would buy you a beer if you were close by.

Now that I have ports 80 and 443 open…any suggestions on securing them better with pfsense?

[PiBa]

Well.. these services should be available from 'the internet' right? So the ports must be open.. nothing can be done about that part..

:80 doesn't need much securing as it is, as all requests are redirected to :443 anyhow.

You should move the pfSense webgui to a different port, 1443 or something perhaps, and disable the webgui-redirect as that would keep listening on :80 also otherwise.., so that if haproxy for some reason stops running external people wont end up on the webgui if they visit/scan your wan-ip..

You could try and use pfBlocker to limit the country's that can request the pages.. However geo-location aint a exact science. But maybe these items are only for a very limited set of known people.?. In that case you could add client-certificates to use for authentication on haproxy frontend ssl options if its only for yourself being 'on the road' then noone will be able to pass if they dont have the right client cert..

Other than that there aint much i can think of a.t.m. .. Basically you need to trust that the security of the website itself and the separated network segment / hardware its hosted from are secure.. Unless someone else has a great idea and is willing to share that :)

[zanesavage]

Thanks buddy.

I'm almost embarrassed to ask you this seeings you've already helped so much.

How do I get the SSL certificate to work for "In that case you could add client-certificates to use for authentication on haproxy frontend ssl options if its only for yourself being 'on the road' then noone will be able to pass if they dont have the right client cert.."

I have a wildcard certificate *.mydomain.com

Under SSL offloading in my Frontend my certificate shows up and I have it chosen. Underneath are some tick boxes for:

• Add ACL for certificate CommonName. (host header matches the "CN" of the certificate)
• Add ACL for certificate Subject Alternative Names.
• Load certificate ocsp responses for easy certificate validation by the client.

Do I check off all of theses?

[PiBa]

To use client certificates you would first create a CA certificate in pfSense System/CertManager. Then also create a UserCert that is signed by that CA (just select the ca while creating the usercert it will sign it automatically)
(dont try use your real publicly signed cert for this.. it actually make thing less secure..)

Then in haproxy configure the "Client verification CA certificates" select the created CA. And on your client devices download and import the user-certificate into the certificate manager of the OS your using or the browser certificate store..

Now when visiting the website it should ask for the certificate and fail for users that don't have it..

Another thing i just though of that you should do is to specify the ciphers that can be used. use below to generate the cipher settings for haproxy
  https://mozilla.github.io/server-side-tls/ssl-config-generator/
To keep it 'simple' i would stick with the 2 ssl-default-bind-ciphers and ssl-default-bind-options settings and put them on the global tab. That should allow a A rating on ssllabs test iirc..

• OCSP can be enabled it usually makes little difference, but checking the box is easy so why not ;)..
• the acl's for CN and Alternative names, well they dont really add much functionality as your already performing these actions also yourself..

[zanesavage]

Thanks man.

Followed what you said step by step but keep getting:

Errors found while starting haproxy
[ALERT] 122/084430 (51319) : parsing [/var/etc/haproxy_test/haproxy.cfg:29] : 'bind xx.xxx.xxx.xx:443' : unable to load SSL private key from PEM file '/var/etc/haproxy_test/webreverse.pem'. 
[ALERT] 122/084430 (51319) : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg 
[ALERT] 122/084430 (51319) : Fatal errors found in configuration.

no matter how many times i create a certificate and CA

[PiBa]

Not sure whats going wrong here..

Perhaps try to give either the CA or the *.domain.com a different name.?.  The /webreverse.pem file itself was working previously.. And there really is no good reason for that to have changed.. Maybe something got broken somewhere.. Try and re-import or perhaps restore a config from before it broke.?. (last 30 configs are under diagnostics/backup/history  in the gui)

[zanesavage]

Thanks for your response.

Unfortunately neither of those two options worked.

I created a few different CA's and user certificates to try with different names; also tried rebooting and then creating a CA/certificate - didn't work.

Reverted to an earlier version of HAProxy, but that didn't work.

I even tried uninstalling HAPROXY and installing HAProxy DEV but no good.

You would happen to have any other ideas?

# Automaticaly generated, dont edit manually.
# Generated on: 2018-05-03 16:16
global
	maxconn			20
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	uid			80
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state
	# set default parameters to the modern configuration
	ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
	ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			xx.xx.xxx.xx:80 name xx.xx.xxx.xx:80   
	bind			xx.xx.xxx.xx:443 name xx.xx.xxx.xx:443  ssl no-sslv3 crt /var/etc/haproxy/webreverse.pem crt-list /var/etc/haproxy/webreverse.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	maxconn			100
	timeout client		7200000
	# Remove headers that expose security-sensitive information.
	rspidel ^Server:.*$
	rspidel ^X-Powered-By:.*$
	rspidel ^X-AspNet-Version:.*$

	# add some security related headers
	rspadd Content-Security-Policy:\ default-src\ https:\ data:\ \‘unsafe-inline\\’\ \\'unsafe-eval\'
	rspadd X-Frame-Options:\ SAMEORIGIN
	rspadd X-Content-Type-Options:\ nosniff
	rspadd X-Xss-Protection:\ 1;\ mode=block
	acl			radarr	var(txn.txnhost) -m str -i radarr.mydomain.com
	acl			ombi	var(txn.txnhost) -m str -i ombi.mydomain.com
	acl			sonarr	var(txn.txnhost) -m str -i sonarr.mydomain.com
	acl			nzbget	var(txn.txnhost) -m beg -i nzbget.mydomain.com
	acl			tautulli	var(txn.txnhost) -m beg -i tautulli.mydomain.com
	http-request set-var(txn.txnhost) hdr(host)
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 
	use_backend ombi_http_ipvANY  if  ombi 
	use_backend nzbget_http_ipvANY  if  nzbget 
	use_backend tautulli_http_ipvANY  if  tautulli 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:6787 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000  

backend ombi_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			ombi 10.4.0.18:8976 check inter 1000  

backend nzbget_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			nzbget 10.4.0.18:6789 check inter 1000  

backend tautulli_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			tautulli 10.4.0.18:8660 check inter 1000

Thanks in advance…