HAProxy noob (SONARR NZBGET RADARR etc…)

PiBa

So what does your haproxy.conf look like after working on it for a week? Can you share it perhaps there are some small things that need improving.?.

PiBa

Okay beside that haproxy listens on 10.101.101.101 while your ipalias is 10.10.9.1 and that your domain names don't match websever vs webserver vs mydomain vs myregisteredomain.. (i presume typo's while obfuscating..) Is there a reason that your not using 'wan-ip' to listen on?

What happens.?
Is your domain radarr.webserver.media is pointing to the ip haproxy is listening on with a public DNS record? Or is it in fact a private ip.? And the dns points to your wan-ip and youve got a nat rule in place that forwards the traffic to haproxy?
Do you get any error when trying to visit either domain name? Are the servers 'up' on the stats page?
Is there a timeout or a 503 error? Or does perhaps the page load 'partially' with images or css missing.?.

Locally the http://10.4.0.18:32402 does works properly? (Its not using ssl right?)

zanesavage

Sorry webserver DNS was just an example in my previous post (hence typos).

This is what I have, and I'll use mydomain.com as an example DNS url.

# Automaticaly generated, dont edit manually.
# Generated on: 2018-04-29 20:04
global
	maxconn			5
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			10
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			10.101.101.101:80 name 10.101.101.101:80   
	bind			10.101.101.101:443 name 10.101.101.101:443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			http-keep-alive
	maxconn			100
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	acl			aclcrt_webreverse	hdr_reg(host) -i ^([^\.]*)\.mydomain\.com(:([0-9]){1,5})?$
	acl			aclcrt_webreverse	hdr_reg(host) -i ^mydomain\.com(:([0-9]){1,5})?$
	use_backend sonarr_http_ipvANY  if  sonarr aclcrt_webreverse
	use_backend radarr_http_ipvANY  if  radarr aclcrt_webreverse

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			sonarr 10.4.0.18:32401  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	server			radarr 10.4.0.18:32402

So you're saying it's makes more sense to use my wan ip instead of the virtual ip correct? So I should NAT port forward 80 and 443 to 10.101.101.101; is that correct - I tried this and no difference.

My current public DNS record is pointing to my WAN.

when i go to radarr.mydomain.com I get this:

Safari can't find the server
safari can't open the page "https://radarr.mydomain.com" because it can't find the server "radarr.mydomain.com"

Also when I enable HAProxy I get this warning:

[WARNING] 118/194348 (44770) : stats socket will not work as expected in multi-process mode (nbproc > 1), you should force process binding globally using 'stats bind-process' or per socket using the 'process' attribute.
[WARNING] 118/194348 (44770) : Proxy 'HAProxyLocalStats': in multi-process mode, stats will be limited to process assigned to the current request.
[WARNING] 118/194348 (44770) : Proxy 'HAProxyLocalStats': stats admin will not work correctly in multi-process mode.

I am able to access both radarr and sonarr at http://10.4.0.18:32401 or 32402 - both work internally on local address. They are not using SSl that's one of the reasons I wanted to use HAProxy SSL offloading.

Thanks for your patience.

PiBa

To get rid of the haproxy warnings do this, it'll work better also. On settings tab:
-Decrease processes from 10 to 1
-Increase maximum connections from 5 to 50
Running with multiple processes only makes sense if you need hundreds of ssl connections or (ten-)thousands of http ones.. Which haproxy should be able to.. but i'm not sure if pfSense is then still the correct platform, probably a dedicated haproxy machine would become more convenient. But thats not yet the case here it seems..

If you let haproxy use the wan-ip then there is no need to nat anything.. You do need to allow the traffic in through the firewall though with a rule.
Make a 'pass' rule on the wan that allows from source any:any to destinations wan-ip:443 (and also a rule for wan-ip:80)
Also i like to run the pfSense webgui on a different port than the default 80/443.

I'm not completely sure what the message of Safari means that it cant find the server. Could be that it didn't find the dns record, but if you can send a ping to the name from the client device and it finds the correct ip that should be okay.. Or it just cant connect, in which case the isp / modem might be blocking traffic, or the firewall isnt allowing it yet.

b.t.w. can you also change the healthcheck on the backend to do a 'http' healthcheck.? And then check on stats page again that the servers stay UP 'green'.

zanesavage

I feel like I'm close to getting this to work, but no cigar yet.

I decreased processes to 1 and increased maximum connections to 10 which got rid of the error warning
removed the nat and made a pass rule in WAN as you suggested; any to destinations wan-ip:443 and any to destinations wan-ip:80
changed health check to http
I am able to ping my domain.com and receive a response with the DNS IP

Still no go… :(

I'm attaching images of my HAProxy GUI to see if theres something I'm not checking off that's causing the issue.

Thanks again.

PS. I have SNORT and pfBlockerNG running on the pfsense; just in case it makes a difference?

PiBa

"Still no go…" What does that mean? Error 503 ? or something else?

It looks like the frontend got at leat one connection.. But i cant tell if thats on :80 or :443 (there is a checkbox to split frontend socket stats..) The :80 would get a redirect to https so never gets to a backend.. But the 443 one should (if the acls match)

But it seems the acl's did not direct any connection to the backends yet.. That could be because your making a 'wrong request' perhaps while testing trying to visit haproxy by IP, and then the hostname requested doesnt match also the acl's gor the certificate would interfere with such a 'test'.

b.t.w. you realize that a single browser loading a website can create multiple connections to the sane site right ;) , 10 connections is a really low limit for the global setting.. if you want to limit the number of connections to a server better do that on the backend config or the server config inside the backend.

zanesavage

lol. Still no go means:

When I type in mydomain.com by itself I get 503 "Service unavailable" "No server is available to handle this request." in Firefox.

When I type in sonarr.mydomain.com in firefox I get:

Hmm. We’re having trouble finding that site.

"We can’t connect to the server at server.webserver.media.
If that address is correct, here are three other things you can try:

Try again later.
Check your network connection.
If you are connected but behind a firewall, check that Firefox has permission to access the Web."

Should I be using Transparent ClientIP on the backend? I haven't checked this to on.

Should Http check method be "OPTIONS" or "GET"? I've been using "OPTIONS"?

My Frontend ACL (sonarr) matches my Backend (sonarr)?

# Automaticaly generated, dont edit manually.
# Generated on: 2018-04-30 18:14
global
	maxconn			50
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			(my external WAN IP):80 name (my external WAN IP):80   
	bind			(my external WAN IP):443 name (my external WAN IP):443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			socket-stats
	option			http-keep-alive
	maxconn			100
	timeout client		30000
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	acl			aclcrt_webreverse	hdr_reg(host) -i ^([^\.]*)\.mydomain\.com(:([0-9]){1,5})?$
	acl			aclcrt_webreverse	hdr_reg(host) -i ^mydomain\.com(:([0-9]){1,5})?$
	use_backend sonarr_http_ipvANY  if  sonarr aclcrt_webreverse
	use_backend radarr_http_ipvANY  if  radarr aclcrt_webreverse

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:32401 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000

Just read your BTW.

I'll change the amount of connections. thanks

zanesavage

FYI…

when I type in mydomain.com I get some action in the stats:

but still I get 503 "Service unavailable" "No server is available to handle this request.

PiBa

-Healthchecks are good as they are the L7OK message is what you want there.

-Transparent-client-ip can be nice in the end, but doesn't fix things.. It can break things if enabled though.. Leave it off for now.

-The 503 error is 'good'

The domain name mydomain.com itself is not 'matched' by the 2 acl's that check for the complete host headers.

mydomain.com v.s. webserver.media is that indeed what happens? (or obfuscation typo?) If that really happens then we need to figure out where that redirect is coming from and fix that.? Could mean there was some response from the server.?. As haproxy wouldn't just return a different domain.

zanesavage

webserver.media is a typo.

I'm using mydomain.com (I have an actual DNS address) as an example/obfuscation.

Anytime I refresh firefox using the example mydomain.com the front end bytes increase which means there is a flow of traffic going through correct?

zanesavage

BTW…

This a more than likely a dumb question but do I need to do a pass though on the LAN as I did with the WAN side?

By the way this is the site I was initially using as an example:

https://www.edwork.org/2017/06/27/pfsense-with-haproxy/

PiBa

No extra firewall rules are needed on lan.

mydomain.com is supposed to give a 503 error as you dont have a acl that checks for that domainname. But firewallrule is okay as it did connect to haproxy.

If you visit sonarr.mydomain.com does the frontend also count some new connection / bytes transfered?

The backends as in the last stats screenshot does not show traffic going to the backends yet.. So your request is not matching the defined acl's yet.

Perhaps try and configure a 'default backend' in the frontend.? That would skip some of the acl's.. Youve got a good domainname in that certificate also.? Perhaps disable the automatically added certificate acl ? (disable checkboxes in cert section)

zanesavage

Ok disabled the, "Add ACL for certificate Subject Alternative Names".

The certificate is wildcard certificate *.mydomain.com

The only two options under Frontend "default backend" is sonarr or radarr.

i tried choosing one of them but no bytes/traffic went through in stat. :(

BTW in the Frontend advance section I have:

Use "forwardfor" option - checked

and

In the Advanced pass thru section I have:

redirect scheme https code 301 if !{ ssl_fc }

Not sure if that matters.

PiBa

The :443 frontend does show traffic arriving right? Can you post the current latest haproxy.conf?

If there are no acls on the default_backend than basically every request should end up on that default backend..
Unless perhaps if there is a different issue..

Can you fill in the Logging options on settings tab as follows?:
Syslog Host: /var/run/log
Syslog Facility: local0
Syslog Level: Informational
Log Hostname: haproxy

On the frontend enable 'detailed logging'
Save and apply settings..

Then try a request from the browser..

Check under status/packagelogs/haproxy what shows up there.?

zanesavage

I removed the default backend as it didn't do anything but here's the status page and log with the default backend I created:

Apr 30 21:04:08 haproxy haproxy[39695]: Proxy webreverse started.
Apr 30 21:04:08 haproxy haproxy[39695]: Proxy sonarr_http_ipvANY started.
Apr 30 21:04:08 haproxy haproxy[39695]: Proxy radarr_http_ipvANY started.
Apr 30 21:04:08 haproxy haproxy[39695]: Proxy default_backend_http_ipvANY started.
Apr 30 21:04:08 haproxy haproxy[93951]: Stopping frontend webreverse in 0 ms.
Apr 30 21:04:08 haproxy haproxy[93951]: Stopping backend sonarr_http_ipvANY in 0 ms.
Apr 30 21:04:08 haproxy haproxy[93951]: Stopping backend radarr_http_ipvANY in 0 ms.
Apr 30 21:04:08 haproxy haproxy[93951]: Proxy webreverse stopped (FE: 1 conns, BE: 1 conns).
Apr 30 21:04:08 haproxy haproxy[93951]: Proxy sonarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 21:04:08 haproxy haproxy[93951]: Proxy radarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 21:04:14 haproxy haproxy[40185]: 10.4.0.10:62273 [30/Apr/2018:21:04:14.559] webreverse~ default_backend_http_ipvANY/ <nosrv>0/-1/-1/-1/0 503 212 - - SC-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"</nosrv>

Below is traffic from me going to mydomain.com no traffic from radarr/sonarr without default backend enabled:

# Automaticaly generated, dont edit manually.
# Generated on: 2018-04-30 20:38
global
	maxconn			50
	log			/var/run/log	local0	info
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		haproxy
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			(my external WAN IP):80 name (my external WAN IP):80   
	bind			(my external WAN IP):443 name (my external WAN IP):443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			socket-stats
	option			httplog
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	maxconn			100
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:32401 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000

This is the output from the log:

Apr 30 20:38:44 haproxy haproxy[57627]: Proxy webreverse started.
Apr 30 20:38:44 haproxy haproxy[57627]: Proxy sonarr_http_ipvANY started.
Apr 30 20:38:44 haproxy haproxy[57627]: Proxy radarr_http_ipvANY started.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping frontend webreverse in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping backend sonarr_http_ipvANY in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping backend radarr_http_ipvANY in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Stopping backend Default_backend_http_ipvANY in 0 ms.
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy webreverse stopped (FE: 1 conns, BE: 0 conns).
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy sonarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy radarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:44 haproxy haproxy[51916]: Proxy Default_backend_http_ipvANY stopped (FE: 0 conns, BE: 1 conns).
Apr 30 20:38:54 haproxy haproxy[65002]: Proxy webreverse started.
Apr 30 20:38:54 haproxy haproxy[65002]: Proxy sonarr_http_ipvANY started.
Apr 30 20:38:54 haproxy haproxy[65002]: Proxy radarr_http_ipvANY started.
Apr 30 20:38:54 haproxy haproxy[57878]: Stopping frontend webreverse in 0 ms.
Apr 30 20:38:54 haproxy haproxy[57878]: Stopping backend sonarr_http_ipvANY in 0 ms.
Apr 30 20:38:54 haproxy haproxy[57878]: Stopping backend radarr_http_ipvANY in 0 ms.
Apr 30 20:38:54 haproxy haproxy[57878]: Proxy webreverse stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:54 haproxy haproxy[57878]: Proxy sonarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:38:54 haproxy haproxy[57878]: Proxy radarr_http_ipvANY stopped (FE: 0 conns, BE: 0 conns).
Apr 30 20:39:12 haproxy haproxy[65416]: 10.4.0.10:61464 [30/Apr/2018:20:39:12.810] webreverse~ webreverse/ <nosrv>-1/-1/-1/-1/1 503 212 - - SC-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
Apr 30 20:39:50 haproxy haproxy[65416]: 185.227.153.226:53328 [30/Apr/2018:20:39:50.737] webreverse webreverse/ <nosrv>-1/-1/-1/-1/0 400 187 - - CR-- 0/0/0/0/0 0/0 "<badreq>"
Apr 30 20:39:50 haproxy haproxy[65416]: 185.227.153.226:53367 [30/Apr/2018:20:39:50.967] webreverse webreverse/ <nosrv>1/-1/-1/-1/1 301 102 - - LR-- 0/0/0/0/0 0/0 "PROPFIND / HTTP/1.1"
Apr 30 20:39:51 haproxy haproxy[65416]: 185.227.153.226:53449 [30/Apr/2018:20:39:51.411] webreverse webreverse/ <nosrv>0/-1/-1/-1/0 301 137 - - LR-- 0/0/0/0/0 0/0 "POST /wls-wsat/CoordinatorPortType HTTP/1.1"</nosrv></nosrv></badreq></nosrv></nosrv>

Thanks..hope this helps

PiBa

Currently it still reports 'webreverse/<nosrv>' so traffic was not forwarded to a webserver..

Can you add this in "Advanced pass thru" of the frontend:

capture request header Host len 100

And check again what the package logfile shows?</nosrv>

zanesavage

Hi,

This is what i got:

May 1 15:55:46 haproxy haproxy[8219]: 10.4.0.10:53016 [01/May/2018:15:55:46.399] webreverse~ webreverse/ <nosrv>-1/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 {mydomain.com} "GET / HTTP/1.1"
May 1 15:55:46 haproxy haproxy[8219]: 10.4.0.10:53014 [01/May/2018:15:55:46.304] webreverse webreverse/ <nosrv>0/-1/-1/-1/0 301 89 - - LR-- 1/1/0/0/0 0/0 {mydomain.com} "GET / HTTP/1.1"</nosrv></nosrv>

If I type in sonarr.mydomain.com in the browser the log doesn't show anything. If I just type mydomain.com I get the above log event.

PiBa

And your really really sure that mydomain.com and sonarr.mydomain.com point to the same IP ?

zanesavage

mydomain.com is definitely pointing to my WAN address. I've triple checked it.

sonarr is running on my NAS with an ip of 10.4.0.18 and sonarr being on port 32401. Which matches with the info I put on the backend server.

I must have not configured HAProxy correctly?

# Automaticaly generated, dont edit manually.
# Generated on: 2018-05-01 16:08
global
	maxconn			50
	log			/var/run/log	local0	info
	stats socket /tmp/haproxy.socket level admin
	uid			80
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	log-send-hostname		haproxy
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend webreverse
	bind			xx.xx.xx.xx:80 name xx.xx.xx.xx:80   
	bind			xx.xx.xx.xx:443 name xx.xx.xx.xx:443 ssl  crt /var/etc/haproxy/webreverse.pem  
	mode			http
	log			global
	option			socket-stats
	option			httplog
	option			http-keep-alive
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	maxconn			100
	timeout client		30000
	redirect scheme https code 301 if !{ ssl_fc }
	capture request header Host len 100
	acl			sonarr	hdr(host) -i sonarr.mydomain.com
	acl			radarr	hdr(host) -i radarr.mydomain.com
	use_backend sonarr_http_ipvANY  if  sonarr 
	use_backend radarr_http_ipvANY  if  radarr 

backend sonarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			sonarr 10.4.0.18:32401 check inter 1000  

backend radarr_http_ipvANY
	mode			http
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			radarr 10.4.0.18:32402 check inter 1000

I feel like it would be easier to just configure the haproxy.cfg file. Which, I've tried to do but it just gets reset as soon as I launch HAProxy in the pfsense GUI.

PiBa

you want requests from a browser to sonarr to be handled by haproxy right? Then the dns record for sonarr must be pointing to the ip where haproxy is listening.. so sonarr.domain.com must be pointing to the wan-address as well.?

basically you would need (imho) would be that there 3 commands:
ping domain.com
ping sonarr.domain.com
ping radarr.domain.com
All would perform the same ping to the wan-ip.

If thats not the case, then in as i currently 'think' the desired state is you would need to reconfigure the DNS records for those names to point to the wanip..