Varnish 3 can't get it to work without NAT

jmack

I accomplished what I want, WAN-IP => muliple subsites. It works by trial and error…

But if I read - especially MARCELLOC's - posts in the forum all should function WITHOUT making use of NAT. Due to security.
https://forum.pfsense.org/index.php?topic=47962.0
https://forum.pfsense.org/index.php?topic=60297.15

My configuration now:
Pfsense 2.1.4 amd64 - Intel i3 530 - 12Gb RAM - 4 NIC's
WAN: PPPoE with subnet /29 (Have set Virtual IP's)
Packages: Snort and PfBlocker (switched off during testing)
Varnish widget: backends are "green"

NAT_rules:
WAN_IP 80 = DMZ-B_IP 80
WAN_Virtual_IP 80 = DMZ-A_IP 80

FW_rules:

* DMZ-B_IP 80
* DMZ-A_IP 80
* WAN_IP 80
* WAN_Virtual_IP 80

We are planning to downgrade in 2 weeks from now to single WAN IP instead of WAN with subnet IP's.
Does the fact that I use Virtual IP's make a difference?
In that case I might have to test it again with the new situation with single WAN IP.
But I don't want downtime, thats why I start testing it in old situation.

Something else I've tried also with Squid reverse:

listening on loopback
added virt_ip's to virt_IP/CARP field
FW_rule to ALL => WAN_address 80 or 127.0.0.1 80 (Think I used both)
NAT to 127.0.0.1 80.
The problem this gives that I can't use FW filtering IP rules for one of the sites.

If I can get it to work with Squid-reverse or HAproxy is also fine. Maybe better because then I can use https for future site.
All suggestions are welcome.

jmack

PiBa

If i understand correctly, in the end you will only have 1 public ip. I don't see how virtual ip's would make a difference there. And you want to host multiple websites on different (sub-)domains.

That means you must allow incoming traffic on port 80 to that wan-ip. And you cannot have different firewall rules for different domains on that side.

Haven't used varnish myself, but can recommend haproxy(1.5) for 'routing' traffic from 1 public ip to multiple web servers.

jmack

Mmm, new situation will have 2 sites to be proxied:
Site A - public www - 80
Site B - extranet - 80 (filtered on IP-ranges)

When all traffic goes to WAN 80, where to split the filtering on source_ip?
I guess this should be done then on proxy-level somehow? Is this possible with HAproxy?

PiBa

haproxy itself can accept/block traffic from a list of network subnets..
configuration would be something like this:

tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }

or

acl goodguys src 10.0.0.0/24
acl badguys  src 10.0.1.0/24
tcp-request content accept if goodguys
tcp-request content reject if badguys WAIT_END
tcp-request content reject

You can configure such settings in the 'advanced' textbox in the webgui. However the whitelist.lst file would need to created manually..

See http://cbonte.github.io/haproxy-dconv/configuration-1.5.html for the complete manual of options haproxy itself offers. (not everything is possible with the pfsense package webgui build in options..)

jmack

What I want to accomplish in HAproxy is following:


IF host matches www.mysite.com ==> tcp-request connection accept ALL
IF host matches extranet.mysite.com ==> tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
ELSE deny access

Om monday they are about to migrate my connection to single IP…
(Above "language" probably maybe doesn make sense, hopefully it explains what I want...)

jmack

here my new config, it does NOT block the IP's which are NOT whitelisted yet…:


global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon

frontend mhi.nl-merged
	bind			37.111.111.111:80  
	mode			http
	log			global
	option			http-keep-alive
	option			forwardfor
	reqadd X-Forwarded-Proto:\ http
	maxconn			10000
	timeout client		30000
	acl			0_www.site.nl	hdr(host) -i www.site.nl
	acl			1_extranet.site.nl	hdr(host) -i extranet.site.nl
	acl			2_whitelist { src -f /var/etc/haproxy_whitelist.lst }
	use_backend		www.site.nl_http if 0_www.site.nl 
	use_backend		extranet.site.nl_http if 2_whitelist 1_extranet.site.nl

backend www.site.nl_http
	mode			http
	balance			roundrobin
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			www.site.nl 10.11.12.13:80 check inter 1000  weight 100 

backend extranet.site.nl_http
	mode			http
	balance			roundrobin
	timeout connect		30000
	timeout server		30000
	retries			3
	source 0.0.0.0 usesrc clientip
	option			httpchk OPTIONS / 
	server			extranet.site.nl 10.14.15.16:80 check inter 1000  weight 100

PiBa

the acl is invalid and haproxy should trow and fatal alert for that remove the accolades:
acl 2_whitelist src -f /var/etc/haproxy_whitelist.lst

only when writing the acl directly behind a 'if' it needs to be between { } characters.

jmack

It all works now, here is my configfile ==>
ALL what I needed could be done from the pfsense web-config-page, I did not read this sentence properly, thats why I started to edit cfg manually….:
"acl's with the same name wil be 'combined', acl's with different names will be evaluated seperately."
So do as written, and the particular backend uses BOTH acls combined as needed... ;-)

Also, for other users, try to stick as long as possible with the web-config-page.
I was lost for at least an 2 hours, to be able to restart HAproxy properly... and maintain the right config file?!?
(it was when I began in /var/etc/haproxy.cfg, later in /var/etc/haproxy/haproxy.cfg of some reason)

The custom error pages can be pasted in Pfsense config-page / Frontend / Advanced pass thru

PfSense 2.1.5, with haproxy-devel 1.5.3 pkg v 0.10

global
	maxconn			1000
	stats socket /tmp/haproxy.socket level admin
	gid			80
	nbproc			1
	chroot			/tmp/haproxy_chroot
	daemon

listen HAProxyLocalStats
	bind 127.0.0.1:2200
	mode http
	stats enable
	stats refresh 10
	stats admin if TRUE
	stats uri /haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend site.com-merged
	bind			80.90.100.110:80  
	errorfile 400 /home/jmack/haproxy400error.http
	errorfile 403 /home/jmack/haproxy403error.http
	errorfile 408 /home/jmack/haproxy408error.http
	errorfile 500 /home/jmack/haproxy500error.http
	errorfile 502 /home/jmack/haproxy502error.http
	errorfile 503 /home/jmack/haproxy503error.http
	errorfile 504 /home/jmack/haproxy504error.http
	mode			http
	log			global
	option			log-separate-errors
	option			httplog
	option			http-keep-alive
	option			forwardfor
	reqadd X-Forwarded-Proto:\ http
	maxconn			10000
	timeout client		30000
	acl			0_www.site.com	hdr(host) -i www.site.com
	use_backend		www.site.com_http if 0_www.site.com 
	acl			1_extranet.site.com	hdr(host) -i extranet.site.com
	acl			2_extranet.site.com	src -f /home/jmack/haproxy_whitelist.1st
	use_backend		extranet.site.com_http if 1_extranet.site.com 2_extranet.site.com 

backend www.site.com_http
	mode			http
	balance			roundrobin
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			www.site.com 10.22.22.22:80 check inter 1000  weight 100 

backend extranet.site.com_http
	mode			http
	balance			roundrobin
	timeout connect		30000
	timeout server		30000
	retries			3
	source 0.0.0.0 usesrc clientip
	option			httpchk OPTIONS / 
	server			extranet.site.com 10.11.12.13:80 check inter 1000  weight 100

jmack

2 questions left:

A.)
I cannot put custom error files and whitelist file in /var/etc/ they get erased after restart.
When make readonly, they get deleted after server reboot.
I added files to /home/[user] is this place OK?

B.)
How to add to the whitelist-file CIDR subnet IP's?
It seems only possible to add single IP's….

PiBa

A) indeed /var and /tmp are deleted on every pfsense reboot
as for the proper location im not sure what would be..

B) cidr ranges should be possible (seems to work for me), ill add some basic support for ip aliases in the webgui.. (changes only loaded when haproxy config is generated again..)

jmack

B) cidr ranges should be possible (seems to work for me),

To confirm:
I tested my whitelist "cidr-only", so when single IP's; they were /32 etc.
Haproxy refuses to start…

To be able to use PFsense's aliasses would be a welcome function!

PiBa

reinstall the package, and give the 'ip matches ip or alias' acl a try, you can just put the name of a ip-alias there and it should work..

remember the list is only reloaded when config is written again while starting haproxy.. changing an alias does not currently trigger that.