Varnish 3 can't get it to work without NAT



  • I accomplished what I want, WAN-IP => muliple subsites. It works by trial and error…

    But if I read - especially MARCELLOC's - posts in the forum all should function WITHOUT making use of NAT. Due to security.
    https://forum.pfsense.org/index.php?topic=47962.0
    https://forum.pfsense.org/index.php?topic=60297.15

    My configuration now:
    Pfsense 2.1.4 amd64 - Intel i3 530 - 12Gb RAM - 4 NIC's
    WAN: PPPoE with subnet /29 (Have set Virtual IP's)
    Packages: Snort and PfBlocker (switched off during testing)
    Varnish widget: backends are "green"

    NAT_rules:
    WAN_IP 80 = DMZ-B_IP 80
    WAN_Virtual_IP 80 = DMZ-A_IP 80

    FW_rules:

    • *  DMZ-B_IP 80
    • *  DMZ-A_IP 80
    • *  WAN_IP 80
    • *  WAN_Virtual_IP 80

    We are planning to downgrade in 2 weeks from now to single WAN IP instead of WAN with subnet IP's.
    Does the fact that I use Virtual IP's make a difference?
    In that case I might have to test it again with the new situation with single WAN IP.
    But I don't want downtime, thats why I start testing it in old situation.

    Something else I've tried also with Squid reverse:

    • listening on loopback
    • added virt_ip's to virt_IP/CARP field
    • FW_rule to ALL => WAN_address 80 or 127.0.0.1 80 (Think I used both)
    • NAT to 127.0.0.1 80.
      The problem this gives that I can't use FW filtering IP rules for one of the sites.

    If I can get it to work with Squid-reverse or HAproxy is also fine. Maybe better because then I can use https for future site.
    All suggestions are welcome.

    jmack



  • If i understand correctly, in the end you will only have 1 public ip. I don't see how virtual ip's would make a difference there. And you want to host multiple websites on different (sub-)domains.

    That means you must allow incoming traffic on port 80 to that wan-ip. And you cannot have different firewall rules for different domains on that side.

    Haven't used varnish myself, but can recommend haproxy(1.5) for 'routing' traffic from 1 public ip to multiple web servers.



  • Mmm, new situation will have 2 sites to be proxied:
    Site A - public www - 80
    Site B - extranet - 80  (filtered on IP-ranges)

    When all traffic goes to WAN 80, where to split the filtering on source_ip?
    I guess this should be done then on proxy-level somehow? Is this possible with HAproxy?



  • haproxy itself can accept/block traffic from a list of network subnets..
    configuration would be something like this:

    tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
    

    or

    acl goodguys src 10.0.0.0/24
    acl badguys  src 10.0.1.0/24
    tcp-request content accept if goodguys
    tcp-request content reject if badguys WAIT_END
    tcp-request content reject
    

    You can configure such settings in the 'advanced' textbox in the webgui. However the whitelist.lst file would need to created manually..

    See http://cbonte.github.io/haproxy-dconv/configuration-1.5.html for the complete manual of options haproxy itself offers. (not everything is possible with the pfsense package webgui build in options..)



  • What I want to accomplish in HAproxy is following:

    
    IF host matches www.mysite.com ==> tcp-request connection accept ALL
    IF host matches extranet.mysite.com ==> tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
    ELSE deny access
    
    

    Om monday they are about to migrate my connection to single IP…
    (Above "language" probably maybe doesn make sense, hopefully it explains what I want...)



  • here my new config, it does NOT block the IP's which are NOT whitelisted yet…:

    
    global
    	maxconn			1000
    	stats socket /tmp/haproxy.socket level admin
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    
    frontend mhi.nl-merged
    	bind			37.111.111.111:80  
    	mode			http
    	log			global
    	option			http-keep-alive
    	option			forwardfor
    	reqadd X-Forwarded-Proto:\ http
    	maxconn			10000
    	timeout client		30000
    	acl			0_www.site.nl	hdr(host) -i www.site.nl
    	acl			1_extranet.site.nl	hdr(host) -i extranet.site.nl
    	acl			2_whitelist { src -f /var/etc/haproxy_whitelist.lst }
    	use_backend		www.site.nl_http if 0_www.site.nl 
    	use_backend		extranet.site.nl_http if 2_whitelist 1_extranet.site.nl
    
    backend www.site.nl_http
    	mode			http
    	balance			roundrobin
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			www.site.nl 10.11.12.13:80 check inter 1000  weight 100 
    
    backend extranet.site.nl_http
    	mode			http
    	balance			roundrobin
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source 0.0.0.0 usesrc clientip
    	option			httpchk OPTIONS / 
    	server			extranet.site.nl 10.14.15.16:80 check inter 1000  weight 100
    
    


  • the acl is invalid and haproxy should trow and fatal alert for that remove the accolades:
    acl 2_whitelist src -f /var/etc/haproxy_whitelist.lst

    only when writing the acl directly behind a 'if' it needs to be between {  } characters.



  • It all works now, here is my configfile ==>
    ALL what I needed could be done from the pfsense web-config-page, I did not read this sentence properly, thats why I started to edit cfg manually….:
    "acl's with the same name wil be 'combined', acl's with different names will be evaluated seperately."
    So do as written, and the particular backend uses BOTH acls combined as needed... ;-)

    Also, for other users, try to stick as long as possible with the web-config-page.
    I was lost for at least an 2 hours, to be able to restart HAproxy properly... and maintain the right config file?!?
    (it was when I began in /var/etc/haproxy.cfg, later in /var/etc/haproxy/haproxy.cfg of some reason)

    The custom error pages can be pasted in Pfsense config-page / Frontend / Advanced pass thru

    PfSense 2.1.5, with haproxy-devel 1.5.3 pkg v 0.10

    global
    	maxconn			1000
    	stats socket /tmp/haproxy.socket level admin
    	gid			80
    	nbproc			1
    	chroot			/tmp/haproxy_chroot
    	daemon
    
    listen HAProxyLocalStats
    	bind 127.0.0.1:2200
    	mode http
    	stats enable
    	stats refresh 10
    	stats admin if TRUE
    	stats uri /haproxy_stats.php?haproxystats=1
    	timeout client 5000
    	timeout connect 5000
    	timeout server 5000
    
    frontend site.com-merged
    	bind			80.90.100.110:80  
    	errorfile 400 /home/jmack/haproxy400error.http
    	errorfile 403 /home/jmack/haproxy403error.http
    	errorfile 408 /home/jmack/haproxy408error.http
    	errorfile 500 /home/jmack/haproxy500error.http
    	errorfile 502 /home/jmack/haproxy502error.http
    	errorfile 503 /home/jmack/haproxy503error.http
    	errorfile 504 /home/jmack/haproxy504error.http
    	mode			http
    	log			global
    	option			log-separate-errors
    	option			httplog
    	option			http-keep-alive
    	option			forwardfor
    	reqadd X-Forwarded-Proto:\ http
    	maxconn			10000
    	timeout client		30000
    	acl			0_www.site.com	hdr(host) -i www.site.com
    	use_backend		www.site.com_http if 0_www.site.com 
    	acl			1_extranet.site.com	hdr(host) -i extranet.site.com
    	acl			2_extranet.site.com	src -f /home/jmack/haproxy_whitelist.1st
    	use_backend		extranet.site.com_http if 1_extranet.site.com 2_extranet.site.com 
    
    backend www.site.com_http
    	mode			http
    	balance			roundrobin
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	option			httpchk OPTIONS / 
    	server			www.site.com 10.22.22.22:80 check inter 1000  weight 100 
    
    backend extranet.site.com_http
    	mode			http
    	balance			roundrobin
    	timeout connect		30000
    	timeout server		30000
    	retries			3
    	source 0.0.0.0 usesrc clientip
    	option			httpchk OPTIONS / 
    	server			extranet.site.com 10.11.12.13:80 check inter 1000  weight 100
    


  • 2 questions left:

    A.)
    I cannot put custom error files and whitelist file in /var/etc/ they get erased after restart.
    When make readonly, they get deleted after server reboot.
    I added files to /home/[user]  is this place OK?

    B.)
    How to add to the whitelist-file CIDR subnet IP's?
    It seems only possible to add single IP's….



  • A) indeed /var and /tmp are deleted on every pfsense reboot
    as for the proper location im not sure what would be..

    B) cidr ranges should be possible (seems to work for me), ill add some basic support for ip aliases in the webgui.. (changes only loaded when haproxy config is generated again..)



  • B) cidr ranges should be possible (seems to work for me),

    To confirm:
    I tested my whitelist "cidr-only", so when single IP's; they were /32 etc.
    Haproxy refuses to start…

    To be able to use PFsense's aliasses would be a welcome function!



  • reinstall the package, and give the 'ip matches ip or alias' acl a try, you can just put the name of a ip-alias there and it should work..

    remember the list is only reloaded when config is written again while starting haproxy.. changing an alias does not currently trigger that.


Log in to reply