Requirements or restrictions on policy based routing ?

gb

Hi Everyone!

I have been using PFSense for a year with no problems - great work by everone for such a great product!
I'm stuck though.

I have 3 production deployments - 2 work for this scenario and 1 doesn't. Same version (2.0.1 release) on all. Same hardware etc.

My scenario is simple:

External squid box on WAN subnet - configured as a gateway in addition to the 'normal' gateway which is set as 'default'.
LAN Firewall rule for all outbound traffic on port 80 setting the gateway to be the squidbox.
Outbound NAT rule on WAN interface set to not nat traffic going to squid ( I want to see the source IPs, and have a return route on thes quid host).

This all works fine for 2 deployments. My third environment is more complex but the above scenrio is still being deployed - just the pfsense host is doing a few more things, including:

Terminating IPSec tunnels
* NAT reflection settings

I'm stumped because I have enabled logging on my firewall rule and clearly can see the rule which (supposedly) sets the outbound gateway to be the squid box IS being matched. However packet capture on the wan (and squid host) show that the gateway is definately NOT being set despite the rule match being confirmed in the logs.

Where should I start digging do you think? Is there an order of events I should be aware of - eg NAT rules override Policy based settings etc? Policy based routing is not compatible with IPSec? I can't help but think that my PBR rule is working but something else is re-setting the gateway..or something..

Thanks for any ideas / pointers / debugging tips!

GB

jits

Hello.

Looks like you're having some issues there.

If you've searched through the forums and cannot find help with this topic, we apologize for that. We realize you have a choice, and we're thankful you chose PFsense. We do have a paid support service available for your convenience, especially suited to helping you resolve issues that are difficult and complicated, where answers on the forums may be, sometimes hard to come by.

Your continued support is appreciated!

That wasn't hard was it?

cmb

If you need to policy route to multiple gateways on the same interface, you need a floating rule to bypass the default pass out policy route. Pass out quick on WAN with no gateway specified.

gb

Hi Jits and cmd,

Thanks for the replies. I have not resolved this issue and my workaround was to build another box in parallel (ESX VM) and progressively migrate the config over until I hit the problem again. Alas time has not been availble to complete that so I have two firewalls for now (ouch) and I haven't encountered the problem again.

But I am very interested in your comments cmb about multiple gateways on the same interface - I think that is the key here. I have avoided floating rules until now to keep things simple, but having done (only a little) reading up since your post I can see that (for me at least), understanding floating rules and their priority (processed before any other interface rules) is very important to understand. In the case of my problems above, I can see that a floating rule will probably get things working and I'm willing to admit that I am quite capable of making mistakes and another testing pbr or something rule may be triggering this scenario despite my claims of 'everything being equal'. Off to do another audit I am and some after-hours testing of floating rules..

I have just had a quick look at the commercial support and I must admit it seems worthwhile now. We have 4 production deployments of PF covering business critical service delivery for 600 staff. The initial price points for support seems reasonable - considering even this example where a quick question to support would probably have resolved a simple problem and avoided all the messing around with building 2nd boxes etc..! Hindsight is wonderful ! I'm off to talk to the boss for some $$

Thanks for the help!

GB

dhatz

@gb:

My scenario is simple:

External squid box on WAN subnet - configured as a gateway in addition to the 'normal' gateway which is set as 'default'.

Unrelated to the original question, but rather than putting the squid box on the WAN subnet, have you considered putting it on a 3rd ("dmz") subnet?

gb

@dhatz:

Unrelated to the original question, but rather than putting the squid box on the WAN subnet, have you considered putting it on a 3rd ("dmz") subnet?

Hi dhatz,

Yep - absolutely. Originally, the squid server was being shared by other firewalls who also use transparent NAT, and the only location in common was the WAN subnet. Since then the number of firewalls has been reducing following the transition period so now the best thing is to relocate to a DMZ. There are three main reasons I can think of:

Less security concerns over locking down the squid box (when on the WAN subnet it is on a public internet address with no firewall protecting it)
Simpler with more fexibility - in the multi-wan environment we are considering, the 'smarts' to choose different WAN gateways based on failover or load-balancing is built into pfsense. With the current (external squid) design I have to duplicate wan failover/loadbalancing into the squid host. So moving squid into DMZ I can set up all the WAN load-balancing/rate-limiting/filtering once in pf and squid can take advantage of it just like any other client on the internal networks.
Not really a DMZ response and I know this not what you were suggesting, but I caution people deploying squid on a primary pfsense firewall in a more 'complex' environment. Squid on pfsense is great - no problems with me there! However I am always cautious when a proxy (or load-balance) function is deployed on a firewall. For example. My 'guest' dmz is not allowed to connect to anything on the 'inside' network. But if guests are transparently captured to squid on the firewall, how can I be absolutely sure that squid is not forwarding requests internally on behalf of the guests? Admittedly, squid has all the configuration options needed to enforce this - and probably pf as well, however I just feel better knowing that squid is completely separated and has no chance of being mis-configured in this way. Best to put that role (squid) in the correct network location to start with. For me at home though - squid goes on the firewall :)

..and just to tie this back to the thread topic a little, as I start working with floating rules and more 'generic' PBR for transparent port 80 capturing - then in my mind at least - this introduces more chance of user error (me :) ) such that things could be going to squid that shouldn't - again, more reason to ensure squid is correctly located on the network so that squid's role is enforced by the network security architecture, and less dependant on my late night stumbling around on keyboards…

GB