Static routes ignored
I've recently built myself a little network for learning some of the basics of pfSense firewalling and routing. The idea is to have a perimeter firewall controlling Internet access, with separate firewalls behind that for controlling access to a DMZ and protected LANs.
The internal firewalls (fw1 and fw2) have fw0 set as their default gateway, accessible via their route1 interface. Additionally, fw1 has fw2 as a second gateway on its route1 interface, and static routes to lan1 and lan2 via this gateway. Likewise, fw2 has fw1 as a second gateway on its route1 interface, with a static route to the DMZ.
Unfortunately, it seems that my static routes are being ignored. Example: ping from svr3 to svr1 only works when fw0 is turned on. This traffic should be routed directly from fw2 to fw1, leaving fw0 out of the loop completely.
I am pretty sure it has something to do with firewall rules and traffic being sent to the default gateway instead of the gateway specified in the static route. This thread: https://forum.pfsense.org/index.php?topic=49963.0 seems to be dealing with the same issue. Accordingly, I have tried creating floating rules to fix the problem, but no luck. Perhaps I just haven't implemented the rules correctly yet, but I've tried every permutation I can think of with no success.
Am I on the right track with the floating rules thing, or am I missing something else here?
My first instinct upon seeing your diagram was to ask why 3 firewalls?
Unless I'm misreading something about your setup (entirely possible) you should be able to do everything you've described with one firewall and three LAN NICS or one LAN NIC and three VLANS plus a VLAN switch. I'm betting that would drastically simplify your rules and your ability to debug this setup.
What hardware are you using to implement this?
You're right; it would be much simpler with a single firewall. That's how I initially had it built out, but my company has this thing about "separation of services" and our security guy wanted to see a little more "separation."
I was asked to start learning about ways to build an inexpensive, secure, modular network that could be easily brought online from backups at a DR site in the event of a failure at the production site. The whole thing is currently virtual on ESXi.
If you really need to show the separation capabilities, I would do this in multiple stages. First create the all in one solution and test the various rules you need to isolate all the pieces. Second, pick one of the three firewalls to separate out and remove that functionality from the integrated firewall and create the new unit to fulfill the functions you need. Thirdly move out the last piece and test again.
The advantage of this approach is it's much easier to test a single piece at a time as you convert from integrated to separate firewalls and secondly you'll have examples you can show to the powers that be of the pluses and minuses of each approach. The entire setup could be done in VM as required (which kind of blows the argument for "separation" of function when all the functions can be virtualized in one machine ::) )
Just my $.02
OK… so I have taken your strategy and adapted it somewhat.
I already had the system working with a single pfSense box (dmz, lan1, and lan2 all directly behind fw0), but needed to segregate the functionality a bit. (Yes, you might argue that it's all running on one machine anyway since it's all virtual, but that's not exactly true -- it's running on a cluster of failover-capable physical servers. So the separation thing isn't really coming from a concern about hardware reliability, but more out of an interest in keeping the virtual machines -- and the platforms that might be compromised by an attacker -- separate.)
So from there I broke out the dmz firewall (fw1) and everything worked as expected. At this point, however, this second firewall still only has one gateway and the default route to get to everything else.
If I then add fw2 as in the original design, adding the appropriate gateways and routes, I run into the original problem: traffic sent out on the "wan" (route1) interface of fw1 or fw2 uses the default gateway (fw0) instead of the one specified in the static route.
However, if I add fw2 BEHIND fw1 -- so that all three firewalls are in sequence-- all the routing works as I would expect:
Again, however, in this scenario, each firewall still only has one gateway per interface.
You might ask why I don't just use this second design instead of the first, since it works. My answer is that it is, in my opinion, more complicated, and increases the firewalls' inter-dependency. I like that in the original design, any one of the three firewalls could be offline, while the other two (and their respective networks) could still function together.
Which leads me back to my original problem: how do I get traffic to go out the correct gateway when there are multiple gateways on an interface, and simply creating static routes doesn't seem to work?
by implementing security by obscurity one usually ends up with holes in his/her feet