Static Routing Issue

[sreece]

Hello,

I have two pfsense 2.2.6 boxes in use. One is for the standard WAN/LAN setup, and the other acts as an internal firewall between our LAN (192.168.5.0/24) and a DEVNET (10.101.101.0/24) subnet.

WAN – pfSense -- 192.168.5.1/24 -- LAN
                                    |_ 192.168.5.2/24 -- pfSense -- 10.101.101.1/24

In order to allow our LAN to reach the DEVNET, I have a static route on the 192.168.5.1 firewall that sends traffic looking for 10.101.101.0/24 to 192.168.5.2. I can ping and pass traffic, but I'm running into some strange issue with large file transfers from the LAN to DEVNET and RDP into DEVNET from the LAN. For example, RDP to some of the VMs connects, but then I have nothing but a static screen until I receive a message that the session has disconnected. If I create a route in Windows using route add 10.101.101.0 MASK 255.255.255.0 192.168.5.2, I can successfully remote in and gain control of the VM.

I've used Wireshark to examine the traffic from my PC, the main firewall, and the DEVNET firewall, and I see quite a few malformed packet messages and TCP retransmissions and continuations, but I'm having trouble figuring out why. My gut is telling me it has to do with NAT, but my gut is also stupid. If anything stands out to you, or you'd like to see the packet captures, please let me know.

Thanks!

[sreece]

I managed to fix the issue with large file transfers from the 10.101.101.0 subnet to the main LAN. The issue was my doing. I'd added a gateway to the WAN interface even though it was a private subnet. I only caught it because I was seeing ICMP redirects from the 192.168.5.1 pfsense box to the 10.101.101.X hosts, and I realized that the internal pfSense box was forwarding all the traffic to the main pfsense box on a suboptimal route. File transfers are at wire speed now.

However! Still having issues with the static route from 192.168.5.1 to 19.168.5.2 for traffic looking for the 10.101.101.0/24 subnet. The only way I've managed to work around this is to manually add a route in Windows. I did find this post from a while ago:
https://forum.pfsense.org/index.php?topic=45857.0

Sadly, my settings were already correct and my issue remains. Has anyone had issues with static routes?

[johnpoz]

you have a asymmetrical setup there.  Downstream networks should be connected via a transit network.

Any hosts you have sitting on your lan network are going to have problems talking to that downstream network even if you put a route in pfsense.  Since the answer will come direct from the downstream pfsense.

You have to either run host routing on all your hosts in your lan network that want to talk to your downstream network or that your downstream network will talk too, you need to use a transit network to connect your pfsense boxes or you need to nat at your downstream and use the nat IP to get to downstream from your lan network.

The correct solution is transit network.  Connect your downstream pfsense to your upstream pfsense via a transit say 172.16.0.0/30 and now your problem goes away.  And you have to do nothing special on any of the hosts in either network.  You can use a different interface in pfsense, or it can be a vlan.

[sreece]

So the downstream pfSense box should not have a WAN address that exists in the LAN subnet of the primary pfSense? The main pfSense has all interfaces in use or I would have just used it instead of setting up an internal pfSense box.

1. Does the transit network still allow traffic filtering rules? I need to prevent the nested LAN from having internet access except for specific VMs when they need to be updated.
2. Is there a doc about setting something like this up? Can't seem to find any walkthroughs.

The attached diagram is what I'm picturing, but I'm really confused now. Do I need to trunk vlan1 (LAN) and vlan7? Is the transit an entire subnet or just a single IP on the WAN on the internal pfSense box?

It sounds like it would be easier to add ports to the main firewall and set up a new optional interface.


[sreece]

Thanks for the advice! I ended up adding a quad i350 NIC and removing the other firewall. Things are working great now.

[johnpoz]

So you just got rid of the downstream router.. that is better solution if you ask me.

As to doc on downstream routers and pfsense.  Not sure why there really needs to be any docs on this..  This is basic networking 101 sort of stuff.. There is no doc's on what mask to use for your networks based upon number of hosts and or issue with large broadcast domains, etc. either.

As to pfsense filtering on a transit network - yeah it can filter on any interface.  You would have to create the rules so that it included the downstream networks, etc.

But this sort of issue does seem to come up quite a bit.. Maybe we should start a networking 101 section in the docs ;)  This is a mask, this is the difference between tagged and untagged vlan ;) This is why asymmetrical routing is bad, and even worse with a stateful firewall, etc..

[sreece]

OK, OK I'll take the hint. :) Going to brush up on my networking. This was a situation I'd never tried before, so at least I've learned something from it!

[johnpoz]

It always surprises me the lack of understanding transit networks and downstream routers.  Even from people that supposedly work with routing all the time.  So don't feel all that bad ;)

There is this article int he docs
https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules

This article seems to address work arounds and causes to it that might be set on pfsense itself.  But doesn't really address a common mistake not using transit networks and or placing hosts on what amounts to a transit network, etc..

Should prob take some time and round out the information provided in the above doc, this would prob be a good location for more information on the issue.  I currently just don't have the self motivation to do so ;)