Weird routing issue
-
I have a pfSense box with multiple WAN interfaces and multiple LAN interfaces. Lets assume:
WAN1: 1.2.3.4
WAN2: 1.2.3.5LAN1: 10.1.0.0/16
LAN2: 10.2.1.0/24LAN2 is connected to a Sonicwall firewall that has behind it, a subnet 10.5.1.0/24.
When I ping from 10.1.0.0/16 to 10.5.1.0/24, I get replies. That is great - except, I have no static routes defined. How does pfSense know to route 10.5.1.0/24 to LAN2 GW? Is it because it knows that is the only possible route that could work? Since 10.5.1.0/24 is a private LAN hence it cannot be the WAN interfaces, it is not on the LAN1 subnet so it has a chance of being behind the LAN2 gateway?
If that is accurate I assume this will fail when a LAN3 is added, say 10.3.1.0/24, as 10.5.1.0/24 could then be behind either, right?
So I should set up a static route?
-
"10.1.0.0/16"
Really – why??? For what possible reason could you have to use a /16 on a lan segment?? That is a summary route type of mask, not a something you would put on actual network.
"LAN2 GW?"
Why would lan2 have a gateway?? If it has a gateway its not a lan interface but a wan interface.. Do you mean you created a gateway in pfsense? Or you actually put gateway on lan2 interface?
If you do in fact have downstream router then pfsense should be connected to such a router with a transit network, or you run into asynchronous routing issue when devices from lan between pfsense and your downstream and devices in the downstream network talk to each other.
Why don't you draw up your network and we can discuss how to make work what you want to make work.
edit: So for example, you have a network setup like on the left where computers A and B both point to router 1 IP as their gateway off their respective networks. So of A and B talking to each other all is fine. If router 1 has route to computer C network then A and C can talk to each other without issue.
But when B and C want to talk to each other you run into 2 problems. 1 most likely you have a hairpin, where your sending traffic to router 1 to just get sent to router 2 that is in the same "transit" network. This now causes an async routing condition unless computer B has host routing on it that says when talk to computer C send traffic to router 2 IP in its network. Since if he is just using his gateway of router 1, when the return traffic comes back from computer C through router 2 it just goes to computer B and does not go through router 1.. This is bad...
Picture on the right is how you would connect routers together so that no async happens. You do not put devices on the transit network.. If you do then ever device on that transit needs to know specific routing for which gateway it uses for each network..
-
"10.1.0.0/16"
Really – why??? For what possible reason could you have to use a /16 on a lan segment?? That is a summary route type of mask, not a something you would put on actual network.
Well AFAIK there is no performance impact or any other negative unless you actually put 65k devices (read: many devices) on such a subnet. We have way more than 254 devices, so a class C subnet is not going to work. I guess /20 would have been better, but it makes the IP address ranges harder to read and I wanted the new IT person to be able to quickly understand the network without having to figure out netmasks. Not an excuse, but since we only have about 700 or so devices I do not see an issue?
"LAN2 GW?"
Why would lan2 have a gateway?? If it has a gateway its not a lan interface but a wan interface.. Do you mean you created a gateway in pfsense? Or you actually put gateway on lan2 interface?
Perhaps we have different reference points, hence our terminology does not align. LAN2 GW referred to the point of view of a device attached to the LAN subnet. It would see the LAN2 interface (IP) on the pfSense as its gateway for its subnet. The LAN2 interface in pfSense does not have an upstream gateway as it is a LAN interface as you correctly mentioned.
If you do in fact have downstream router then pfsense should be connected to such a router with a transit network, or you run into asynchronous routing issue when devices from lan between pfsense and your downstream and devices in the downstream network talk to each other.
I think this might be the issue. Unfortunately I do not (yet) know the internals of what is behind LAN2 (Sonicwall firewall in my case, in your picture - the 192.168.1/24 network is unknown to me), I just know that LAN2 of my FW is attached to a Sonicwall firewall that has behind it some network 10.5.1.0/24.
I fixed this by adding a static route - seems like the ping responses were a fluke and it did not work reliably as per your explanation. A static route from 10.1.0.0/16 to 10.5.1.0/24 via LAN2 was the key.
Thanks for your assistance.
