PfSense routing connected subnets through NAT
-
Hello,
I'm trying to make hosts in two different subnets (both directly connected to the pfSense box, and the system has IP addresses in both) talk to each other.
The two subnets are:-
LAN (igb1) - 10.42.2.1/16
-
TIMENET_MODEM (igb0) - 10.41.0.2/30
EDIT/CLARIFICATION: these are both not WANs.
The routing table looks fine (attachment pf1.png), but pinging 10.42.0.0/16 hosts results in the packets being dropped. Further investigation with traceroute highlighted the problem: the packets are policy-routed and NATted through the Internet gateway (the lowest tier gateway is through PPPoE interface to eolo-gw.net.ngi.it, which is what you see in the traceroute), and are thus leaving on to the ISP's network instead of the other local network.
[from a computer with IP address 10.42.20.37] >tracert 10.41.0.1 Tracing route to 10.41.0.1 over a maximum of 30 hops 1 10 ms 10 ms 10 ms eolo-gw.net.ngi.it [81.174.0.21] <- this is the Internet gateway 2 12 ms 10 ms 14 ms 10.221.0.1 <- ISP stuff 3 36 ms 44 ms 38 ms 10.40.36.74 <- ISP stuff 4 * * * Request timed out. 5 * * * Request timed out. 6 * * * Request timed out. [...] 16 * * * Request timed out. 17 * * ^C
Attachments pf2.png and pf3.png contain firewall rules for the LAN and TIMENET_MODEM interfaces. NAT is in hybrid mode, with no manual rules added.
I don't know how to further debug the issue. Can someone help?
P.S. I also posted this on the Reddit, will cross-post any solution as well.
-
-
In what scenario would lan be source on your Wan? Your timenet_modem..
In what scenario would bogon ever be a source in your lan?
You have a default route of 192.168.197.1??? What networks are you trying to talk to each other? Setting a gateway on an interface makes it a WAN and nat will be done..
I would suggest you draw up your network and what is your wan and what is not a wan.. Do you have downstream networks? You have 65k hosts on your lan? Why else would you use a /16?? You have a MESS is what I see.. Why do you have a 10.42.2.2**/32**?
What kind of ISP uses rfc1918 space in their network after a public IP? Your 2 hops your calling isp stuff.
-
In what scenario would lan be source on your Wan? Your timenet_modem..
In what scenario would bogon ever be a source in your lan?
TIMENET_MODEM is a /30 I set up to manage one of the modems. It is going to have access control in the future, at the moment it is a simple LAN interface like any other. Nothing weird.
I since removed the bogon network rule.
You have a default route of 192.168.197.1??? What networks are you trying to talk to each other? Setting a gateway on an interface makes it a WAN and nat will be done..
192.168.197.1 is the system's default gateway because… it is the router I'm using as the default gateway for my LAN network. I would have used a /30 here as well, but this is how our ISP has set it up. Nothing weird going on, other than general "less-than-good ISP" stuff.
I would suggest you draw up your network and what is your wan and what is not a wan.. Do you have downstream networks? You have 65k hosts on your lan? Why else would you use a /16?? You have a MESS is what I see.. Why do you have a 10.42.2.2**/32**?
My network is documented fine. It is more complex than the usual SOHO network, it doesn't mean it is a mess, it means it serves a more complex purpose. I know what is WAN (EOLO_HLAN, which is a PPPoE, and TI_ADSL, which is Ethernet) and what is not (everything else). NAT works fine over both WANs, same for IPsec tunnels. No problems here.
The /16 is a temporary mask I have in place while I lay out the network numbering, and it surely is not the problem.
No idea about the 10.42.2.2/32 route, I noticed it too and it is on my investigation list, but it is surely not the problem as it is pointing to the correct gateway and network interface (and it doesn't serve a role in the issue I'm currently investigating).What kind of ISP uses rfc1918 space in their network after a public IP? Your 2 hops your calling isp stuff.
A small & local ISP. They probably haven't got enough money to buy enough IPv4 to number their internal network. It is really not a problem, just simply more "less-than-good ISP" stuff that could make investigating difficult in edge cases, not a worry atm.
I appreciate you taking the time to point out general defects of my configuration. I know it looks bad. That is partly because of pfSense's poor interface management UI (as acknowledge by gonzopacho with whom I discussed the matter a few weeks ago), partly because I have to work with whatever ISPs give me, and partly because there's still stuff I'm fixing. Also, in general, that is what a small/medium-sized business network configuration looks like.
Now that I hopefully cleared my network configuration, can we move on to the routing issue I'm having?
-
Your problem would be related to the rule you have on your lan that forces all traffic out this MGW_MAIN gateway.
How is suppose to get to your 10.41.0.0/30 on igb0 when the allow rule that allows traffic will force all its traffic out this MGW_MAIN gateway.
And your rule on your timenet_modem /30 network has your allowed source as your lan..
Your problem is not so much routing, but your firewall rules.
Rules are evaluated top down as traffic enters an interface.. First rule to trigger wins, no other rules are evaluated. So traffic trying to go to timenet from lan would be sent down the mgw_main gateway. Which not sure what that is from the info you provided. You need to put a rule above this rule that forces it out the gateway with no gateway set so pfsense can just use its routing table to get to the other network attached to pfsense.
Your traffic coming from your timenet network trying to go to lan woudln't go anywhere, because the source of the traffic would be from the timenet /30 not lan - your rule on timenet inteface says the source has to be lan net - which never going to be the case.
BTW - pfsense is not going to nat traffic between networks directly attached to it, unless you set it up to do so. It would only create nats for traffic going out what it considers a wan, ie has a gateway set on the interface.