No routing between subnets even with firewall disabled
-
Hey all, Long time user, first time poster.
========= HARDWARE Summary ==========
I have a small home setup with a single "server". Server was recently rebuilt, and it has two NICs now. One NIC I want on a totally separate subnet (192.168.2.0/24).My pfsense box is an old Dell SFF desktop with three physical interfaces. An onboard NIC (previously unused, now known as OPT2), and an Intel Dual-NIC (WAN/LAN). The server sits on the same shelf as the pfsense box, so I have a direct cable connection from the secondary interface on the server, to OPT2 on the PFsense box
========= PFSense Config ==========
The WAN (em0) interface has a public IP. ISP-provided.
The LAN (em1) interface has a subnet of 192.168.1.0/24 and has been working fine for years.
The OPT2 (em2) interface has a subnet of 192.168.2.0/24 and is the new subnet I added this week.
I also have an "ovpn1" interface, that I route specific IPs on 192.168.1.0/24 over, but that's not being used right now. Relic from my old server, but ultimately, I will define the ovpn1 as the only gateway to allow egress for this new subnet. But i'll cross that bridge when I get there.My problem, after HOURS of troubleshooting, is anything on the 192.168.2.0/24 cannot ping anything outside of that subnet. I can ping the gateway (192.168.2.1) no problem, but nothing beyond that. Nothing on the internet, nothing on the 192.168.1.1/24 subnet. I ruled out it being a firewall issue on the server by doing ping tests from the pfsense box.
DHCP is running on both interfaces, identical minus the subnet itself.
The OPT2 interface looks as I figure it should
And here is my routing table.
And here are the firewall rules used by both LAN and WAN. I even went nuclear and disabled the entire firewall in System > Advanced > Firewall & NAT and still wasn't able to ping anything.
Before I create a bald spot pulling out my hair...halp.
-
@tendonut said in No routing between subnets even with firewall disabled:
after HOURS of troubleshooting, is anything on the 192.168.2.0/24 cannot ping anything outside of that subnet. I can ping the gateway (192.168.2.1) no problem, but nothing beyond that. Nothing on the internet, nothing on the 192.168.1.1/24 subnet.
For problem pinging from the new subnet, check the outbound NAT. If it is working in automatic mode pfSense should have add 192.168.2.0/24 to the WAN rule. If it hasn't you have to add a rule manually.
Try to ping public IPs, not hostnames to rule out DNS issues.For pinging the other subnet, ensure the destination device does respond to ping request from outside its own subnet.
You can simply test it with the ping tool on pfSense. Change the source to another interface IP and look if you get a request. -
@viragomann I forgot to include my NAT table. I'm using manual outbound NAT because I was NATing specific IPs to specific outbound gateways (the VPN thing).
But here's what it looks like now.
Theoretically, I should be able to ping out using the PIAVPN gateway with this NAT rule. After screenshotting this, though, I added a 192.168.2.0/24 > WAN address on the WAN interface to (bottom of the list), and it didn't change anything. I was pinging IPs, not hostnames, as you suggested.
As for pinging the LAN subnet, I grabbed a half-dozen other IPs, including a printer, and am unable to ping them from the OPT2 subnet (using the PFSense ping utility). I AM able to ping them from the LAN interface (as I would expect)
-
Don't use manual outbound NAT, use Hybrid Outbound NAT instead.
-
@tendonut said in No routing between subnets even with firewall disabled:
Theoretically, I should be able to ping out using the PIAVPN gateway with this NAT rule.
From the point of outbound NAT, yes you should. However, outbound NAT does no more than NAT, it does not route the traffic. The routing to a non-default gateway has to be done by policy routing rule.
@tendonut said in No routing between subnets even with firewall disabled:
After screenshotting this, though, I added a 192.168.2.0/24 > WAN address on the WAN interface to (bottom of the list)
That's the one I am missing. However, it is correct??
You upper two rules are totally useless at all. They won't match any traffic, since the destination of packets will never be the subnet which is connected to the interface the rule is added to.
-
@bob-dig I just flipped it. It just added two additional rules, allowing all subnets to NAT with the WAN address. No change with my issue though.
I don't think I'd be able to keep this setting permanently though, because the way it reads, the automatic rules are read first, and I explicitly want to block the OPT2 subnet from being able to use the WAN interface at all, but for troubleshooting purposes, I'm OK with it.
-
@tendonut said in No routing between subnets even with firewall disabled:
I don't think I'd be able to keep this setting permanently though, because the way it reads, the automatic rules are read first, and I explicitly want to block the OPT2 subnet from being able to use the WAN interface at all
You are wrong.
Especially take note about this.
This can be generalized by making an alias for any RFC1918 traffic which would cover all private networks, and then using that in a rule. The alias contains 192.168.0.0/16, 172.16.0.0/12, and 10.0.0.0/8.
-
From the point of outbound NAT, yes you should. However, outbound NAT does no more than NAT, it does not route the traffic. The routing to a non-default gateway has to be done by policy routing rule.
That I understand. I stripped those rules out as I was troubleshooting. Previously, I'd both tag the packets with a "no_wan_egress" tag, and block that as a WAN rule, and also set up rules to block traffic from using the WAN gateway and explicitly allow the VPN one as rules on the respective interfaces. But like I said, those have been stripped out for the time being.
The rule I added is this:
I removed those top two rules. I agree, they didn't make any sense, just my multi-hour brain frying.
@Bob-Dig Then I will keep it at Hybrid :)
-
@tendonut said in No routing between subnets even with firewall disabled:
The rule I added is this:
This is for the source of LAN subnet, not OPT1.
-
I think you need something like this for your vpn-only interface.
Later create a VPN Killswitch.
-
@viragomann
Whoops! Like I said, brain fried. Same issue, though.
-
@bob-dig The RFC1918 alias is a good idea. I just created it. Didn't fix the issue, unfortunately, but I am gonna keep that in my back pocket when I start adding more elaborate rules.
-
I dropped off to go watch a movie, but am back now. One thing I noticed, is when I am doing a endless ping of www.google.com, going back to the dashboard traffic graph, I am seeing a pretty constant 50 B/s load on the VPN interface. So that tells me traffic IS going to the VPN interface from OPT2. You see exactly when I stopped the ping too.
