floating rule works, interface rule does not
-
Here's some more context:
OSPF is enabled on all my switches and pfSense boxes. Routing tables look good.
This pfSense VM (rt-2) sits behind a physical pfSense box (rt-1) that connects to our ISPs. NAT is disabled on rt-2, and I've renamed the WAN interface as MO_PFSENSE. rt-2 has a default gw assigned. I think I've tried removing it but failed. I feel like that could be causing me problems.
MO_SVR interface has no gw assigned:
Traceroute shows same path in both directions (names have been obfuscated)
From server:
traceroute to rpi-2 (10.21.3.5), 30 hops max, 60 byte packets
1 rt-2 (10.20.3.1) 0.180 ms 0.137 ms 0.162 ms
2 dsw-1 (10.10.10.9) 0.384 ms 0.427 ms 0.424 ms
3 asw-1 (10.10.10.22) 0.630 ms 0.753 ms 0.735 ms
4 rpi-2 (10.21.3.5) 0.977 ms 0.951 ms 0.924 msFrom client:
traceroute to librenms (10.20.3.29), 30 hops max, 60 byte packets
1 asw-1 (10.21.3.1) 0.445 ms 0.550 ms 0.487 ms
2 dsw-1 (10.10.10.17) 0.714 ms 0.792 ms 0.734 ms
3 rt-2 (10.10.10.11) 0.677 ms 0.764 ms 0.706 ms
4 librenms (10.20.3.29) 1.201 ms !X 1.140 ms !X 1.083 ms !X -
it seems to be asymmetric
probably you are sending syn but the syn-ack answer is going out from somewhere else -
@acaronmd : OSPF is enabled on all my switches and pfSense boxes. Routing tables look good.
Are you sure the Layer 3 switching infrastructure is not also routing between those subnets?
That would certainly create a scenario where the firewall does not see all of the traffic - aka asymmetric routing.
Even the floating rule you say works doesn't look right. there should not be a portion of the state that is ESTABLISHED while the rest are FIN_WAIT_2.
-
Here's a rough diagram:
on rt-2:
O 0.0.0.0/0 [110/10] via 10.10.10.10, vmx0, 22:34:21 K>* 0.0.0.0/0 [0/0] via 10.10.10.10, vmx0, 22:34:32 O 10.10.10.8/29 [110/10] is directly connected, vmx0 inactive, 21:51:11 C>* 10.10.10.8/29 is directly connected, vmx0, 21:51:21 C>* 10.20.3.0/24 is directly connected, vmx1.103, 21:19:22 O>* 10.21.3.0/24 [110/10] via 10.10.10.9, vmx0, 21:51:10On dsw-1:
Destination Gateway VLAN Type Sub-Type Metric Dist. ------------------ --------------- ---- --------- ---------- ---------- ----- 0.0.0.0/0 10.10.10.10 2000 ospf External2 10 110 10.10.10.8/29 mo-pfsense 2000 connected 1 0 10.10.10.16/29 mo-tu 2021 connected 1 0 10.20.3.0/24 10.10.10.11 2000 ospf External2 20 110 10.21.3.0/24 10.10.10.22 2021 ospf External2 10 110On asw-1:
Destination Gateway VLAN Type Sub-Type Metric Dist. ------------------ --------------- ---- --------- ---------- ---------- ----- 0.0.0.0/0 10.10.10.17 2021 ospf External2 10 110 10.10.10.8/29 10.10.10.17 2021 ospf IntraArea 2 110 10.10.10.16/29 mo-tu 2021 connected 1 0 10.20.3.0/24 10.10.10.17 2021 ospf External2 20 110 10.21.3.0/24 tu-svr 103 connected 1 0 -
OK those are the zebra routes.
What is actually in the rtr-2 routing table?
netstat -rn4 -
Routing tables Internet: Destination Gateway Flags Netif Expire default 10.10.10.10 UGS vmx0 10.10.10.8/29 link#1 U vmx0 10.10.10.11 link#1 UHS lo0 10.10.10.16/29 10.10.10.9 UG1 vmx0 10.10.10.24/29 10.10.10.9 UG1 vmx0 10.10.10.32/29 10.10.10.9 UG1 vmx0 10.10.10.40/29 10.10.10.9 UG1 vmx0 10.10.55.0/24 10.10.10.10 UG1 vmx0 10.10.56.0/22 10.10.10.10 UG1 vmx0 10.15.5.2 10.10.10.10 UGH1 vmx0 10.20.0.0/24 link#8 U vmx1.200 10.20.0.1 link#8 UHS lo0 10.20.1.0/25 link#7 U vmx1.200 10.20.1.1 link#7 UHS lo0 10.20.2.0/26 10.10.10.10 UG1 vmx0 10.20.2.64/26 link#9 U vmx1.200 10.20.2.65 link#9 UHS lo0 10.20.2.128/26 10.10.10.10 UG1 vmx0 10.20.2.192/26 10.10.10.10 UG1 vmx0 10.20.3.0/24 link#14 U vmx1.103 10.20.3.1 link#14 UHS lo0 10.20.4.0/25 link#11 U vmx1.210 10.20.4.1 link#11 UHS lo0 10.20.5.0/24 10.10.10.10 UG1 vmx0 10.20.6.0/24 10.10.10.10 UGS vmx0 10.20.7.0/24 10.10.10.10 UGS vmx0 10.20.8.0/24 10.10.10.10 UG1 vmx0 10.20.10.0/24 10.10.10.10 UG1 vmx0 10.20.11.0/24 10.10.10.12 UG1 vmx0 10.21.0.1 10.10.10.9 UGH1 vmx0 10.21.1.0/24 10.10.10.9 UG1 vmx0 10.21.2.0/24 10.10.10.9 UG1 vmx0 10.21.3.0/24 10.10.10.9 UG1 vmx0 10.21.4.0/24 10.10.10.9 UG1 vmx0 10.21.5.0/24 10.10.10.9 UG1 vmx0 10.21.6.0/24 10.10.10.10 UGS vmx0 10.21.7.0/24 10.10.10.10 UGS vmx0 10.22.0.1 10.10.10.9 UGH1 vmx0 10.22.1.0/24 10.10.10.9 UG1 vmx0 10.22.2.0/24 10.10.10.9 UG1 vmx0 10.22.4.0/24 10.10.10.9 UG1 vmx0 10.22.5.0/24 10.10.10.9 UG1 vmx0 10.22.6.0/24 10.10.10.10 UGS vmx0 10.22.7.0/24 10.10.10.10 UGS vmx0 10.23.0.1 10.10.10.9 UGH1 vmx0 10.23.1.0/24 10.10.10.9 UG1 vmx0 10.23.2.0/24 10.10.10.9 UG1 vmx0 10.23.3.0/24 10.10.10.9 UG1 vmx0 10.23.4.0/24 10.10.10.9 UG1 vmx0 10.23.5.0/24 10.10.10.9 UG1 vmx0 10.23.6.0/24 10.10.10.10 UGS vmx0 10.23.7.0/24 10.10.10.10 UGS vmx0 24.142.24.0/21 10.10.10.10 UG1 vmx0 72.172.127.32/27 10.10.10.10 UG1 vmx0 127.0.0.1 link#4 UH lo0 192.168.32.0/30 10.10.10.10 UG1 vmx0 192.168.91.2 10.10.10.10 UGH1 vmx0 192.168.92.2 10.10.10.10 UGH1 vmx0 192.168.93.2 10.10.10.10 UGH1 vmx0 192.168.94.0/29 10.10.10.10 UG1 vmx0 192.168.96.2 10.10.10.10 UGH1 vmx0 192.168.97.6 10.10.10.10 UGH1 vmx0 192.168.97.10 10.10.10.10 UGH1 vmx0 192.168.97.14 10.10.10.10 UGH1 vmx0 192.168.98.2 10.10.10.10 UGH1 vmx0 -
Ah. Multiple routers on "WAN".
It doesn't work that way by default. The WAN interface has route-to to 10.10.10.10 for anything not in the WAN network, which in that case would be 10.10.10.8/29.
Looking at
/tmp/rules.debugwould be where I would go next.Your diagram shows what looks like multiple interfaces but they are not. They are all just vmx0. That is probably the issue.
You could also remove the gateway from the pfSense 10.10.10.8/29 interface (remove the gateway setting from the interface itself) and add a gateway in System > Routing for 10.10.10.10 on WAN and set it to be the default and see if that doesn't clear things up a little. You will still have a default route that way but you won't have route-to there and won't have automatic outbound NAT for it.
-
I believe this is it here:
pass out route-to ( vmx0 10.10.10.10 ) from 10.10.10.11 to !10.10.10.8/29 tracker 1000010111 keep state allow-opts label "let out anything from firewall host itself"the GW already exists
And here is the interface
When I get a chance, I'll try removing it from the interface itself. I seem to remember having issues trying to remove it when I tried long ago... if all else fails, I could add more interfaces to the VM.
-
BTW, as you might have noticed already, there's multiple VLANs assigned to the vmx1 interface, while the virtual machine has two interfaces.
-
Yeah that will eliminate it from having route-to. The gateway should still be there and should still be the default gateway but double-check it.
There are ways to bypass that using floating rules but in that situation I would just get rid of the gateway on the WAN interface and see how that goes.
-
@acaronmd said in floating rule works, interface rule does not:
BTW, as you might have noticed already, there's multiple VLANs assigned to the vmx1 interface, while the virtual machine has two interfaces.
VLAN interfaces are discrete as far as pfSense is concerned for routing purposes.
-
Thanks Derelict, you nailed it. Removed the upstream GW from the WAN, everything still working OK, so then disabled the floating rule, and the interface rule is working:
-
glad you solved, i was following the progress myself. I find that 3D of more experienced people are always instructive
-
Thanks kiokoman, you were on the right track.
-
That
FIN_WAIT_2:ESTABLISHEDis still strange. Maybe something to do with the app itself though. Glad it's working.
