Static routes ignored in 2.7.0 ?
-
Hi,
I have quite simple setup, with default route going via 192.168.1.1 gateway. I added another gateway - 192.168.1.200 - as I need to access 172.30/16 network, which is behind it. Both gateways are online, in the same VLAN, no issues pinging them. When I check routing table I get:
[2.7.0-RELEASE][root@pfsense.local]/root: netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.1.1 UGS igb1 (...) 172.30.0.0/16 192.168.1.200 UGS igb1 (...)also route to this specific destination seems to be ok:
[2.7.0-RELEASE][root@pfsense.local]/root: route -n show 172.30.222.1 route to: 172.30.222.1 destination: 172.30.0.0 mask: 255.255.0.0 gateway: 192.168.1.200 fib: 0 interface: igb1 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0But when I run traceroute, pfSense seems to be going via 192.168.1.1 default gateway:
[2.7.0-RELEASE][root@pfsense.local]/root: traceroute -n 172.30.222.1 traceroute to 172.30.222.1 (172.30.222.1), 64 hops max, 40 byte packets 1 192.168.1.1 2.430 ms 3.148 ms 2.332 ms (...)What is going on? Seems like pfSense is ignoring its own routing table...
I do not assume such a basic functionality could working wrongly on pfSense. What else am I missing though?
EDIT: quick topology diagram:
┌──────────────────────────┐ ┌────────────────────────────┐ │ │ │ │ │ PFSENSE WITH STATIC │ 192.168.1.0/24 │ DEFAULT GATEWAY │ │ ROUTING TABLE │ .100 ──────────────────────────┬──────────────► .1 │ 0.0.0.0/0 │ │ │ │ │ │ └──────────────────────────┘ │ └────────────────────────────┘ │ │ │ │ ▼ .200 ┌─────────────────────────────┐ │ │ │ SECOND GATEWAY │ │ 172.30.0.0/16 │ │ │ └─────────────────────────────┘ -
@John888 Probably not the best idea to have a second gateway on your local lan... But still, pfsense is only on .1, so I don't see the problem.
-
@John888 ok doing the the best I can do to simulate your setup... I created a gateway and route on my 2.7 VM..
[2.7.0-RELEASE][admin@test.mydomain.tld]/root: netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.3.253 UGS em0 127.0.0.1 link#4 UH lo0 172.30.0.0/16 192.168.3.32 UGS em0 192.168.3.0/24 link#1 U em0 192.168.3.109 link#4 UHS lo0 192.168.9.0/24 link#2 U em1 192.168.9.34 link#4 UHS lo0Now when I do a traceroute to some IP behind that route.. Your right its going to the default gateway.. Not the route I setup?
[2.7.0-RELEASE][admin@test.mydomain.tld]/root: traceroute 172.30.1.1 traceroute to 172.30.1.1 (172.30.1.1), 64 hops max, 40 byte packets 1 192.168.3.253 (192.168.3.253) 1.280 ms 1.136 ms 0.810 msThat doesn't seem right... In the middle of watching football - and quite a few beers in.. But I wanted to respond real quick, so I remember to take a deeper look at this. But off the top the first hop you would think would be the new gateway.. hmmmm
-
@johnpoz said in Static routes ignored in 2.7.0 ?:
(...) That doesn't seem right... In the middle of watching football - and quite a few beers in.. But I wanted to respond real quick, so I remember to take a deeper look at this. But off the top the first hop you would think would be the new gateway.. hmmmm
Thanks for checking it out so promptly - especially in the middle of a football match and after few pints, respect! ;)
Good to hear that it can be replicated. The first step to get it solved :).
-
@John888 ok real quick - this is odd.. it works if do lan interface, traceroute isn't going to work because the 192.168.9.100 is just some windows box IP..
But from sniff when doing the trace I see the traffic sent to the correct mac address.
Ethernet II, Src: MS-NLB-PhysServer-17_32:21:d9:c2 (02:11:32:21:d9:c2), Dst: Dell_0b:fd:16 (b0:4f:13:0b:fd:16)Ethernet adapter Local: Connection-specific DNS Suffix . : local.lan Description . . . . . . . . . . . : Killer E2600 Gigabit Ethernet Controller Physical Address. . . . . . . . . : B0-4F-13-0B-FD-16 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IPv4 Address. . . . . . . . . . . : 192.168.9.100(Preferred)
[2.7.0-RELEASE][admin@test.mydomain.tld]/root: netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 192.168.3.253 UGS em0 127.0.0.1 link#4 UH lo0 172.30.0.0/16 192.168.3.32 UGS em0 172.31.0.0/16 192.168.9.100 UGS em1 192.168.3.0/24 link#1 U em0 192.168.3.109 link#4 UHS lo0 192.168.9.0/24 link#2 U em1 192.168.9.34 link#4 UHS lo0Odd - but prob have to wait til tmrw to dig in.. I am quite a few beers in at this point ;) heheh
-
@johnpoz said in Static routes ignored in 2.7.0 ?:
ok real quick - this is odd.. it works if do lan interface, traceroute isn't going to work because the 192.168.9.100 is just some windows box IP..
But from sniff when doing the trace I see the traffic sent to the correct mac address.
Will need to check mac addresses, but definitely the traffic is going through the default gateway, both from the pfsense itself, as well as from any hosts behind its LAN interface. Sending trace route hits first hop (default gateway) and the next one is public IP, which then obviously drops the packages.
EDIT: I'm actually wondering if this is pfSense or more FreeBSD related issue... Maybe worth checking if it is the same on 2.6.x ?
-
When you want to do things like that you are going to be fighting the built-in route-to functionality in place on a pfSense WAN interface.
An interface is a WAN if it has an upstream gateway set on the interface itself.
All traffic for any destination other than the interface subnet (192.168.1.0/24 in your case) is forced out to that gateway regardless of the routes in the routing table.
Remove the gateway from the pfSense 192.168.1.100 interface. It can still be the default route. You might also have to add manual outbound NAT rules because it will no longer automatically be detected as a WAN.
-
@Derelict Thanks for your answer. So it is a feature, and not a bug? If it is a bug though, any chance to get it corrected in some future pfSense software?
To be honest I would prefer to avoid doing such non-standard things like removing the gateway from WAN interface...
-
@John888 can't you just connect to this other router with some transit network you create?
-
@John888 Not a bug. If you want to route on an interface with multiple routers, turn off route-to (remove the gateway from the interface).
You could also make a separate transit network to that second router on another interface and leave WAN alone.
The "non-standard" element in your configuration is two routers on the WAN interface.
Chattanooga, Tennessee, USA
A comprehensive network diagram is worth 10,000 words and 15 conference calls.
DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
Do Not Chat For Help! NO_WAN_EGRESS(TM) -
@johnpoz said in Static routes ignored in 2.7.0 ?:
can't you just connect to this other router with some transit network you create?
Seems I will need to think of something, but basically I do not control 192.168.1.1/24 network. It happens I also have no control which gateway is default, nor that I have 172.30/16 network behind another router in the same network. It is just given, real life situation.
-
@Derelict said in Static routes ignored in 2.7.0 ?:
Not a bug. If you want to route on an interface with multiple routers, turn off route-to (remove the gateway from the interface).
You could also make a separate transit network to that second router on another interface and leave WAN alone.
The "non-standard" element in your configuration is two routers on the WAN interface.
It is for sure not common, but I would not call it "non-standard". Just a real life use-case I have to deal with.
Normally, when it comes to routing only, it is perfectly fine to have 0.0.0.0/0 route via one gateway, and use other gateways for accessing specific networks. It is actually very common scenario. Every known to me routing implementation will just use the most narrow choice and send traffic accordingly. In fact this is exactly what FreeBSD is doing in that case, as shown above - route -n show 172.30.222.1 is directing traffic via 192.168.1.200 as expected. Just then - as I understood - some higher pfSense layers kicks-in and override standard routing behaviour. If you do not consider it as a bug - fine - at least I know I need to find some other solution to my problem.
-
@Derelict said in Static routes ignored in 2.7.0 ?:
route-to
Where can read more about this? I can not seem to find any documentation about this.. In routing, the default route would only be used if there is not a more specific route. When you add a route for 172.30/16 that should be more specific than default and you would think that should be used other than the default..
-
@johnpoz https://docs.netgate.com/pfsense/en/latest/interfaces/wanvslan.html#wan-type-interface
The firewall adds route-to to automatic firewall rules for outbound traffic on a WAN type interface which ensures outbound traffic on the interface is sent to the configured gateway.
So effectively this means you can only have one gateway on a wan-type-interface.
-
@Bob-Dig yeah I see the route-to in the rules
pass out route-to ( igb1 209.x.x.x) from 209.x.x.x to !209.x.x.0/20 ridentifier 1000013161 keep state allow-opts label "let out anything from firewall host itself"But why is this set? I would think there should/could be some check box under advanced or something to not set the route-to
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
@johnpoz There can be only one.
Just make it a LAN and do stuff manually I guess.
Probably a good idea anyways if you put gateways on your "LAN".
-
@Bob-Dig well it doesn't effect me in any way shape or form, if I was going to add a 2nd wan, I would bring it in on its own connection, etc..
Just curious more than anything - why add the route-to to the rule?
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
@johnpoz said in Static routes ignored in 2.7.0 ?:
I would think there should/could be some check box under advanced or something to not set the route-to
There is. Don't put an upstream gateway on the interface.
-
@Bob-Dig said in Static routes ignored in 2.7.0 ?:
@johnpoz https://docs.netgate.com/pfsense/en/latest/interfaces/wanvslan.html#wan-type-interface
The firewall adds route-to to automatic firewall rules for outbound traffic on a WAN type interface which ensures outbound traffic on the interface is sent to the configured gateway.
So effectively this means you can only have one gateway on a wan-type-interface.
Yes. In the vast, vast majority of cases the WAN is an ISP-type connection. route-to makes things like multi-wan possible without a lot more user wrangling.
If more routers need to be placed on that subnet, remove the upstream gateway from the interface. If you switch to Manual Outbound NAT before you remove the upstream gateway, all of the necessary Outbound NAT rules will be automatically created for you.
The only other thing that changes that I can think of is you could enable a DHCP server out there if you wanted to. And probably RA and DHCP6.
It is a form of policy routing. It overrides the routing table. If you look at policy routing rules, they also have route-to inserted. That's the mechanism in pf that makes it all possible.
-
@johnpoz said in Static routes ignored in 2.7.0 ?:
well it doesn't effect me in any way shape or form, if I was going to add a 2nd wan, I would bring it in on its own connection, etc..
For me it kinda is.
-
S stiff referenced this topic on
