Static routes ignored in 2.7.0 ?

johnpoz

@John888 ok doing the the best I can do to simulate your setup... I created a gateway and route on my 2.7 VM..

[2.7.0-RELEASE][admin@test.mydomain.tld]/root: netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.3.253      UGS         em0
127.0.0.1          link#4             UH          lo0
172.30.0.0/16      192.168.3.32       UGS         em0
192.168.3.0/24     link#1             U           em0
192.168.3.109      link#4             UHS         lo0
192.168.9.0/24     link#2             U           em1
192.168.9.34       link#4             UHS         lo0

Now when I do a traceroute to some IP behind that route.. Your right its going to the default gateway.. Not the route I setup?

[2.7.0-RELEASE][admin@test.mydomain.tld]/root: traceroute 172.30.1.1
traceroute to 172.30.1.1 (172.30.1.1), 64 hops max, 40 byte packets
 1  192.168.3.253 (192.168.3.253)  1.280 ms  1.136 ms  0.810 ms

That doesn't seem right... In the middle of watching football - and quite a few beers in.. But I wanted to respond real quick, so I remember to take a deeper look at this. But off the top the first hop you would think would be the new gateway.. hmmmm

John888

@johnpoz said in Static routes ignored in 2.7.0 ?:

(...) That doesn't seem right... In the middle of watching football - and quite a few beers in.. But I wanted to respond real quick, so I remember to take a deeper look at this. But off the top the first hop you would think would be the new gateway.. hmmmm

Thanks for checking it out so promptly - especially in the middle of a football match and after few pints, respect! ;)

Good to hear that it can be replicated. The first step to get it solved :).

johnpoz

@John888 ok real quick - this is odd.. it works if do lan interface, traceroute isn't going to work because the 192.168.9.100 is just some windows box IP..

But from sniff when doing the trace I see the traffic sent to the correct mac address.

Ethernet II, Src: MS-NLB-PhysServer-17_32:21:d9:c2 (02:11:32:21:d9:c2), Dst: Dell_0b:fd:16 (b0:4f:13:0b:fd:16)

Ethernet adapter Local:

   Connection-specific DNS Suffix  . : local.lan
   Description . . . . . . . . . . . : Killer E2600 Gigabit Ethernet Controller
   Physical Address. . . . . . . . . : B0-4F-13-0B-FD-16
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.9.100(Preferred)

[2.7.0-RELEASE][admin@test.mydomain.tld]/root: netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.3.253      UGS         em0
127.0.0.1          link#4             UH          lo0
172.30.0.0/16      192.168.3.32       UGS         em0
172.31.0.0/16      192.168.9.100      UGS         em1
192.168.3.0/24     link#1             U           em0
192.168.3.109      link#4             UHS         lo0
192.168.9.0/24     link#2             U           em1
192.168.9.34       link#4             UHS         lo0

Odd - but prob have to wait til tmrw to dig in.. I am quite a few beers in at this point ;) heheh

John888

@johnpoz said in Static routes ignored in 2.7.0 ?:

ok real quick - this is odd.. it works if do lan interface, traceroute isn't going to work because the 192.168.9.100 is just some windows box IP..

But from sniff when doing the trace I see the traffic sent to the correct mac address.

Will need to check mac addresses, but definitely the traffic is going through the default gateway, both from the pfsense itself, as well as from any hosts behind its LAN interface. Sending trace route hits first hop (default gateway) and the next one is public IP, which then obviously drops the packages.

EDIT: I'm actually wondering if this is pfSense or more FreeBSD related issue... Maybe worth checking if it is the same on 2.6.x ?

Derelict

@John888

When you want to do things like that you are going to be fighting the built-in route-to functionality in place on a pfSense WAN interface.

An interface is a WAN if it has an upstream gateway set on the interface itself.

All traffic for any destination other than the interface subnet (192.168.1.0/24 in your case) is forced out to that gateway regardless of the routes in the routing table.

Remove the gateway from the pfSense 192.168.1.100 interface. It can still be the default route. You might also have to add manual outbound NAT rules because it will no longer automatically be detected as a WAN.

John888

@Derelict Thanks for your answer. So it is a feature, and not a bug? If it is a bug though, any chance to get it corrected in some future pfSense software?

To be honest I would prefer to avoid doing such non-standard things like removing the gateway from WAN interface...

johnpoz

@John888 can't you just connect to this other router with some transit network you create?

Derelict

@John888 Not a bug. If you want to route on an interface with multiple routers, turn off route-to (remove the gateway from the interface).

You could also make a separate transit network to that second router on another interface and leave WAN alone.

The "non-standard" element in your configuration is two routers on the WAN interface.

John888

@johnpoz said in Static routes ignored in 2.7.0 ?:

can't you just connect to this other router with some transit network you create?

Seems I will need to think of something, but basically I do not control 192.168.1.1/24 network. It happens I also have no control which gateway is default, nor that I have 172.30/16 network behind another router in the same network. It is just given, real life situation.

John888

@Derelict said in Static routes ignored in 2.7.0 ?:

Not a bug. If you want to route on an interface with multiple routers, turn off route-to (remove the gateway from the interface).

You could also make a separate transit network to that second router on another interface and leave WAN alone.

The "non-standard" element in your configuration is two routers on the WAN interface.

It is for sure not common, but I would not call it "non-standard". Just a real life use-case I have to deal with.

Normally, when it comes to routing only, it is perfectly fine to have 0.0.0.0/0 route via one gateway, and use other gateways for accessing specific networks. It is actually very common scenario. Every known to me routing implementation will just use the most narrow choice and send traffic accordingly. In fact this is exactly what FreeBSD is doing in that case, as shown above - route -n show 172.30.222.1 is directing traffic via 192.168.1.200 as expected. Just then - as I understood - some higher pfSense layers kicks-in and override standard routing behaviour. If you do not consider it as a bug - fine - at least I know I need to find some other solution to my problem.

johnpoz

@Derelict said in Static routes ignored in 2.7.0 ?:

route-to

Where can read more about this? I can not seem to find any documentation about this.. In routing, the default route would only be used if there is not a more specific route. When you add a route for 172.30/16 that should be more specific than default and you would think that should be used other than the default..

Bob.Dig

@johnpoz https://docs.netgate.com/pfsense/en/latest/interfaces/wanvslan.html#wan-type-interface

The firewall adds route-to to automatic firewall rules for outbound traffic on a WAN type interface which ensures outbound traffic on the interface is sent to the configured gateway.

So effectively this means you can only have one gateway on a wan-type-interface.

johnpoz

@Bob-Dig yeah I see the route-to in the rules

pass out  route-to ( igb1 209.x.x.x) from 209.x.x.x to !209.x.x.0/20 ridentifier 1000013161 keep state allow-opts label "let out anything from firewall host itself"

But why is this set? I would think there should/could be some check box under advanced or something to not set the route-to

Bob.Dig

@johnpoz There can be only one.

Just make it a LAN and do stuff manually I guess.

Probably a good idea anyways if you put gateways on your "LAN".

johnpoz

@Bob-Dig well it doesn't effect me in any way shape or form, if I was going to add a 2nd wan, I would bring it in on its own connection, etc..

Just curious more than anything - why add the route-to to the rule?

Derelict

@johnpoz said in Static routes ignored in 2.7.0 ?:

I would think there should/could be some check box under advanced or something to not set the route-to

There is. Don't put an upstream gateway on the interface.

Derelict

@Bob-Dig said in Static routes ignored in 2.7.0 ?:

@johnpoz https://docs.netgate.com/pfsense/en/latest/interfaces/wanvslan.html#wan-type-interface

The firewall adds route-to to automatic firewall rules for outbound traffic on a WAN type interface which ensures outbound traffic on the interface is sent to the configured gateway.

So effectively this means you can only have one gateway on a wan-type-interface.

Yes. In the vast, vast majority of cases the WAN is an ISP-type connection. route-to makes things like multi-wan possible without a lot more user wrangling.

If more routers need to be placed on that subnet, remove the upstream gateway from the interface. If you switch to Manual Outbound NAT before you remove the upstream gateway, all of the necessary Outbound NAT rules will be automatically created for you.

The only other thing that changes that I can think of is you could enable a DHCP server out there if you wanted to. And probably RA and DHCP6.

It is a form of policy routing. It overrides the routing table. If you look at policy routing rules, they also have route-to inserted. That's the mechanism in pf that makes it all possible.

Bob.Dig

@johnpoz said in Static routes ignored in 2.7.0 ?:

well it doesn't effect me in any way shape or form, if I was going to add a 2nd wan, I would bring it in on its own connection, etc..

For me it kinda is.