Routing to wrong interface

[CyberTiVo]

pfSense 2.4.4 routing to wrong interface from LAN.

igb0: WAN, Public.IP, DHCP from carrier
igb1: LAN 172.18.1.62/26
igb2: TROY_LINK 172.18.0.253/24
igb3: TEMP_LINK 192.168.10.254/30

When I ping 192.168.10.253 from the firewall it works fine; If I ping from the LAN, the firewall sends the requests out the WAN interface:
/root: ping 192.168.10.253
PING 192.168.10.253 (192.168.10.253): 56 data bytes
64 bytes from 192.168.10.253: icmp_seq=0 ttl=64 time=0.678 ms
64 bytes from 192.168.10.253: icmp_seq=1 ttl=64 time=0.287 ms
64 bytes from 192.168.10.253: icmp_seq=2 ttl=64 time=0.277 ms
64 bytes from 192.168.10.253: icmp_seq=3 ttl=64 time=0.268 ms

tcpdump -ni igb0 host 192.168.10.253
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:08:14.495833 IP PUBLIC.IP > 192.168.10.253: ICMP echo request, id 49222, seq 0, length 64
19:08:15.500994 IP PUBLIC.IP > 192.168.10.253: ICMP echo request, id 49222, seq 1, length 64
19:08:16.503233 IP PUBLIC.IP > 192.168.10.253: ICMP echo request, id 49222, seq 2, length 64
19:08:17.506452 IP PUBLIC.IP > 192.168.10.253: ICMP echo request, id 49222, seq 3, length 64
19:08:18.507785 IP PUBLIC.IP > 192.168.10.253: ICMP echo request, id 49222, seq 4, length 64
19:08:19.510622 IP PUBLIC.IP > 192.168.10.253: ICMP echo request, id 49222, seq 5, length 64

It shows the correct route:
.local]/root: route get 192.168.10.252
route to: 192.168.10.252
destination: 192.168.10.252
mask: 255.255.255.252
fib: 0
interface: igb3
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0

I'm so confused

[Derelict]

https://docs.netgate.com/pfsense/en/latest/book/multiwan/index.html

[CyberTiVo]

Thanks for the quick reply; not quite sure what multi-wan has to do with this. I'm trying to get to a server on another subnet not trying to use TEMP_LINK as a WAN link

[Derelict]

Are you policy routing on the LANs? Setting gateways on rules there?

[CyberTiVo]

No

[Derelict]

You're right. That doesn't make sense. It looks like you have policy routing enabled.

Post your LAN rules.

[chpalmer]

What is the address of the client on the LAN you are trying to ping from? NM

I just tried this via the GUI and the only thing I saw was that I saw no packet captures to my interface but only pings through it.

I have a very similar /30 between two routers.

Does this happen to you when you use the GUI? Ill try via command line in a bit.

[CyberTiVo]

Pinging from 172.18.1.48

[CyberTiVo]

pinging fro the GUI works, if I select the source interface or automatic

[CyberTiVo]

Once I get this working, I plan to change it to a /31, not really necessary, just want too; does pfSense support /31, I see it as an option?

[CyberTiVo]

I figured it out! Somewhere along the way, things weren't working and I created a gateway for 192.168.10.252. Once I removed it, it started working; it was somewhat related to Multi-WAN; thanks for the help

[chpalmer]

I cannot reproduce this via command line either.

edit-
Glad you found it.

[CyberTiVo]

While it is working now, it got me thinking, there still seems to be a problem with the Multi-WAN routing. I know I am just a user but, seems you should be able to get to your secondary WAN device without having to disable your primary WAN link. There may be an architecture reason this will not work. Using my example above, shouldn't the routing logic know this is a "connected" network and send the packets to the igb3 interface before it decides to send it out the current "WAN"?

[johnpoz]

you can get to any network that is connected to pfsense as long as the firewall rules allow it. You can get to any network that is downstream via these networks as long as you setup a gateway and routes.

I have no idea what you did with your gateway setup - but you sure do not need to disable your wan to use a secondary path, etc.

Keep in mind there is a difference between creating simple routing path via a gateway and routes, and putting a gateway on an interface that makes a "wan" to pfsense which pfsense would by default nat too, etc. And now you would have to take into account your outbound nat settings.

Also policy routing comes into play if you set a gateway on a firewall rule, etc. etc.

[CyberTiVo]

Sounds like I need to create a separate rule on the LAN interface to use the specific gateway on igb3 instead of the Gateway group. I tried to add a route to 192.168.10.252 but, of course, pfSense would not let me, since it was connected. I will test when I get home and report back.

[JeGr]

@CyberTiVo said in Routing to wrong interface:

Once I get this working, I plan to change it to a /31, not really necessary, just want too; does pfSense support /31, I see it as an option?

BTW /31 makes no sense to me. /32 is host. /30 is 2 usable addresses. You can't use /31 without some point2point blackmagic in the UI IMHO.

Edit: Actually it's selectable. Don't know if your peer supports it though or if it works at all. Was never needed actually ;)

[Derelict]

Yes, pfSense supports /31.

[johnpoz]

don't really see the point though. Not like your short on ip space to use for your transit networks ;)

[Derelict]

It would make sense in something like a colo or metro-e environment. Or anywhere where RFC1918 is the exception not the rule. Which should be everywhere, actually.