Packets take unexpected route

clarknova

2.4.5-RELEASE (amd64)

Client Interface: IPSEC
LAN IP: 10.13.1.43
AON: IPSEC networks to 10.0.0.0/8 via LAN IP

I have a couple of gateways on the LAN network with overlapping destination networks:

Destination        Gateway            Flags     Netif Expire
default            a.b.c.d            UGS        vmx0
10.0.0.0/8         10.13.1.1          UGS        vmx1
10.13.8.0/24       10.13.1.55         UGS        vmx1
###redacted###

I am able to reach most hosts on the 10.0.0.0/8 network just fine. The problem is that when I try to reach 10.13.8.6, 'tcpdump -e' shows that the packet is being passed to 10.13.1.1:

[2.4.5-RELEASE][root@vpn03.org.com]/root: tcpdump -enivmx1 dst port 443 and dst host 10.13.8.6
listening on vmx1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:26:22.291136 00:50:56:89:13:ef > 00:00:5e:00:01:83, ethertype IPv4 (0x0800), length 74: 10.13.1.43.10994 > 10.13.8.6.443: Flags [S], seq 459130270, win 64240, options [mss 1460,sackOK,TS val 767702107 ecr 0,nop,wscale 7], length 0

[2.4.5-RELEASE][root@vpn03.org.com]/root: arp -an | grep 00:00:5e:00:01:83
? (10.13.1.1) at 00:00:5e:00:01:83 on vmx1 expires in 838 seconds [ethernet]

I expected it to be passed to 10.13.1.55, since that destination has the longer mask.

In an attempt to remedy this, I created a firewall rule at the top of the IPSEC interface like this:

Pass [IPSEC networks] to 10.13.8.0/24 via [10.13.1.55 gateway]

Then I cleared all states for 10.13.8.6 and tried again to reach that host. Again, tcpdump showed the packets being passed to 10.13.1.1.

Have I misunderstood how this works? What do I need to do differently to be able to reach 10.13.8.0/24 from the IPSEC network?

clarknova

Attached screen caps of first firewall rule, routes and firewall logs. We can see that the rule was active while attempting to open https://10.13.8.6. Yet tcpdump still shows pfSense passing the ethernet frame to the MAC address of 10.13.1.1. So the firewall rule is matching, but the gateway and static route are not being applied.

clarknova

I've now disabled the static route for 10.0.0.0/8 and disabled 10.13.1.1 as a gateway. I can confirm this in a shell:

[2.4.5-RELEASE][root@vpn03.org.com]/root: netstat -rnfinet
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            w.x.y.z     UGS        vmx0
10.12.3.0/24       link#3             U          vmx2
10.12.3.43         link#3             UHS         lo0
10.13.1.0/24       link#2             U          vmx1
10.13.1.43         link#2             UHS         lo0
10.13.8.0/24       10.13.1.55         UGS        vmx1
a.b.c.d            w.x.y.z            UGHS       vmx0
e.f.g.h            w.x.y.z            UGHS       vmx0
i.j.k.l            w.x.y.z            UGHS       vmx0
127.0.0.1          link#5             UH          lo0
w.x.y.64/26        link#1             U          vmx0
wan.ip.address.65  link#1             UHS         lo0

Yet after I clear all states with 10.13.8 and try again tcpdump still shows frames going to 10.13.1.1. It's using a gateway that is not even in the routing table!

Crunk_Bass

Have you tried to disable negate rules?
See: https://docs.netgate.com/pfsense/en/latest/book/config/advanced-firewall-nat.html#disable-negate-rules

I had a similar problem with multi-wan routing and it seems to be working as expected after I disabled this.