pfSense using the wrong GW ethernet address.

tbyte

The kernel is 14.0-CURRENT FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_1-n255918-774957be06d: Wed Nov 15 17:41:06 UTC 2023
I have a simple routing table on pfSense 2.7.1 (10.10.1.104) :

Destination        Gateway            Flags     Netif Expire
default            10.10.1.100        UGS      vtnet0
10.10.1.0/24       link#1             U        vtnet0
10.10.1.104        link#4             UHS         lo0
10.10.16.0/23      10.10.1.102        UGS      vtnet0

And this in the arp table:

? (10.10.1.100) at 0c:c4:7a:90:7e:c4 on vtnet0 expires in 906 seconds [ethernet]
? (10.10.1.102) at ac:1f:6b:68:09:22 on vtnet0 expires in 396 seconds [ethernet]

But when I ping 10.10.1.104 (the pfSense) from 10.10.16.250 this happens:

19:08:34.487285 ac:1f:6b:68:09:22 > a2:b9:d4:a0:34:08, ethertype IPv4 (0x0800), length 98: 10.10.16.250 > 10.10.1.104: ICMP echo request, id 60167, seq 0, length 64
19:08:34.487372 a2:b9:d4:a0:34:08 > 0c:c4:7a:90:7e:c4, ethertype IPv4 (0x0800), length 98: 10.10.1.104 > 10.10.16.250: ICMP echo reply, id 60167, seq 0, length 64
19:08:35.550616 ac:1f:6b:68:09:22 > a2:b9:d4:a0:34:08, ethertype IPv4 (0x0800), length 98: 10.10.16.250 > 10.10.1.104: ICMP echo request, id 60167, seq 1, length 64
19:08:35.550678 a2:b9:d4:a0:34:08 > 0c:c4:7a:90:7e:c4, ethertype IPv4 (0x0800), length 98: 10.10.1.104 > 10.10.16.250: ICMP echo reply, id 60167, seq 1, length 64

For some reason pfSense sends the packet to the DEFAULT gateway's ethernet address (0c:c4:7a:90:7e:c4) and not to the actual gateway for 10.10.16.0/23 which is 10.10.1.102 (ac:1f:6b:68:09:22).
And this is the output of route get:

route to: 10.10.16.250
destination: 10.10.16.0
    mask: 255.255.254.0
 gateway: 10.10.1.102
     fib: 0
interface: vtnet0
   flags: <UP,GATEWAY,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
    0         0         0         0      1500         1         0

That is not supposed to happen and there are no other FIBs. Am I missing something?

tbyte

I just saw this
"pass in quick on vtnet0 reply-to (vtnet0 10.10.1.100) inet all flags S/SA keep state label "USER_RULE" label "id:1701228974" ridentifier 1701228974"

What is the point of the reply-to (vtnet0 10.10.1.100) because it seems like it might be a part of the problem?

viragomann

@tbyte
The reply-to is a mechanism used in pfSense to send response packets to the correct gateway.
It should only come into play if pfSense has no defined route for the source address.

So I"m wondering if pfSense really sees the origin source IP. Your packet capture does"t show the IPs.

tbyte

@viragomann I disabled this and now everything works. I fail to really understand what is the point of this rule.