pfSense async routing doesn'follow route table

simon.cornet · Nov 13, 2023, 9:48 AM

Hello everyone,

When using multiple pfSense devices for routing purposes, I've noticed that messages don't always return to the same pfSense unit hosting the IPsec tunnel. A ping was set up to test this. Please refer to this image for clarification.

The following packet capture's have been taken during the test shown in Image 1.

Vpn terminator IPsec tunnel interface

09:51:21.858225 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 97, length 64
09:51:22.874421 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 98, length 64
09:51:23.875152 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 99, length 64
09:51:24.889981 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 100, length 64
09:51:25.913845 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 101, length 64
09:51:26.913278 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 102, length 64
09:51:27.929881 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 103, length 64
09:51:28.929319 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 104, length 64

Vpn terminator vlan 3000(192.168.2.0/24)

09:50:31.321945 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 47, length 64
09:50:32.322892 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 48, length 64
09:50:33.337867 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 49, length 64
09:50:34.339051 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 50, length 64
09:50:35.338478 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 51, length 64
09:50:36.350034 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 52, length 64
09:50:37.351806 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 53, length 64
09:50:38.362082 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 54, length 64

Fw02 vlan3000(192.168.2.0/24)

09:52:23.546101 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 158, length 64
09:52:23.546339 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 158, length 64
09:52:24.570128 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 159, length 64
09:52:24.570467 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 159, length 64
09:52:25.595317 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 160, length 64
09:52:25.595632 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 160, length 64
09:52:26.618187 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 161, length 64
09:52:26.618481 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 161, length 64
09:52:27.619457 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 162, length 64
09:52:27.619795 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 162, length 64
09:52:28.637458 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 163, length 64
09:52:28.637858 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 163, length 64
09:52:29.635625 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 164, length 64
09:52:29.635896 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 164, length 64
09:52:30.635462 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 165, length 64
09:52:30.635714 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 165, length 64

FW01 vlan 3000(192.168.2.0/24)

09:57:05.465419 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 87, length 64
09:57:06.465731 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 88, length 64
09:57:07.466265 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 89, length 64
09:57:08.466331 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 90, length 64
09:57:09.497255 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 91, length 64
09:57:10.497449 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 92, length 64
09:57:11.517532 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 93, length 64

The pfSense has the following routing tables.

VPN terminator

default 193.186.37.161 UGS 6 1500 vmx0
192.168.3.0.0/24 192.168.1.253 UGS 7 1500 vmx1
192.168.1.0/24 link#2 U 4 1500 vmx1
192.168.1.142 link#4 UHS 5 16384 lo0
192.168.2.0/24 192.168.1.253 UGS 7 1500 vmx1
10.130.36.0/24 192.168.1.253 UGS 7 1500 vmx1
80.239.143.134 193.186.37.161 UGHS 8 1500 vmx0
127.0.0.1 link#4 UH 2 16384 lo0
193.186.37.160/27 link#1 U 1 1500 vmx0
193.186.37.180 link#4 UHS 3 16384 lo0

Routing table fw02

Routes
default 192.168.1.254 UGS 140837 1500 em0
8.8.8.8 00:50:56:a5:5c:c6 UHS 90315 1500 em0
10.62.16.0/20 192.168.1.142 UGS 0 1500 em0
10.100.96.0/21 192.168.1.142 UGS 0 1500 em0
10.100.100.0/24 192.168.1.142 UGS 0 1500 em0
10.100.112.0/21 192.168.1.142 UGS 0 1500 em0
192.168.0.0/24 link#5 U 156707 1500 em4
192.168.0.253 link#5 UHS 0 16384 lo0
192.168.0.254 link#5 UHS 0 16384 lo0
192.168.3.0.0/24 link#4 U 32329259 1500 em3
192.168.3.0.251 link#4 UHS 0 16384 lo0
192.168.3.0.253 link#4 UHS 0 16384 lo0
192.168.3.0.254 link#4 UHS 0 16384 lo0
192.168.1.0/24 link#1 U 360529 1500 em0
192.168.1.252 link#1 UHS 0 16384 lo0
192.168.1.253 link#1 UHS 0 16384 lo0
192.168.2.0/24 link#3 U 411168 1500 em2
192.168.2.253 link#3 UHS 0 16384 lo0
192.168.2.254 link#3 UHS 0 16384 lo0
10.130.36.0/24 link#6 U 5221138 1500 em5
10.130.36.253 link#6 UHS 0 16384 lo0
10.130.36.254 link#6 UHS 0 16384 lo0
10.130.37.0/24 link#9 U 0 1500 em8
10.130.37.253 link#9 UHS 0 16384 lo0
10.130.37.254 link#9 UHS 0 16384 lo0
10.130.51.0/24 192.168.0.130 UGS 0 1500 em4
10.130.254.0/24 link#8 U 13955985 1500 em7
10.130.254.254 link#8 UHS 0 16384 lo0
10.211.1.0/24 192.168.1.142 UGS 0 1500 em0
10.212.134.0/24 192.168.1.254 UGS 0 1500 em0
10.254.253.0/29 192.168.3.0.250 UGS 0 1500 em3
10.254.253.8/29 192.168.3.0.250 UGS 0 1500 em3
127.0.0.1 link#11 UH 0 16384 lo0
191.16.10.0/24 192.168.3.0.250 UGS 0 1500 em3
191.16.20.0/24 192.168.3.0.250 UGS 0 1500 em3
191.17.96.0/24 192.168.1.142 UGS 0 1500 em0
191.17.200.0/24 192.168.1.142 UGS 0 1500 em0
191.17.113.0/24 192.168.1.142 UGS 0 1500 em0
191.17.121.0/24 192.168.1.142 UGS 180 1500 em0
191.17.123.0/24 192.168.1.142 UGS 0 1500 em0
192.168.100.0/24 192.168.3.0.250 UGS 0 1500 em3
192.168.101.0/24 192.168.3.0.250 UGS 0 1500 em3

Anyone have an idea why this is happening (and how to fix it)? We've been stumped so far.

viragomann · Nov 13, 2023, 1:10 PM

@simon-cornet
Is the gateway 192.168.1.142 monitored on fw02, and if so are there any regarding failures in the gateway log by any chance?

Royplaisier · Nov 13, 2023, 1:10 PM

@viragomann
The gateway is monitored, but there don't seem to be any failures on the gateway at the time of the test. A few days before, there were, though I think it's unrelated.

Nov 9 21:50:18 dpinger[81217]: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.1.142 bind_addr 192.168.1.252 identifier "vpn_terminator "

Nov 9 13:36:27 dpinger[72170]: vpn_terminator 192.168.1.142: Clear latency 765us stddev 4155us loss 5%

Royplaisier · Nov 13, 2023, 1:13 PM

@Royplaisier
By the way, I'm a colleague of Simon-cornet