Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense async routing doesn'follow route table

    Scheduled Pinned Locked Moved Routing and Multi WAN
    4 Posts 3 Posters 577 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      simon.cornet
      last edited by

      Hello everyone,

      When using multiple pfSense devices for routing purposes, I've noticed that messages don't always return to the same pfSense unit hosting the IPsec tunnel. A ping was set up to test this. Please refer to this image for clarification.
      093380ac-c973-4794-aeec-4dc9b9c1bd6f-image.png

      The following packet capture's have been taken during the test shown in Image 1.

      Vpn terminator IPsec tunnel interface

      09:51:21.858225 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 97, length 64
      09:51:22.874421 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 98, length 64
      09:51:23.875152 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 99, length 64
      09:51:24.889981 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 100, length 64
      09:51:25.913845 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 101, length 64
      09:51:26.913278 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 102, length 64
      09:51:27.929881 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 103, length 64
      09:51:28.929319 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 104, length 64

      Vpn terminator vlan 3000(192.168.2.0/24)

      09:50:31.321945 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 47, length 64
      09:50:32.322892 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 48, length 64
      09:50:33.337867 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 49, length 64
      09:50:34.339051 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 50, length 64
      09:50:35.338478 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 51, length 64
      09:50:36.350034 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 52, length 64
      09:50:37.351806 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 53, length 64
      09:50:38.362082 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 54, length 64

      Fw02 vlan3000(192.168.2.0/24)

      09:52:23.546101 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 158, length 64
      09:52:23.546339 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 158, length 64
      09:52:24.570128 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 159, length 64
      09:52:24.570467 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 159, length 64
      09:52:25.595317 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 160, length 64
      09:52:25.595632 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 160, length 64
      09:52:26.618187 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 161, length 64
      09:52:26.618481 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 161, length 64
      09:52:27.619457 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 162, length 64
      09:52:27.619795 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 162, length 64
      09:52:28.637458 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 163, length 64
      09:52:28.637858 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 163, length 64
      09:52:29.635625 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 164, length 64
      09:52:29.635896 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 164, length 64
      09:52:30.635462 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 165, length 64
      09:52:30.635714 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 165, length 64

      FW01 vlan 3000(192.168.2.0/24)

      09:57:05.465419 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 87, length 64
      09:57:06.465731 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 88, length 64
      09:57:07.466265 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 89, length 64
      09:57:08.466331 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 90, length 64
      09:57:09.497255 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 91, length 64
      09:57:10.497449 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 92, length 64
      09:57:11.517532 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 93, length 64

      The pfSense has the following routing tables.

      VPN terminator

      default 193.186.37.161 UGS 6 1500 vmx0
      192.168.3.0.0/24 192.168.1.253 UGS 7 1500 vmx1
      192.168.1.0/24 link#2 U 4 1500 vmx1
      192.168.1.142 link#4 UHS 5 16384 lo0
      192.168.2.0/24 192.168.1.253 UGS 7 1500 vmx1
      10.130.36.0/24 192.168.1.253 UGS 7 1500 vmx1
      80.239.143.134 193.186.37.161 UGHS 8 1500 vmx0
      127.0.0.1 link#4 UH 2 16384 lo0
      193.186.37.160/27 link#1 U 1 1500 vmx0
      193.186.37.180 link#4 UHS 3 16384 lo0

      Routing table fw02

      Routes
      default 192.168.1.254 UGS 140837 1500 em0
      8.8.8.8 00:50:56:a5:5c:c6 UHS 90315 1500 em0
      10.62.16.0/20 192.168.1.142 UGS 0 1500 em0
      10.100.96.0/21 192.168.1.142 UGS 0 1500 em0
      10.100.100.0/24 192.168.1.142 UGS 0 1500 em0
      10.100.112.0/21 192.168.1.142 UGS 0 1500 em0
      192.168.0.0/24 link#5 U 156707 1500 em4
      192.168.0.253 link#5 UHS 0 16384 lo0
      192.168.0.254 link#5 UHS 0 16384 lo0
      192.168.3.0.0/24 link#4 U 32329259 1500 em3
      192.168.3.0.251 link#4 UHS 0 16384 lo0
      192.168.3.0.253 link#4 UHS 0 16384 lo0
      192.168.3.0.254 link#4 UHS 0 16384 lo0
      192.168.1.0/24 link#1 U 360529 1500 em0
      192.168.1.252 link#1 UHS 0 16384 lo0
      192.168.1.253 link#1 UHS 0 16384 lo0
      192.168.2.0/24 link#3 U 411168 1500 em2
      192.168.2.253 link#3 UHS 0 16384 lo0
      192.168.2.254 link#3 UHS 0 16384 lo0
      10.130.36.0/24 link#6 U 5221138 1500 em5
      10.130.36.253 link#6 UHS 0 16384 lo0
      10.130.36.254 link#6 UHS 0 16384 lo0
      10.130.37.0/24 link#9 U 0 1500 em8
      10.130.37.253 link#9 UHS 0 16384 lo0
      10.130.37.254 link#9 UHS 0 16384 lo0
      10.130.51.0/24 192.168.0.130 UGS 0 1500 em4
      10.130.254.0/24 link#8 U 13955985 1500 em7
      10.130.254.254 link#8 UHS 0 16384 lo0
      10.211.1.0/24 192.168.1.142 UGS 0 1500 em0
      10.212.134.0/24 192.168.1.254 UGS 0 1500 em0
      10.254.253.0/29 192.168.3.0.250 UGS 0 1500 em3
      10.254.253.8/29 192.168.3.0.250 UGS 0 1500 em3
      127.0.0.1 link#11 UH 0 16384 lo0
      191.16.10.0/24 192.168.3.0.250 UGS 0 1500 em3
      191.16.20.0/24 192.168.3.0.250 UGS 0 1500 em3
      191.17.96.0/24 192.168.1.142 UGS 0 1500 em0
      191.17.200.0/24 192.168.1.142 UGS 0 1500 em0
      191.17.113.0/24 192.168.1.142 UGS 0 1500 em0
      191.17.121.0/24 192.168.1.142 UGS 180 1500 em0
      191.17.123.0/24 192.168.1.142 UGS 0 1500 em0
      192.168.100.0/24 192.168.3.0.250 UGS 0 1500 em3
      192.168.101.0/24 192.168.3.0.250 UGS 0 1500 em3

      Anyone have an idea why this is happening (and how to fix it)? We've been stumped so far.

      V 1 Reply Last reply Reply Quote 1
      • V Offline
        viragomann @simon.cornet
        last edited by

        @simon-cornet
        Is the gateway 192.168.1.142 monitored on fw02, and if so are there any regarding failures in the gateway log by any chance?

        R 1 Reply Last reply Reply Quote 0
        • R Offline
          Royplaisier @viragomann
          last edited by

          @viragomann
          The gateway is monitored, but there don't seem to be any failures on the gateway at the time of the test. A few days before, there were, though I think it's unrelated.

          Nov 9 21:50:18 dpinger[81217]: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.1.142 bind_addr 192.168.1.252 identifier "vpn_terminator "

          Nov 9 13:36:27 dpinger[72170]: vpn_terminator 192.168.1.142: Clear latency 765us stddev 4155us loss 5%

          R 1 Reply Last reply Reply Quote 0
          • R Offline
            Royplaisier @Royplaisier
            last edited by

            @Royplaisier
            By the way, I'm a colleague of Simon-cornet

            1 Reply Last reply Reply Quote 0
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.