Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense async routing doesn'follow route table

    Routing and Multi WAN
    3
    4
    386
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      simon.cornet
      last edited by

      Hello everyone,

      When using multiple pfSense devices for routing purposes, I've noticed that messages don't always return to the same pfSense unit hosting the IPsec tunnel. A ping was set up to test this. Please refer to this image for clarification.
      093380ac-c973-4794-aeec-4dc9b9c1bd6f-image.png

      The following packet capture's have been taken during the test shown in Image 1.

      Vpn terminator IPsec tunnel interface

      09:51:21.858225 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 97, length 64
      09:51:22.874421 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 98, length 64
      09:51:23.875152 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 99, length 64
      09:51:24.889981 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 100, length 64
      09:51:25.913845 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 101, length 64
      09:51:26.913278 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 102, length 64
      09:51:27.929881 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 103, length 64
      09:51:28.929319 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 104, length 64

      Vpn terminator vlan 3000(192.168.2.0/24)

      09:50:31.321945 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 47, length 64
      09:50:32.322892 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 48, length 64
      09:50:33.337867 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 49, length 64
      09:50:34.339051 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 50, length 64
      09:50:35.338478 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 51, length 64
      09:50:36.350034 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 52, length 64
      09:50:37.351806 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 53, length 64
      09:50:38.362082 (authentic,confidential): SPI 0xc01d9e6d: IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 54, length 64

      Fw02 vlan3000(192.168.2.0/24)

      09:52:23.546101 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 158, length 64
      09:52:23.546339 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 158, length 64
      09:52:24.570128 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 159, length 64
      09:52:24.570467 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 159, length 64
      09:52:25.595317 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 160, length 64
      09:52:25.595632 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 160, length 64
      09:52:26.618187 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 161, length 64
      09:52:26.618481 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 161, length 64
      09:52:27.619457 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 162, length 64
      09:52:27.619795 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 162, length 64
      09:52:28.637458 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 163, length 64
      09:52:28.637858 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 163, length 64
      09:52:29.635625 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 164, length 64
      09:52:29.635896 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 164, length 64
      09:52:30.635462 IP 191.17.200.63 > 192.168.2.201: ICMP echo request, id 63894, seq 165, length 64
      09:52:30.635714 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 63894, seq 165, length 64

      FW01 vlan 3000(192.168.2.0/24)

      09:57:05.465419 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 87, length 64
      09:57:06.465731 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 88, length 64
      09:57:07.466265 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 89, length 64
      09:57:08.466331 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 90, length 64
      09:57:09.497255 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 91, length 64
      09:57:10.497449 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 92, length 64
      09:57:11.517532 IP 192.168.2.201 > 191.17.200.63: ICMP echo reply, id 6798, seq 93, length 64

      The pfSense has the following routing tables.

      VPN terminator

      default 193.186.37.161 UGS 6 1500 vmx0
      192.168.3.0.0/24 192.168.1.253 UGS 7 1500 vmx1
      192.168.1.0/24 link#2 U 4 1500 vmx1
      192.168.1.142 link#4 UHS 5 16384 lo0
      192.168.2.0/24 192.168.1.253 UGS 7 1500 vmx1
      10.130.36.0/24 192.168.1.253 UGS 7 1500 vmx1
      80.239.143.134 193.186.37.161 UGHS 8 1500 vmx0
      127.0.0.1 link#4 UH 2 16384 lo0
      193.186.37.160/27 link#1 U 1 1500 vmx0
      193.186.37.180 link#4 UHS 3 16384 lo0

      Routing table fw02

      Routes
      default 192.168.1.254 UGS 140837 1500 em0
      8.8.8.8 00:50:56:a5:5c:c6 UHS 90315 1500 em0
      10.62.16.0/20 192.168.1.142 UGS 0 1500 em0
      10.100.96.0/21 192.168.1.142 UGS 0 1500 em0
      10.100.100.0/24 192.168.1.142 UGS 0 1500 em0
      10.100.112.0/21 192.168.1.142 UGS 0 1500 em0
      192.168.0.0/24 link#5 U 156707 1500 em4
      192.168.0.253 link#5 UHS 0 16384 lo0
      192.168.0.254 link#5 UHS 0 16384 lo0
      192.168.3.0.0/24 link#4 U 32329259 1500 em3
      192.168.3.0.251 link#4 UHS 0 16384 lo0
      192.168.3.0.253 link#4 UHS 0 16384 lo0
      192.168.3.0.254 link#4 UHS 0 16384 lo0
      192.168.1.0/24 link#1 U 360529 1500 em0
      192.168.1.252 link#1 UHS 0 16384 lo0
      192.168.1.253 link#1 UHS 0 16384 lo0
      192.168.2.0/24 link#3 U 411168 1500 em2
      192.168.2.253 link#3 UHS 0 16384 lo0
      192.168.2.254 link#3 UHS 0 16384 lo0
      10.130.36.0/24 link#6 U 5221138 1500 em5
      10.130.36.253 link#6 UHS 0 16384 lo0
      10.130.36.254 link#6 UHS 0 16384 lo0
      10.130.37.0/24 link#9 U 0 1500 em8
      10.130.37.253 link#9 UHS 0 16384 lo0
      10.130.37.254 link#9 UHS 0 16384 lo0
      10.130.51.0/24 192.168.0.130 UGS 0 1500 em4
      10.130.254.0/24 link#8 U 13955985 1500 em7
      10.130.254.254 link#8 UHS 0 16384 lo0
      10.211.1.0/24 192.168.1.142 UGS 0 1500 em0
      10.212.134.0/24 192.168.1.254 UGS 0 1500 em0
      10.254.253.0/29 192.168.3.0.250 UGS 0 1500 em3
      10.254.253.8/29 192.168.3.0.250 UGS 0 1500 em3
      127.0.0.1 link#11 UH 0 16384 lo0
      191.16.10.0/24 192.168.3.0.250 UGS 0 1500 em3
      191.16.20.0/24 192.168.3.0.250 UGS 0 1500 em3
      191.17.96.0/24 192.168.1.142 UGS 0 1500 em0
      191.17.200.0/24 192.168.1.142 UGS 0 1500 em0
      191.17.113.0/24 192.168.1.142 UGS 0 1500 em0
      191.17.121.0/24 192.168.1.142 UGS 180 1500 em0
      191.17.123.0/24 192.168.1.142 UGS 0 1500 em0
      192.168.100.0/24 192.168.3.0.250 UGS 0 1500 em3
      192.168.101.0/24 192.168.3.0.250 UGS 0 1500 em3

      Anyone have an idea why this is happening (and how to fix it)? We've been stumped so far.

      V 1 Reply Last reply Reply Quote 1
      • V
        viragomann @simon.cornet
        last edited by

        @simon-cornet
        Is the gateway 192.168.1.142 monitored on fw02, and if so are there any regarding failures in the gateway log by any chance?

        R 1 Reply Last reply Reply Quote 0
        • R
          Royplaisier @viragomann
          last edited by

          @viragomann
          The gateway is monitored, but there don't seem to be any failures on the gateway at the time of the test. A few days before, there were, though I think it's unrelated.

          Nov 9 21:50:18 dpinger[81217]: send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.1.142 bind_addr 192.168.1.252 identifier "vpn_terminator "

          Nov 9 13:36:27 dpinger[72170]: vpn_terminator 192.168.1.142: Clear latency 765us stddev 4155us loss 5%

          R 1 Reply Last reply Reply Quote 0
          • R
            Royplaisier @Royplaisier
            last edited by

            @Royplaisier
            By the way, I'm a colleague of Simon-cornet

            1 Reply Last reply Reply Quote 0
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.