• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

VTI gateways not adding static routes in 24.03

IPsec
5
88
11.0k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic was forked from 24.03 causes issue with remote VPN stephenw10 May 15, 2024, 10:34 PM
This topic has been deleted. Only users with topic management privileges can see it.
  • O
    OhYeah 0 @stephenw10
    last edited by May 19, 2024, 10:54 AM

    @stephenw10 said in VTI gateways in 24.03:

    I mean if you edit the static route and resave it (without changing anything) does the route then appear?

    Nope...

    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by May 19, 2024, 1:12 PM

      Is there anything you can do to make the static routes return?

      Do you see any errors logged when you resaved the static route? In the System or Routing logs?

      O 1 Reply Last reply May 19, 2024, 9:42 PM Reply Quote 0
      • O
        OhYeah 0 @stephenw10
        last edited by May 19, 2024, 9:42 PM

        @stephenw10 said in VTI gateways in 24.03:

        Do you see any errors logged when you resaved the static route? In the System or Routing logs?

        I did see this bit in the "general" section of system logs after I resaved the static routes. These log entries repeated for every static route.

        May 19 13:32:14 php-fpm 54069 /system_routes_edit.php: Configuration Change: admin@xxx.xxx.xx.xx (Local Database): Saved static route configuration.
        May 19 13:32:14 check_reload_status 646 Syncing firewall
        May 19 13:32:16 php-fpm 594 /system_routes.php: Gateway, NONE AVAILABLE
        May 19 13:32:16 check_reload_status 646 Reloading filter

        PS. Obscured my IP address.

        1 Reply Last reply Reply Quote 0
        • O
          OhYeah 0 @LarryFahnoe
          last edited by May 19, 2024, 9:55 PM

          @LarryFahnoe said in VTI gateways in 24.03:

          I'm not meaning to hijack your thread, but it would appear we're both stumbling over the same (or related) bug: the static route for a remote network across an IPsec VTI is not being loaded.

          No no, I was actually relieved to find out that someone else had ran into the same issue.

          PS. When you resave the static route do you get the same messaged in system logs/general?

          L 1 Reply Last reply May 19, 2024, 10:04 PM Reply Quote 0
          • L
            LarryFahnoe @OhYeah 0
            last edited by May 19, 2024, 10:04 PM

            @OhYeah-0 Yes, same as the messages you show.

            May 19 16:02:36 pfs-m php-fpm[67932]: /system_routes_edit.php: Configuration Change: fahnoe@192.168.5.67 (Local Database): Saved static route configuration.
            May 19 16:02:36 pfs-m check_reload_status[645]: Syncing firewall
            May 19 16:02:36 pfs-m php-fpm[67932]: /system_routes_edit.php: Beginning configuration backup to https://acb.netgate.com/save
            May 19 16:02:40 pfs-m php-fpm[594]: /system_routes.php: Gateway, NONE AVAILABLE
            May 19 16:02:40 pfs-m check_reload_status[645]: Reloading filter
            
            1 Reply Last reply Reply Quote 0
            • S
              stephenw10 Netgate Administrator
              last edited by May 19, 2024, 10:19 PM

              No errors? Nothing in the Routing log?

              O L 2 Replies Last reply May 20, 2024, 6:03 AM Reply Quote 0
              • O
                OhYeah 0 @stephenw10
                last edited by May 20, 2024, 6:03 AM

                @stephenw10 said in VTI gateways in 24.03:

                No errors? Nothing in the Routing log?

                Nothing else apart from the "gateway not available" one.

                I booted the device back into 23.09 until a fix is found.

                1 Reply Last reply Reply Quote 0
                • L
                  LarryFahnoe @stephenw10
                  last edited by LarryFahnoe May 20, 2024, 11:55 AM May 20, 2024, 11:54 AM

                  @stephenw10 said in VTI gateways in 24.03:

                  No errors? Nothing in the Routing log?

                  No. This is in part why I opened the redmine and am trying to provide information. I believe my config to be quite simple: just a pair of 4200s with an IPsec VTI between them & and static routes to the LANs on either side, so I would have expected that others would be seeing the same thing. It sounds like @OhYeah-0 has a somewhat more complex config but is seeing a similar issue. That such a simple config (as mine is) that was working properly prior to the upgrade to 24.03 spells BUG to me.

                  Earlier I'd asked on the support thread about enabling debugging but got crickets. I see debug is set to false in /etc/inc/globals.inc and am tempted to turn that on. Is there a better or supported way to do that via the GUI somewhere? If so, I haven't found it.

                  --Larry

                  1 Reply Last reply Reply Quote 0
                  • O
                    OhYeah 0
                    last edited by May 20, 2024, 12:13 PM

                    I just remember that I installed another new Netgate 4100 for a new client and that device isn't actively being used, so I can use it for testing. It was immediately updated to 24.03 and it is showing exactly the same behavior.

                    I tried deleting the existing static route and re-create it, it is still not appearing in the routes table. No error messages in system logs -> routing.

                    My gut feeling is that the core reason of the bug is pfsense not considering 0.0.0.0/0 routing valid and thus not applying the static routes to the routes table.

                    L 1 Reply Last reply May 20, 2024, 12:20 PM Reply Quote 0
                    • L
                      LarryFahnoe @OhYeah 0
                      last edited by May 20, 2024, 12:20 PM

                      @OhYeah-0 As mentioned above, mine are using a /30 transit network rather than the 0.0.0.0/0 config you have, but we seem to be seeing the same thing: the static route doesn't load. My curious gut says: is there a timing issue where the tunnel hasn't come up yet which makes the static route seem invalid? Seems like the logs are not telling us the whole story though.

                      --Larry

                      O 1 Reply Last reply May 20, 2024, 12:30 PM Reply Quote 0
                      • O
                        OhYeah 0 @LarryFahnoe
                        last edited by OhYeah 0 May 20, 2024, 12:39 PM May 20, 2024, 12:30 PM

                        @LarryFahnoe said in VTI gateways in 24.03:

                        My curious gut says: is there a timing issue where the tunnel hasn't come up yet which makes the static route seem invalid?

                        IPSEC P1 instances have come online in both cases without problems for me.

                        EDIT: I think I might've have slightly misunderstood your point. It's an interesting thought that it could be a timing issue but I don't ever recall seeing such a problem with pfsense before.

                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by May 20, 2024, 1:00 PM

                          Yeah it seems likely it fails to add the route because the gateway is not yet available. If you have a dynamic gateway like that it won't show as up until the link is established.

                          However I would expect it to then be able to add routes after the VTI and hence the gateway is up.

                          Using 0.0.0.0/0 means there is not a dynamic gateway so that could be a problem. I'm not sure why that would be any different in 23.09 though.

                          But I'm surprised the route command doesn't throw an error.

                          Can you manually add a route at the CLI?

                          O L 2 Replies Last reply May 20, 2024, 1:07 PM Reply Quote 0
                          • O
                            OhYeah 0 @stephenw10
                            last edited by May 20, 2024, 1:07 PM

                            @stephenw10 said in VTI gateways in 24.03:

                            Using 0.0.0.0/0 means there is not a dynamic gateway so that could be a problem.

                            BTW, just to clarify: using 0.0.0.0/0 routing, the gateway IP always showed as "dynamic" in previous versions (in GUI under System -> Routing -> Gateways). In the dashboard it shows as "n/a" as before.

                            1 Reply Last reply Reply Quote 0
                            • S
                              stephenw10 Netgate Administrator
                              last edited by May 20, 2024, 1:10 PM

                              Yeah so to add a static route there it would need to be via the interface directly. I'd have to dig into the syntax to test that.

                              Do you know how that static route appeared in the routing table in 23.09?

                              O 1 Reply Last reply May 20, 2024, 1:21 PM Reply Quote 0
                              • L
                                LarryFahnoe @stephenw10
                                last edited by May 20, 2024, 1:13 PM

                                @stephenw10 said in VTI gateways in 24.03:

                                Can you manually add a route at the CLI?

                                Yes, but the question is when. With the system up and tunnel functioning I can add another route (to a bogus network for test):

                                # route add -net 192.168.15.0/24 192.168.8.2
                                add net 192.168.15.0: gateway 192.168.8.2
                                

                                I can reboot later this morning but the tunnel comes up immediately, so it will likely not throw any error by the time I log in to add the route via the CLI. Again, one one system, rc.newwanip triggers the route to be added about 15 min after reboot.

                                Any insight to offer on my question about enabling debugging?

                                --Larry

                                1 Reply Last reply Reply Quote 0
                                • S
                                  stephenw10 Netgate Administrator
                                  last edited by May 20, 2024, 1:20 PM

                                  Setting that would not give you any additional debug info AFAIK.

                                  Hmm, yet resaving the static route does not create the route which should run that exact same command.... 🤔

                                  L 1 Reply Last reply May 20, 2024, 2:29 PM Reply Quote 0
                                  • O
                                    OhYeah 0 @stephenw10
                                    last edited by May 20, 2024, 1:21 PM

                                    @stephenw10 said in VTI gateways in 24.03:

                                    Do you know how that static route appeared in the routing table in 23.09?

                                    DESTINATION - GATEWAY - FLAGS - USES - MTU - INTERFACE

                                    10.10.24.0/24 link#13 US 7 1400 ipsec2

                                    192.168.24.0/24 link#13 US 7 1400 ipsec2

                                    192.168.131.0/24 link#13 US 7 1400 ipsec2

                                    (From GUI: Diagnostics -> Routes)

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      stephenw10 Netgate Administrator
                                      last edited by May 20, 2024, 1:30 PM

                                      Right so via the link directly.

                                      L 1 Reply Last reply May 20, 2024, 3:34 PM Reply Quote 0
                                      • L
                                        LarryFahnoe @stephenw10
                                        last edited by May 20, 2024, 2:29 PM

                                        @stephenw10 said in VTI gateways in 24.03:

                                        Hmm, yet resaving the static route does not create the route which should run that exact same command

                                        Okay, so I just rebooted and then ssh'd in. The static route to 192.168.3.0/24 is missing, added it without issue. Lightly edited output removing the references to external addresses.

                                        # netstat -rn4
                                        Routing tables
                                        
                                        Internet:
                                        Destination        Gateway            Flags     Netif Expire
                                        127.0.0.1          link#6             UH          lo0
                                        192.168.0.2        link#6             UH          lo0
                                        192.168.5.0/24     link#3             U          igc2
                                        192.168.5.1        link#6             UHS         lo0
                                        192.168.8.1        link#6             UHS         lo0
                                        192.168.8.2        link#9             UH       ipsec1
                                        192.168.10.1       link#6             UH          lo0
                                        # 
                                        # route add -net 192.168.3.0/24 192.168.8.2
                                        add net 192.168.3.0: gateway 192.168.8.2
                                        # 
                                        # netstat -rn4
                                        Routing tables
                                        
                                        Internet:
                                        Destination        Gateway            Flags     Netif Expire
                                        127.0.0.1          link#6             UH          lo0
                                        192.168.0.2        link#6             UH          lo0
                                        192.168.3.0/24     192.168.8.2        UGS      ipsec1
                                        192.168.5.0/24     link#3             U          igc2
                                        192.168.5.1        link#6             UHS         lo0
                                        192.168.8.1        link#6             UHS         lo0
                                        192.168.8.2        link#9             UH       ipsec1
                                        192.168.10.1       link#6             UH          lo0
                                        

                                        From my position, the commonality here is that @OhYeah-0 and I both have systems with static routes not getting loaded. Beyond that there are variations on the theme:

                                        1. One of my systems does not get the static route on boot, but rc.newwanip triggers the route to be loaded about 15 min after boot
                                        2. Another of my systems now does get the static route loaded on boot, but this was a result of the steps Lev suggested in the redmine. I haven't been able to get Lev's steps to work on my other system
                                        3. It sounds like @OhYeah-0 has systems that do not get the static route loaded at all

                                        --Larry

                                        1 Reply Last reply Reply Quote 1
                                        • L
                                          LarryFahnoe @stephenw10
                                          last edited by May 20, 2024, 3:34 PM

                                          @stephenw10 said in VTI gateways in 24.03:

                                          Right so via the link directly.

                                          Hmm, so you've uncovered a new wrinkle, but I wonder if that might be due to @OhYeah-0 using the 0.0.0.0/0?

                                          I have yet to roll back to 23.09.1 and look at how the route was loaded. I would assume however that since I am using a /30 transit network, the route would be via the gateway IP I provided; not sure if an interface route would make sense if the user provides a gateway IP.

                                          Under 24.03 I did just add the route via the link & traffic passes as expected.

                                          # route del -net 192.168.3.0/24 192.168.8.2 
                                          del net 192.168.3.0: gateway 192.168.8.2
                                          # 
                                          # route add -net 192.168.3.0/24 -interface ipsec1
                                          add net 192.168.3.0: gateway ipsec1
                                          # 
                                          # netstat -rn4
                                          Routing tables
                                          
                                          Internet:
                                          Destination        Gateway            Flags     Netif Expire
                                          127.0.0.1          link#6             UH          lo0
                                          192.168.0.2        link#6             UH          lo0
                                          192.168.3.0/24     link#9             US       ipsec1
                                          192.168.5.0/24     link#3             U          igc2
                                          192.168.5.1        link#6             UHS         lo0
                                          192.168.8.1        link#6             UHS         lo0
                                          192.168.8.2        link#9             UH       ipsec1
                                          192.168.10.1       link#6             UH          lo0
                                          

                                          --Larry

                                          1 Reply Last reply Reply Quote 0
                                          32 out of 88
                                          • First post
                                            32/88
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.