Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    BGP convergence with BFD working smoothly with the settings below.

    Scheduled Pinned Locked Moved FRR
    20 Posts 3 Posters 895 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      michmoor LAYER 8 Rebel Alliance @mcury
      last edited by michmoor

      @mcury

      I think a simply fix for this would actually be an updated documentation believe it or not.

      System:Advanced:Firewall & NAT has the following

      84f8bc75-33e3-49e5-ac73-9db074b35dfa-image.png

      So in my mind, make a note here AND in the documentation to suggest disabling if using dynamic routing.

      @mcury I think you are right in that there isn't any harm disabling reply-to. Its added there as a benefit as most likely customers would need it but in advanced scenarios such as BGP routing and the possibility of traffic being asymmetric in nature, this MUST be disabled. There is no alternative.

      Great find my friend. Between this and helping me with Graylog i think i owe you some beers dude.

      Firewall: NetGate,Palo Alto-VM,Juniper SRX
      Routing: Juniper, Arista, Cisco
      Switching: Juniper, Arista, Cisco
      Wireless: Unifi, Aruba IAP
      JNCIP,CCNP Enterprise

      M 1 Reply Last reply Reply Quote 1
      • M
        mcury @michmoor
        last edited by mcury

        @michmoor said in BGP convergence with BFD working smoothly with the settings below.:

        Great find my friend. Between this and helping me with Graylog i think i owe you some beers dude.

        oh, its Friday :)
        🍻 🍻

        dead on arrival, nowhere to be found.

        M 1 Reply Last reply Reply Quote 1
        • M
          michmoor LAYER 8 Rebel Alliance @mcury
          last edited by michmoor

          @mcury
          Speaking to marcos on the side, there is a bit of nuance here.

          If the IPsec Filter Mode is changed from the default to 'Filter IPsec VTI and Transport on assigned interfaces' then reply-to gets applied by default. I think that is what ultimately breaks convergence. If using that mode then the fix is to do what we suggested. Also the state policy mode must be set to floating.

          There are just to many gotchas here and the only way this was discovered was through experimentation.

          In my humble opinion, a change needs to go in that if the IPsec Filter Mode is changed from the default then on the backend all rules created under the VTI Firewall tables have reply-to negated along with state policy mode changed to floating. Just a simple drop-down and apply.

          FRR has been broken for over a year, potentially longer. Im glad you solved it but this is a tad much to take into account if you are an network administrator who simply wants failover.

          Another added wrinkle is that if you do not use IPsec with BGP/OSPF and you simply have pfsense with multiple ISP providers doing BGP as i discovered, you must change the state policy mode to floating otherwise traffic gets blackholed.

          Again -- too many gotchas. Default configuration as used today will blackhole traffic if using FRR.

          Firewall: NetGate,Palo Alto-VM,Juniper SRX
          Routing: Juniper, Arista, Cisco
          Switching: Juniper, Arista, Cisco
          Wireless: Unifi, Aruba IAP
          JNCIP,CCNP Enterprise

          M 1 Reply Last reply Reply Quote 1
          • M
            mcury @michmoor
            last edited by

            If the IPsec Filter Mode is changed from the default to 'Filter IPsec VTI and Transport on assigned interfaces' then reply-to gets applied by default. I think that is what ultimately breaks convergence. If using that mode then the fix is to do what we suggested.

            Tested with both modes, both work with the no-reply option.

            Also the state policy mode must be set to floating.

            Indeed, this is how it is currently set here, but there is an option that does that automatically for IPsec rules if I'm not mistaken, check image posted in my answer below.

            There are just to many gotchas here and the only way this was discovered was through experimentation.

            A bunch of tests here, even using different state options, keep, none (creating outbound floating rules), sloppy..

            In my humble opinion, a change needs to go in that if the IPsec Filter Mode is changed from the default then on the backend all rules created under the VTI Firewall tables have reply-to negated along with state policy mode changed to floating. Just a simple drop-down and apply.

            I don't know what the best approach would be, perhaps give the option you mentioned and update the IPsec VTI documents highlighting this.

            FRR has been broken for over a year, potentially longer. Im glad you solved it but this is a tad much to take into account if you are an network administrator who simply wants failover.

            I agree.

            Another added wrinkle is that if you do not use IPsec with BGP/OSPF and you simply have pfsense with multiple ISP providers doing BGP as i discovered, you must change the state policy mode to floating otherwise traffic gets blackholed.

            I think that is already the default for IPsec VTI tunnels.

            e5f49c82-5025-4438-acce-477a6bfe1631-image.png

            Again -- too many gotchas. Default configuration as used today will blackhole traffic if using FRR.

            First thing I would do is to update the documentation with this workaround, specially in the OSPF/BGP section of the FRR.

            dead on arrival, nowhere to be found.

            M 1 Reply Last reply Reply Quote 0
            • M marcosm referenced this topic on
            • M
              michmoor LAYER 8 Rebel Alliance @mcury
              last edited by

              @mcury

              Ok so ive made some changes to my default configuration since you identified the issue.

              This pertains to me only but when i set up a new firewall AND the firewall is at the edge of the network i do the following

              1. If a single ISP, disable gateway monitoring action. The default is that its enabled but the problem is if there is a ISP hiccup, all packages and services gets restarted. I learned this the hard way. If i get packet loss even for a few seconds, all the VPN tunnels get restarted along with BGP. Why? Thats the gateway monitoring action.

              2. If pfsense is at the edge of the network and is doing OSPF/BGP
                a. Firewall State Policy gets changed to Floating States
                b. Disable reply-to is checked off globally.

              I wouldn't mind the defaults as they are but the problem is there is little to no documentation on how the defaults behave.

              Firewall: NetGate,Palo Alto-VM,Juniper SRX
              Routing: Juniper, Arista, Cisco
              Switching: Juniper, Arista, Cisco
              Wireless: Unifi, Aruba IAP
              JNCIP,CCNP Enterprise

              M 1 Reply Last reply Reply Quote 0
              • M
                mcury @michmoor
                last edited by

                @michmoor said in BGP convergence with BFD working smoothly with the settings below.:

                @mcury

                Ok so ive made some changes to my default configuration since you identified the issue.

                This pertains to me only but when i set up a new firewall AND the firewall is at the edge of the network i do the following

                1. If a single ISP, disable gateway monitoring action. The default is that its enabled but the problem is if there is a ISP hiccup, all packages and services gets restarted. I learned this the hard way. If i get packet loss even for a few seconds, all the VPN tunnels get restarted along with BGP. Why? Thats the gateway monitoring action.

                2. If pfsense is at the edge of the network and is doing OSPF/BGP
                  a. Firewall State Policy gets changed to Floating States
                  b. Disable reply-to is checked off globally.

                I wouldn't mind the defaults as they are but the problem is there is little to no documentation on how the defaults behave.

                Point number 1 is something that I'll start to do, didn't think about that before 👍
                Point number 2, I'm not sure if it should be disabled globally, I'm still trying to figure something that would get the benefits of reply-to but still give the user some warning about that specific scenario.

                I'm recording a video of lab working with the reply-to disabled.
                Soon I'll post it somewhere.

                dead on arrival, nowhere to be found.

                M 1 Reply Last reply Reply Quote 0
                • M
                  michmoor LAYER 8 Rebel Alliance @mcury
                  last edited by

                  @mcury said in BGP convergence with BFD working smoothly with the settings below.:

                  Point number 2, I'm not sure if it should be disabled globally, I'm still trying to figure something that would get the benefits of reply-to but still give the user some warning about that specific scenario.

                  Thats why i mentioned if pfsense is at the Edge of the network - internet facing and doing OSPF/BGP. In that case you are more than likely in a multi-wan scenrio so in my opinion disabling reply-to is OK.

                  Firewall: NetGate,Palo Alto-VM,Juniper SRX
                  Routing: Juniper, Arista, Cisco
                  Switching: Juniper, Arista, Cisco
                  Wireless: Unifi, Aruba IAP
                  JNCIP,CCNP Enterprise

                  M 1 Reply Last reply Reply Quote 0
                  • M
                    mcury @michmoor
                    last edited by

                    @michmoor said in BGP convergence with BFD working smoothly with the settings below.:

                    Thats why i mentioned if pfsense is at the Edge of the network - internet facing and doing OSPF/BGP. In that case you are more than likely in a multi-wan scenrio so in my opinion disabling reply-to is OK.

                    👍
                    Agreed.
                    Sent you a PM, hope you don't mind..

                    dead on arrival, nowhere to be found.

                    1 Reply Last reply Reply Quote 1
                    • A
                      andrew_cb
                      last edited by

                      Nice work @michmoor @mcury @marcosm !

                      I'm not using BGP or OSPF but your troubleshooting has been an interesting read.

                      M 1 Reply Last reply Reply Quote 1
                      • M
                        michmoor LAYER 8 Rebel Alliance @andrew_cb
                        last edited by

                        Redmine has been updated to reflect the testing done by @mcury so there is official guidance regarding treating this set up with dynamic routing.

                        I would take it a step further and make the declaration that if your firewall is running dynamic routing protocols (BGP/OSPF) then disable reply-to system wide and make sure you are using floating state policy. I wouldn't do it on a per-rule basis as that's not scalable - prone to error as you miss a rule with those advanced options set.

                        Firewall: NetGate,Palo Alto-VM,Juniper SRX
                        Routing: Juniper, Arista, Cisco
                        Switching: Juniper, Arista, Cisco
                        Wireless: Unifi, Aruba IAP
                        JNCIP,CCNP Enterprise

                        M 1 Reply Last reply Reply Quote 1
                        • M
                          mcury @michmoor
                          last edited by

                          @michmoor said in BGP convergence with BFD working smoothly with the settings below.:

                          Redmine has been updated to reflect the testing done by @mcury so there is official guidance regarding treating this set up with dynamic routing.

                          Glad to see that..
                          They even tested with HA.. Thanks @marcosm for testing. 👍 👍

                          dead on arrival, nowhere to be found.

                          1 Reply Last reply Reply Quote 1
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.