Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Major issue with QUAGGA-OSPF and VLANs (pfsense 2.3.0)

    Scheduled Pinned Locked Moved Routing and Multi WAN
    81 Posts 23 Posters 34.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      After restarting services and yanking (virtual) cables I did manage to make it break, once.

      If it is related to restarting zebra, this patch might help:

      http://files.atx.pfsense.org/jimp/patches/skip_restart_for_routing_packages-2.3.1.patch

      Ultimately someone that can reproduce this reliably needs to report this directly to quagga since it appears to be a problematic change introduced in their 1.0.x code base.

      Remember: Upvote with the šŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • T
        Trey
        last edited by

        Hi,

        this problem is a real show stopper. Has nobody a config, that we can supply to the quagga team in order to fix the problem? This problem really sucks, as it is only showing itself from time to time…

        Dear pfsense team, what about a paid bugfix? What should it cost?!

        regards

        trey

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          We can't reliably reproduce it here, and it isn't our code to fix. It's something in Quagga 1.x on FreeBSD, so you'd be better off approaching the Quagga developers or maybe FreeBSD developers directly.

          Remember: Upvote with the šŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • R
            reqlez
            last edited by

            Has anybody opened a ticket with quagga yet ? Because I can easily reproduce it here just have to pull the main link cable at any one of the two sides of the link and it breaks.

            If nobody submitted I'll contact them when my projects settle down.

            1 Reply Last reply Reply Quote 0
            • H
              heper
              last edited by

              i don't think anyone submitted anything.

              1 Reply Last reply Reply Quote 0
              • E
                echu2016
                last edited by

                I don't have much experience in submitting bug reports and don't sincerely have any time for all the information/testing they require to accept them

                What I can say right now is that yesterday I upgraded to PFSense 2.3.2 and Quagga package also was upgraded to version 1.+, everything described here before has happened again. Reproducing the issue is quite easy. Just leave Quagga learn a few routes, then just click save or manually restart the service and you will see the routes duplicated. One with the preceding O and the preferred one with the preceding K label.
                Like This:

                O> 10.33.150.128/25 [110/20] via 192.168.45.1, em2, 01:38:55
                K>* 10.33.150.128/25Ā  via 192.168.45.1, em2

                If for some reason this dynamic route disappears or changes the next hop, the Kernel route would still be preferred and consequently the routing will be done incorrectly, like this:

                O> 10.33.150.128/25 [110/20] via 192.168.129.1, em2, 00:05:13
                K>* 10.33.150.128/25Ā  via 192.168.45.1, em2

                My solution again was rolling back to version 0.99 and locking the package to prevent further auto-updates.

                pkg lock quagga

                1 Reply Last reply Reply Quote 0
                • J
                  jmcentire
                  last edited by

                  @reqlez:

                  Has anybody opened a ticket with quagga yet ? Because I can easily reproduce it here just have to pull the main link cable at any one of the two sides of the link and it breaks.

                  If nobody submitted I'll contact them when my projects settle down.

                  Were you able to contact them?Ā  I was slamming my head against the wall for hours this weekend trying to figure out routing problems all over my network when I had a connection go down.

                  1 Reply Last reply Reply Quote 0
                  • R
                    rubic
                    last edited by

                    can it be related? https://lists.quagga.net/pipermail/quagga-dev/2016-February/014777.html

                    1 Reply Last reply Reply Quote 0
                    • R
                      reqlez
                      last edited by

                      @jimp:

                      After restarting services and yanking (virtual) cables I did manage to make it break, once.

                      If it is related to restarting zebra, this patch might help:

                      http://files.atx.pfsense.org/jimp/patches/skip_restart_for_routing_packages-2.3.1.patch

                      Ultimately someone that can reproduce this reliably needs to report this directly to quagga since it appears to be a problematic change introduced in their 1.0.x code base.

                      I saw somewhere in quagga notes that something got fixed recently. About this no restart patch. will it work on latest update ? also …. why not just include an option to TURN OFF restart of network packages ? somewhere in advanced options ? That would really help those unstable lines bringing the network down even if it's lower priority link while quagga reboots.

                      1 Reply Last reply Reply Quote 0
                      • T
                        Taras_
                        last edited by

                        @jimp:

                        We can't reliably reproduce it here, and it isn't our code to fix. It's something in Quagga 1.x on FreeBSD, so you'd be better off approaching the Quagga developers or maybe FreeBSD developers directly.

                        I have 2 fresh pfSenses (SG-4860) with 2 ISPs/4 OpenVPNs and OSPF on top of it.
                        This issue reliably reproduced :) :( :( , i.e. kernel routes aren't removed/updated properly (see 10.0.9.0/24 route):

                        Codes: K - kernel route, C - connected, S - static, R - RIP,
                        Ā  Ā  Ā   O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,
                        Ā  Ā  Ā   > - selected route, * - FIB route
                        
                        K>* 0.0.0.0/0 via 192.168.0.1, igb1
                        OĀ   10.0.9.0/24 [110/60] via 10.255.255.101, igb5, 00:02:03
                        K>* 10.0.9.0/24 via 10.255.2.2, ovpns1
                        OĀ   10.1.102.0/24 [110/50] via 10.255.2.2, ovpns1, 00:02:03
                        K>* 10.1.102.0/24 via 10.255.2.2, ovpns1
                        OĀ   10.11.11.0/24 [110/10] is directly connected, lagg0, 00:02:16
                        C>* 10.11.11.0/24 is directly connected, lagg0
                        OĀ   10.255.1.0/24 [110/70] via 10.255.2.2, ovpns1, 00:02:03
                        K>* 10.255.1.0/24 via 10.255.2.2, ovpns1
                        OĀ   10.255.2.0/24 [110/40] is directly connected, ovpns1, 00:02:16
                        C>* 10.255.2.0/24 is directly connected, ovpns1
                        OĀ   10.255.255.0/24 [110/50] is directly connected, igb5, 00:02:16
                        C>* 10.255.255.0/24 is directly connected, igb5
                        C>* 127.0.0.0/8 is directly connected, lo0
                        C>* 192.168.0.0/24 is directly connected, igb1
                        

                        My primary question is if Quagga introduced some problems in recent updates may be we should return to version which don't have problems and push it through pfsense's packages?
                        I have 2 support incidents from pfsense team, may be I should spent one of them on this problem?

                        1 Reply Last reply Reply Quote 0
                        • R
                          reqlez
                          last edited by

                          @jimp:

                          After restarting services and yanking (virtual) cables I did manage to make it break, once.

                          If it is related to restarting zebra, this patch might help:

                          http://files.atx.pfsense.org/jimp/patches/skip_restart_for_routing_packages-2.3.1.patch

                          Ultimately someone that can reproduce this reliably needs to report this directly to quagga since it appears to be a problematic change introduced in their 1.0.x code base.

                          Okay I tried this patch in 2.3.2 and it wont work …

                          Also I submitted a request in quagga-users lost nobody got back to me yet.

                          1 Reply Last reply Reply Quote 0
                          • R
                            reqlez
                            last edited by

                            Okay got a reply ( from Martin Winters the quagga god himself ! ) https://lists.quagga.net/pipermail/quagga-users/2016-October/014474.html

                            I actually contacted the maintainer of freebsd port for quagga and he referred me to the list as he doesn't think this is port related.

                            If you guys want to pitch in, go ahead… Martin is asking to compile latest code from git ... and honestly I have never complied zebra before, i think the last thing i complied on freebsd was java lol

                            1 Reply Last reply Reply Quote 0
                            • R
                              reqlez
                              last edited by

                              Here is another comment Martin from Quagga made: "I don’t see why pfsense would restart Quagga - so I think this might
                              be a bug. But there might be other reasons for it which I’m unaware
                              of."

                              I actually have some logs that I will be submitting either tonight or tomorrow.

                              1 Reply Last reply Reply Quote 0
                              • R
                                reqlez
                                last edited by

                                OSPFD / ZEBRA Debug logs submitted to Martin. Now we wait and see. I have tried his "latest" development package and it does the same thing.

                                1 Reply Last reply Reply Quote 0
                                • R
                                  reqlez
                                  last edited by

                                  So apparently -9 is a really nasty way of stopping Quagga, as per Martin from Quagga, and he thinks this is not letting it flush routing tables before exit. Maybe there is new code in new version of Quagga that takes a bit more time to flush those routes ? and maybe that is why it was not an issue in 0.99 version but it is with 1.0 ?

                                  See code in pfsense:

                                  rc_stop() {
                                  if [ -e /var/run/quagga/zebra.pid ]; then
                                  /bin/kill -9 /bin/cat /var/run/quagga/zebra.pid
                                  /bin/rm -f /var/run/quagga/zebra.pid
                                  fi
                                  if [ -e /var/run/quagga/ospfd.pid ]; then
                                  /bin/kill -9 /bin/cat /var/run/quagga/ospfd.pid
                                  /bin/rm -f /var/run/quagga/ospfd.pid
                                  fi
                                  }

                                  But then again, why is it being restarted in the first place? Is it because of links that get IPs dynamically allocated ?Ā  A UI option to skip quagga restart would be really appreciated guys! Pulling my hair out here testing this :(

                                  1 Reply Last reply Reply Quote 0
                                  • K
                                    kpa
                                    last edited by

                                    The -9 signal is always a bad idea on any service, it is strictly reserved for the situation where no other signal is able to terminate the process that is stuck for whatever reason. This should be common knowledge among pfSense developers and people working on the packages and I'm really surprised such amateur mistakes are being made with such an important package.

                                    1 Reply Last reply Reply Quote 0
                                    • R
                                      reqlez
                                      last edited by

                                      @kpa:

                                      The -9 signal is always a bad idea on any service, it is strictly reserved for the situation where no other signal is able to terminate the process that is stuck for whatever reason. This should be common knowledge among pfSense developers and people working on the packages and I'm really surprised such amateur mistakes are being made with such an important package.

                                      Here is my idea … why not have two waves of shutdowns ... first wave without -9Ā  Ā  then sleep for a few seconds and do another wave with -9 ? EvenĀ  better ... trigger the second wave only if there are any processes still running...

                                      1 Reply Last reply Reply Quote 0
                                      • K
                                        kpa
                                        last edited by

                                        It's still a very bad idea.

                                        http://unix.stackexchange.com/questions/281439/why-should-i-not-use-kill-9-sigkill

                                        Imagine a very big database that relies on proper shutdown for its integrity if the database has to be taken down. It has battery backed storage and UPS power and survives a power outage easily by performing the proper shutdown procedures when a power outage is detected and it can finish the procedures before the power really goes down. Now, if the main database process gets killed with -9 signal none of the shutdown processes get run because as in the linked document is described, "the process gets the rug pulled from it" and it's just removed forcibly from the system from the exact state it was when it was sent the -9 signal. This would leave that database in a inconsistent state and could cost days in repair time.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          Spydre13
                                          last edited by

                                          @reqlez:

                                          So apparently -9 is a really nasty way of stopping Quagga, as per Martin from Quagga, and he thinks this is not letting it flush routing tables before exit. Maybe there is new code in new version of Quagga that takes a bit more time to flush those routes ? and maybe that is why it was not an issue in 0.99 version but it is with 1.0 ?

                                          I see Martin's reply to you on Oct. 10, but I don't see anything after that.Ā  Are you emailing him off-list?

                                          I was looking through the Quagga code last night, and found something that I'm wondering whether or not could be the problem.Ā  Quagga (zebra daemon) puts routes into the kernel with flag "1" (RTF_PROTO1, see netstat man page).Ā  When zebra starts up it's supposed to ignore (filter out) any kernel routes with flag "1" because it should assume it put those there to begin with.Ā  I think before Quagga version 1 this was working, and in version >= 1 it pulls in those kernel routes into the zebra RIB.

                                          If I reboot a firewall and go to OSPF -> Status -> Zebra routes, I see a bunch of OSPF routes but barely any K (kernel) routes.Ā  If I make any change on the Global Settings or Interface Settings tab quagga restarts, and then when looking at the zebra routes it is filled with kernel routes (one for each OSPF route).

                                          Can you ask Martin to look at this:
                                          Commit: https://github.com/Quagga/quagga/commit/0d0686f98e64017415071e590bde262f0ab5a4c9
                                          File: zebra/zebra_rib.c
                                          Function: rib_sweep_table

                                          This function is commented out starting in version 1, but it was used in version 0.99.24.Ā  There is a block of code in it:

                                          
                                          if (rib->type == ZEBRA_ROUTE_KERNEL &&
                                          Ā  CHECK_FLAG (rib->flags, ZEBRA_FLAG_SELFROUTE))
                                          {
                                          Ā  Ā  ret = rib_uninstall_kernel (rn, rib);
                                          Ā  Ā  if (! ret)
                                          Ā  Ā  Ā  Ā  rib_delnode (rn, rib);
                                          }
                                          
                                          

                                          The rib_weed_tables function that is still being used doesn't seem to do this same thing, from what I can tell.Ā  This URL shows them side-by-side: https://fossies.org/diffs/quagga/0.99.24.1_vs_1.0.20160315/zebra/zebra_rib.c-diff.html

                                          If you can point me to the thread where you are discussing this with Martin, I can pass this along to him if you prefer.

                                          1 Reply Last reply Reply Quote 0
                                          • R
                                            reqlez
                                            last edited by

                                            Sorry I'm a mailing list noob and I just realized when you told me that this stuff is not going via lists … I'll post this and include the list this time, yes you are correct I been just emailing him

                                            @Spydre13:

                                            @reqlez:

                                            So apparently -9 is a really nasty way of stopping Quagga, as per Martin from Quagga, and he thinks this is not letting it flush routing tables before exit. Maybe there is new code in new version of Quagga that takes a bit more time to flush those routes ? and maybe that is why it was not an issue in 0.99 version but it is with 1.0 ?

                                            I see Martin's reply to you on Oct. 10, but I don't see anything after that.Ā  Are you emailing him off-list?

                                            I was looking through the Quagga code last night, and found something that I'm wondering whether or not could be the problem.Ā  Quagga (zebra daemon) puts routes into the kernel with flag "1" (RTF_PROTO1, see netstat man page).Ā  When zebra starts up it's supposed to ignore (filter out) any kernel routes with flag "1" because it should assume it put those there to begin with.Ā  I think before Quagga version 1 this was working, and in version >= 1 it pulls in those kernel routes into the zebra RIB.

                                            If I reboot a firewall and go to OSPF -> Status -> Zebra routes, I see a bunch of OSPF routes but barely any K (kernel) routes.Ā  If I make any change on the Global Settings or Interface Settings tab quagga restarts, and then when looking at the zebra routes it is filled with kernel routes (one for each OSPF route).

                                            Can you ask Martin to look at this:
                                            Commit: https://github.com/Quagga/quagga/commit/0d0686f98e64017415071e590bde262f0ab5a4c9
                                            File: zebra/zebra_rib.c
                                            Function: rib_sweep_table

                                            This function is commented out starting in version 1, but it was used in version 0.99.24.Ā  There is a block of code in it:

                                            	Ā  Ā  Ā  
                                            if (rib->type == ZEBRA_ROUTE_KERNEL &&
                                            Ā  CHECK_FLAG (rib->flags, ZEBRA_FLAG_SELFROUTE))
                                            {
                                            Ā  Ā  ret = rib_uninstall_kernel (rn, rib);
                                            Ā  Ā  if (! ret)
                                            Ā  Ā  Ā  Ā  rib_delnode (rn, rib);
                                            }
                                            
                                            

                                            The rib_weed_tables function that is still being used doesn't seem to do this same thing, from what I can tell.Ā  This URL shows them side-by-side: https://fossies.org/diffs/quagga/0.99.24.1_vs_1.0.20160315/zebra/zebra_rib.c-diff.html

                                            If you can point me to the thread where you are discussing this with Martin, I can pass this along to him if you prefer.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.