Updating to pfSense+ 24.3 breaks routing - kernel routes now gone
-
The update from pfSense+ 23.09.1 to 24.3 immediately broke my network. Note that the update completed successfully.
Upon investigation I saw that kernel routes have gone missing in the FRR 9.1 included with 24.3, but were present in the FRR 9.0.2 included in pfSense+ 23.09.01.
These kernel routes are visible to the system in both old and new versions of pfSense, when viewed with "netstat" (netstat -rn4). Note that for this output, the v23.09.1 and v24.03 outputs are identical, so no need to include a diff.
The reason why my network broke was that there was another route in the FRR "show ip route" - an OSPF-learned default route - which is not shown here as I have since removed it. It was a default route on another router in another part of the network, which had a high OSPF cost, and is used as an alternative path in case the whole firewall dies. It is a low-bandwidth link and should only be used in an emergency. When I upgraded the firewall to pfSense+ 24.3 from 23.09.1, it would then direct all traffic to this other low-bandwidth gateway, instead of sending it straight out its own WAN IP to the ISP.
I have now taken this alternative path's OPSF-learned default route out of the network, so that the firewall can function again.
I suspect that these kernel routes disappearing from FRR is why pfSense preferred some distant OSPF default route over its directly connected default route. I am now relying on the functionality that if FRR doesn't have a route in its routing table, then it defers to the system's routing table, and you can see with the "netstat -rn4" that it does have all the kernel routes.
But relying on this functionality is not robust. It should be fine to have a backup default route in OSPF costed out as to not break anything, but be there to take over in case of an emergency. I shouldn't have to remove that. with the age-old concept of "administrative distance", the firewall should always prefer a directly connected default route, other a high-cost OSPF route.
We really do need these kernel routes back in FRR, so that the FRR can have a proper and complete view of system routing logic. Therefore, I consider this a bug, and not a feature.
-
I have been doing some tests on my GNS3 lab, where I've replicated the production network.
On 23.09.1, no matter whether I advertise the secondary default route or not, the "netstat -rn" only shows one default route out the WAN1 link, and the "show ip route" in FRR only shows the one default route, and there are the 4 kernel routes.
On 24.3, I've had either none, three or all four kernel routes show up, but i have to bounce the WAN links and shake things up a bit to get things to change. There does seem like some kind of race condition going on with the learning and installing of routes.
When I have the secondary remote default route advertised, I end up seeing it in the "show ip route" display on 24.3 but not on 23.09.1 which has the kernel default route instead. Example of the route that shows up on 24.3 is:
O>* 0.0.0.0/0 [110/5] via 10.254.40.2, vmx1.40, weight 1, 00:07:47And in netstat, I see two default routes.
Destination Gateway Flags Nhop# Mtu Netif Expire
default <ISP's WAN IP> UGS 0 1500 vmx2
default 10.254.40.2 UG1 0 1500 vmx1.40It looks like it's load balancing, because some traceroutes work, and others ping-pong towards the firewall and back:
GT_Data> trace 1.2.3.4
trace to 1.2.3.4, 8 hops max, press Ctrl+C to stop
1 192.168.27.1 0.703 ms 0.554 ms 0.605 ms
2 10.254.40.1 1.323 ms 1.261 ms 1.767 ms
3 192.168.27.1 1.953 ms 1.248 ms 1.116 ms
4 10.254.40.1 2.337 ms 2.688 ms 1.876 ms
5 192.168.27.1 1.814 ms 2.516 ms 1.852 ms
6 10.254.40.1 3.325 ms 2.818 ms 2.670 ms
7 192.168.27.1 2.532 ms 3.280 ms *
8 10.254.40.1 4.716 ms 4.612 ms 2.889 msetc etc. There's where the network breaks, as per the title of this thread. Doesn't happen in 23.09.1.
On 23.09.1, the routing always converges to the same view no matter how or when I bounce various interfaces, but in 24.3, the "show ip route" can have different results based on when various interfaces are bounced and different things happen.
What's not satisfactory is how pfSense+ is load-balancing a directly connected default route, with one learned via OSPF. The connected one should always take precedence due to lower administrative distance.
[https://docs.frrouting.org/en/latest/zebra.html#administrative-distance](link url)System, kernel and connected routes should win out over OSPF routes.
Just looking at FRR source files, there are no build instructions for FreeBSD 15 - only up to FreeBSD 14. Is FreeBSD 15 too experimental for third party packages?!
The FRR package installed is "pfSense-pkg-frr-2.0.2_3". Is there a beta package I could test which contains FRR 9.1.1 or even 10.0 that I could test, to see if this issue is still in the newer versions?
-
I have the same issue here after upgrading from 23.09.1 to 24.03.1. Kernel routes just disappeared from FRR. The main problem is that I have distributed the default route to the network through OSPF. So, I cannot do it anymore and my network is broken.
-
W wellsouz referenced this topic on
-
@wellsouz Did you guys open a redmine ticket for this issue ?
-
-
@wellsouz Thank you for opening a ticket. Up until this point I figured I was the only one. When I'm a lone voice in the wilderness (no forum replies), I tend to just wait in the hope that it's been found and fixed elsewhere, retesting with each subsequent release.
I am still on 23.09.1 in production (to keep my network redundancy), but have a GNS3 lab mimicking my production network which can replicate the issue. I can do some tests if there's a beta build of pfSense/FRR recommended to re-test with.
-
I have posted on the ticket asking if the issue is at least confirmed? It doesn't look to be. It is a serious regression which needs to be assigned / confirmed / worked on / remedied. Now Netgate are posting about 24.08, and this serious issue hasn't even been looked at? I really don't think Netgate have enough resources for proper FRR support. I mean look how many tickets there are for FRR, and how old many of the tickets are. Regular maintenance on FRR tickets isn't even being done.
-
In case someone experiencing this issue would like to test with the latest FRR release, I've attached frr9.1.1 for amd64 platforms:
Extract and upload the files using Diagnostics > Command Prompt, stop FRR, then install it like so:
pkg install -fy /tmp/frr9-pythontools-9.1.1.pkg pkg install -fy /tmp/frr9-9.1.1.pkg -
@marcosm Thanks Marcos. I will try to give this a test to see
@Gcon In fairness here...If the problem is in the package then the problem is in upstream FreeBSD and if the problem is there then the problem needs to be corrected by the mainteners. Although I understand the frustration, netgate doesn't seem to be at fault in the regression. They are only pulling down the latest package updates.
That said. I do highly recommend postponing pfSense upgrades until this is rectified.
-
@michmoor I understand that Netgate don't develop the FRR code, but they are responsible for rolling it out to customers. Compare the following:
pfSense+ 23.09.1. = FRR package 2.0.2_1 / FRR 9.0.1
pfSenseCE 2.7.2 = FRR package 2.0.2_1 / FRR 9.0.1 <- Current CE version
pfSense+ 24.03 = FRR package 2.0.2_3 / FRR 9.1 <- Current Plus versionThe 9.0.1 was working fine for me, but then who got to be the crash test dummy for the FRR 9.1 first? Plus users, not CE users!
If you are insinuating that since Netgate don't develop FRR (they merely package it), that they are therefor not culpable - I would say this is a logical fallacy. It's like saying the woman who gave Alec Baldwin a loaded gun which fatally shot someone on the set of that Western movie isn't culpable because well - she didn't manufacture the gun and bullets.
So yeah Netgate - test more thoroughly before release, and I think even more importantly - let the community edition users shoot themselves in the foot first before paying customers do. Business users prioritise stability/reliability over features and the "latest and greatest". Otherwise - why bother having CE at all? Using Plus as the guinea pigs for CE just seems like a cowboy operation (the "Wild West" metaphor just keeps on giving).
Anyway I will test FRR 9.1.1, and hopefully don't have to call 911
-
@Gcon I am using version frr9-9.1_2 and on pfsense version 24.08.a.20240702.0600 and this is still an issue.
-
@Gcon said in Updating to pfSense+ 24.3 breaks routing - kernel routes now gone:
f you are insinuating that since Netgate don't develop FRR (they merely package it), that they are therefor not culpable - I would say this is a logical fallacy. It's like saying the woman who gave Alec Baldwin a loaded gun which fatally shot someone on the set of that Western movie isn't culpable because well - she didn't manufacture the gun and bullets.
- I am not insinuating anything
- Your analogy, honestly, doesn't make any sense at all
- I do agree that care should be taken when upgrading any package I think its probably in everyone's interest that documentation should include a warning that 3rd party packages are not tested fully for compatibility (my words) and could break the security and stability of critical systems. This isn't the first time a 3rd party package has caused production level issues. I would even go as far as not including updated binaries of these packages anytime there is a base software upgrade (23.09 > 24.03).
I dont know if there is an easy solution to this problem tbh.
-
@mAineAc I upgraded my lab 23.09.1 (working fine) to 24.03 and replicated the issue again. I upgraded the two packages as per the above recommendation and I unfortunately still have the issue.
Then I thought about - well what else changed with 24.03? The glaringly obvious answer is the OS itself, moving to "15.0-CURRENT" 14.x was only released late last year https://en.wikipedia.org/wiki/FreeBSD
15.x isn't even "released" yet.Maybe that is OK for apps and features built into the core of the OS. The big issue is when it comes to third-party packages like FRRouting. Why? Because the FRRouting build instructions for FreeBSD 15 simply do not exist:
https://docs.frrouting.org/projects/dev-guide/en/latest/building.htmlMaybe there is more info hiding elsewhere online, but it's conspicuously missing from the page you'd expect to find it on (the building link above). For FreeBSD - it tops out at 14. Did Netgate jump the gun, cross their fingers and hope that all the third-party packages would not break? It certainly seems that way. I don't think there should be any finger pointing at FRRouting if they haven't even certified it to run on FreeBSD 15. Crazy times.
The term "kernel" in "kernel routes" is probably a dead giveaway that the issue stems from the OS update to 15-CURRENT. I'm not saying it is but.. where there is smoke...
So 9.1.1 unfortunately hasn't fixed the issue, and I suspect no version of FRR released to date will fix the issue, when paired with FreeBSD 15-CURRENT. This major OS update has changed something fundamentally in the kernel, and until FRR software gains official build support for FreeBSD 15, we are all probably wasting our time with this.
-
@Gcon What really bothers me about this is in the OSPF settings themselves. If you select to redistribute the default route always the kernel routing table should not make a difference. It should be redistributing the default route whether there is a default route or not.
Edit: I was just looking at the /var/etc/frr/frr.conf file and the ospf config does not even have the redistribute default command in it at all. with or without always.
Edit 2: Actually the originate default is in there. If that option is selected, but originate default always does not appear. If you don't have the redistribute default option selected and just the redistribute default always it will not appear in the config at all. Not sure if both should be selected, but the originate default always never appears in the config.
-
@marcosm
Marcos l, I can’t find anything upstream that indicates this is an issue (maybe I missed it). Is this an issue acknowledged by any maintainers? FRR is just in an unusable state -
Let's try to keep the discussion focused on resolving the issue. To recap a bit...
Reports indicate that:
- The issue does not happen on 23.09.1 with FRR 9.0.2.
- The issue is still present on 24.03 with the latest FRR version; ref 1180159.
- OSPF-learned default routes do not get redistributed; ref 189051.
- These default routes may not get added to the kernel's routing table, though sometimes they do get added; ref 1172657.
Some things to try next in no particular order:
- Test with the latest FRR version on 23.09.1 (which is on FreeBSD 14).
- Diff the FRR and pfSense config between 23.09.1 and 24.03. The package GUI hasn't had any changes that would affect this, but changes in pfSense itself could.
- Check the routing, system, and gateway logs when the issue happens (see "Syslog Logging" option in the package's Global Settings).
- Check if the issue is reproducible with a floating state policy (see "Firewall State Policy" in System > Advanced > Firewall & NAT). It's unlikely this would affect the issue, but it's easy an easy test.
-
@marcosm agreed.
I can spin up a vm in the next few hours but I think the folks here got a working lab now. I’ll report back with the findings -
@marcosm I have done tests. I would have done them sooner but I'm in a different time zone.
In my GNS3 lab I booted up my simulated ISP router then booted just the pfsense+ firewall, but no other ospf peers behind the firewall, which cuts down the amount of routes in the routing table which aren't relevant.
Results:
No issue on 23.09.01 with FRR 9.0.2.
The issue did occur running 23.09.01 with FRR 9.1.1Note the versions and kernel routes missing on the right (in pink).
I can boot up the pfsense+ firewall multiple times, and each time the routes are consistent as shown above. 9.0.2 always has the K routes. 9.1.1 never does - at least on boot up and with no other instability or config changes.
I diffed the /var/etc/frr/frr.conf from 9.0.2 and 9.1.1 and for me - exactly the same. I haven't checked config against 24.3 but might not need to, now that I see that even 23.09.1 is affected with the package update.
When I reboot the virtualized Cisco 7200 I'm using to represent my ISP in my GNS3 simulation lab, the K route shows up:
You can see from the timestamps and see I had the lab up for 20 mins before I rebooted the virtual ISP and then you can see the default route K route show up after that (in green).
It looks like some race condition on boot up, because if you shake things up post-boot, you can coax some K routes out of it as I have shown. Another example. Under System > General Setup, if I change two of the DNS servers to point to "none" and save, then change it back to what I had it pointing down WAN1, I can then get two more K routes to show up. And I could probably toggle something else to get the forth and final one to show up.
Obviously this is not a workable solution. As 24.3 has FRR 9.1, it does seem to me like a race condition was introduced into FRR (at least on FreeBSD14 / FreeBSD15) going from 9.0.2 to 9.1
As for system logs and policy state impact - I will have a look at that next.
@mAineAc I am definitely interested in analysing default-route origination and/or redistribution at some point (as that affects me too) but maybe that's a different issue not related to this apparent race condition? Not sure. I just want my K routes back like they were in 9.0.2, and when those are again reliable - I can look at the other things.
EDIT: I changed "Firewall State Policy" in "System / Advanced / Firewall & NAT" from Floating States to Interface Bound States and it made no difference, so changed it back.
Also it doesn't seem to be a race condition on boot up, because post-boot, if I restart FRR by either unticking then reticking it in "Services / FRR / Global Settings" or by clicking "Force Service Restart" at the bottom of that page - the issues are still there.
If I tickle the things that make the K routes show up, and then restart the process, the K routes disappear again.
With the fault condition in place, and the firewall having two WAN links (one to WAN1 and one to WAN2). If I log into the ISP1 router and I shut the link to the firewall WAN1, after about 20 seconds, a K route (default-route) will show up on the firewall pointing to WAN2. When I unshut the ISP interface on the ISP end, after 60 seconds the K route default-route swapped over to WAN1
I have the two WANs in a Gateway Group with WAN1 as Tier1, and WAN2 as Tier2.
I enabled syslog logging for FRR, with package logging level "extended". Nothing stood out to me in the logs. Tailing the logs when I reload FRR just gives me this:
Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: /pkg_edit.php: Configuration Change: Aug 13 02:05:18 GTpfsense01 check_reload_status[457]: Syncing firewall Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR BGPd: No config data found. Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR OSPF6d: No config data found. Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR RIPd: No config data found. Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR BFDd: No config data found. Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR: Rebuild configuration. Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR: Daemon state: zebra: running | mgmtd: running | staticd: running | ospfd: running Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR: Service restart forced. Aug 13 02:05:18 GTpfsense01 php-fpm[16405]: FRR Package: FRR: Restarting services. Aug 13 02:05:19 GTpfsense01 staticd[16920]: [MRN6F-AYZC4] Terminating on signal Aug 13 02:05:19 GTpfsense01 mgmtd[16272]: [X3G8F-PM93W] BE-adapter: mgmt_msg_read: got EOF/disconnect Aug 13 02:05:19 GTpfsense01 mgmtd[16272]: [J2RAS-MZ95C] Terminating on signal Aug 13 02:05:20 GTpfsense01 mgmtd[87833]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00 Aug 13 02:05:20 GTpfsense01 staticd[88655]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00 -
Ruling out changes between pfSense versions is very helpful. Before we can say it's an issue in FRR or even a bug at all, let's try to understand what exactly is happening. I'm hoping the logs will have a clue.
-
@marcosm I just edited my note with some extra info. Is it possible to get packages built for 10.1 and I can test those? https://www.frrouting.org/release/. Or even 9.0.3, just to confirm even more that the issue stems from the 9.0.x jump to 9.1.x.