Now Available: pfSense Plus 26.03.1
-
@stephenw10 No custom scripts for the LED. Manually setting to ‘ready’ does indeed work — I get the slowly flashing blue.
A pkg check doesn’t reveal any problems:
[26.03.1-RELEASE][admin@firewall.amarillo.johnkline.com]/root: pkg check -s -a
Checking all packages: 54%
pfSense-kernel-pfSense-26.03.1: checksum mismatch for /boot/kernel/linker.hints
Checking all packages: 58%
pfSense-system-26.03.1: checksum mismatch for /etc/bogons
pfSense-system-26.03.1: checksum mismatch for /etc/bogonsv6
Checking all packages: 100% -
Hmm try enabling verbose RC with:
touch /verbose_rcthen check the bootup at the console.You should see:
Netgate pfSense Plus 26.03.1-RELEASE amd64 20260520-1519 + echo 'Bootup complete' Bootup complete + echo 'NOTICE Bootup complete' + /usr/bin/logger -p user.notice + /usr/local/bin/beep.sh start + errexit_off + zfs_on_root + zfs_root_dataset + set +e + /usr/local/sbin/pfSense-led.sh ready + errexit_on -
@jimp said in Now Available: pfSense Plus 26.03.1:
@JeGr said in Now Available: pfSense Plus 26.03.1:
@jimp said in Now Available: pfSense Plus 26.03.1:
I'll check back but as far as I remember, nothing is selected there yet, so all options still default.
The default of the specific option I mentioned is unchecked (and maybe should be checked by default instead), so it's still possible that's contributing. If it is unchecked, try checking it on both and seeing if the problem still happens.
The only thing checked currently is "do not create rules if gateway is down". All other are still default. As there are minimal rules and no PBR at all, that strikes me as odd, that there should be kills where there are no rules as the option is only for policy created rules. But I don't think that's the problem.
What sticks out to me is the strange kernel CARP messages. I checked with another cluster still on 26.03 and also saw those gateway monitoring messages pop up (perhaps that came in 25.11 or 26.03 but I think it's a newer thing) but there's no sign of anything related to those Kernel CARP messages. Just "got a sync, a second later filter reloaded, then the NOTICEs about the GWs, done. In 3s tops.
Whereas the "buggy" cluster in question takes 30s (31), then reloading filter shows up twice.I don't get it.
-
@JeGr said in Now Available: pfSense Plus 26.03.1:
The only thing checked currently is "do not create rules if gateway is down". All other are still default. As there are minimal rules and no PBR at all, that strikes me as odd, that there should be kills where there are no rules as the option is only for policy created rules. But I don't think that's the problem.
That's fine but do try checking the box I mentioned because when it's unchecked, it could still kill states.
Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!
Need help fast? Netgate Global Support!
Do not Chat/PM for help!
-
@jimp said in Now Available: pfSense Plus 26.03.1:
That's fine but do try checking the box I mentioned because when it's unchecked, it could still kill states.
How so? Didn't you talk about that one (the one on top)? We only did the one below because if we do PBRs we don't want them sent to any other gateway when said GW is down, so better no rule at all. But the one above definetly says only created by firewall rules? And we have none.
I can switch it on but currently re-booting into 26.03 snapshots as the staff on premise has powered on the machines again so I can remote-in and check.
I've another idea though what could be the culprit, that 30/31 seconds timeout is too specific...
-
@JeGr said in Now Available: pfSense Plus 26.03.1:
How so? Didn't you talk about that one (the one on top)? We only did the one below because if we do PBRs we don't want them sent to any other gateway when said GW is down, so better no rule at all. But the one above definetly says only created by firewall rules? And we have none.
I meant the "Don't kill policy routing states for lower-priority gateways" option -- I'm not saying it's the likely culprit but it's a suspect to eliminate. There is a chance it's misbehaving. It may not be doing what it says, or in combination with your configuration it's working in an unexpected way.
30 seconds is a common initial/half-open state timeout, the kind of timeout you'd see with asymmetric routing or states otherwise not being fully established.
The pf bulk sync log messages imply the state table was dumped or it thought it needed a full sync, so something else could be mismatched in the state table, maybe different hardware and interface bound states sort of deal.
Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!
Need help fast? Netgate Global Support!
Do not Chat/PM for help!
-
@stephenw10 Thank you so much for this. I’m traveling and won’t reboot until I return, but what you’ve provided should let me track down the problem. I see these statements in /etc/pfSense-rc. I know I get the Bootup complete, so I’ll be able to see where things go wrong after that.
-
@jimp Damn I think I got the culprit and it's an unexpected one!
That whole
carp demotion made me sus, as normally it's a number != 0 indicating some kind of status. CARP weight of 0 normally is fine and master, 240 is demotion either manually or error or something - you get what I mean.Rebooting into 26.03 showed the same "lag" what was curious as pre-update we didn't have that problems. So rebooting back into the .1 I checked side by side, if the boxes differed somewhere serious. Nope. Then checked the interfaces and found, that one of the 4 WANs we prepped was disabled. To be expected, as it's the only one with a /29 instead of /28 so we can't run both old/new clusters in parallel there, so that interface got disabled but still had the VIP we prepared. Which perhaps shouldn't work? If I disable the interface after there's a VIP on it, there was no error when checking again. So I could create a non-functional CARP VIP on a disabled interface. That seems to lead to the misleading carp demotion errors and the inherent lag of 31s. Removing it brought the sync down to 2-3s again and the carp errors stopped. These 31s also allegedly accounted for all the errors by piling up on updates, "randomly rolling back" changes, filling up PHP/nginx workers, etc.
I think the VIP itself should've gone into an INIT state like when "interface down" is happening or when disabling the interface perhaps also disable the VIP on it? Don't know the best way for that, but as I've just experienced, it can be a hell to debug that stuff when someone disables the interface and unplugs it so you have no visual indicator on screen to "oh the interface is gone/disabled, that has to be the culprit".
So seems very much unrelated to the release itself other then a stupid overlap in time where the staff disabled the interface (correctly) and me not being able to correlate the errors seen to the correct source.
My bad for driving folks crazy
-
Hmm, so to be clear a CARP VIP on a disabled interface was still triggering events? That interface had a gateway?
-
@stephenw10 said in Now Available: pfSense Plus 26.03.1:
Hmm, so to be clear a CARP VIP on a disabled interface was still triggering events? That interface had a gateway?
Indeed. The VIP was defined on the disabled interface. And yes, that interface has a gateway so it's correctly recognized (when enabled) as WAN style interface. That gateway shows up in "Routing" but of course not in Status/GWs as it's correctly disabled as the interface is disabled. Seems the VIP didn't get the message though :)
-
@stephenw10 FreeBSD issue ?? Sorry OpenBSD ??
-
The FreeBSD driver no longer attaches to the PCIe bus device.
-
SG-3100 needed a reboot after the upgrade to bring back the admin UI. Restarting webconfigurator didn't work.
From upgrade log:
Checking integrity... done (0 conflicting) >>> Downloading pkg... No packages are required to be fetched. Integrity check was successful. >>> Downloading if_pppoe-kmod... No packages are required to be fetched. Integrity check was successful. >>> Upgrading pfSense-boot...>>> Unmounting /boot/efi...done. Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Checking integrity... done (0 conflicting) The following 1 package(s) will be affected (of 0 checked): Installed packages to be UPGRADED: pfSense-boot: 26.03 -> 26.03.1 [pfSense-core] Number of packages to be upgraded: 1 [1/1] Upgrading pfSense-boot from 26.03 to 26.03.1... [1/1] Extracting pfSense-boot-26.03.1: .......... done >>> Upgrading pfSense kernel... Checking integrity... done (0 conflicting) The following 1 package(s) will be affected (of 0 checked): Installed packages to be UPGRADED: pfSense-kernel-pfSense-3100: 26.03 -> 26.03.1 [pfSense-core] Number of packages to be upgraded: 1 [1/1] Upgrading pfSense-kernel-pfSense-3100 from 26.03 to 26.03.1... [1/1] Extracting pfSense-kernel-pfSense-3100-26.03.1: .......... done ===> Keeping a copy of current kernel in /boot/kernel.old >>> Removing unnecessary packages...done. >>> Upgrading boot code... Unsupported platform. System is going to be upgraded. Rebooting in 10 seconds. Success -
@lohphat said in Now Available: pfSense Plus 26.03.1:
G-3100 needed a reboot
After a kernel (call that the core OS) update, an reboot is always necessary. The now older kernel needs to be replaced by the newer one.
No "help me" PM's please. Use the forum, the community will thank you.
-
Yes it always has to reboot at firmware upgrade. You mean it didn't reboot by itself?
-
@stephenw10 It did to do the update, but then after it came up on the nerw version the web UI didn't work, everything else was working. Another reboot brought the web UI back.
-
Hmm, odd. Any errors logged at that time?
-
@Gertjan Obviously, but after the reboot, the web UI was unresponsive even after restarting the process from the CLI. Another reboot fixed it.
-
@stephenw10 There's always errors in the logs relating to interface issues on reboot. What should I be looking for?
-
Errors showing the nginx process failing to start for example or php processes crashing out.
What sort of interface errors are you seeing?
