Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7
-
Greetings all. Very new to PFSense and trying to migrate to it from a Cisco FTD device. This is a new HA pair CE install on 2.7 which I upgraded to 2.8 when it was released. Running on two SuperMicro SYS-5019D-4C-FN8TP short depth servers. I have been busy learning how PFSense works and migrating by hand my configuration from my FTD box. Everything was going great until I upgraded to 2.8.0 when it was released. Since that time I am getting random crashes on both boxes. I checked the memory using memtest86 and it ran clean twice so I don't think this is a memory issue. It started directly after the upgrade.
Attached is the dump file the last crash reated on the backup firewall. I only have a handful of packages which include FRR, Freeradius, OpenVPN Config Exporter to name a few. Not sure how to go about figuring what is wrong with these but I am stuck from deploying these in my environment until I can figure this out. Thanks in advance for any help here.
textdump.tar -
So its clear, I only did the upgrade to 2.8.0. I did not re-install any packages. IS that a required for PFSense upgrades?
-
@rfranzke The upgrade guide suggests manually uninstalling packages (and reinstalling after) but the upgrade will reinstall them as part of the upgrade.
I don’t have insight on your crashes sorry.
-
@rfranzke Looks like a system crash after booting up
Hardly can be a package upgrade issue causing this.
Also getting random crashes on two different machines can't be easily blamed on hardware too.Are you on a uefi or a bios boot env?
I would suggest downgrading to 2.7 and verify that no issues exist there. Opt for uefi too.
I see lots of options, frr, freeradius, openvpn, all these do take some time to grasp and it can be overwhelming while learning pfsense too.I suggest after downgrading to 2.7, make HA work, don't add any other options if possible
and migrate to 2.8.1 (its beta but has been quite stable)
Establish stability and then proceed with options, one step at a time. -
@netblues Thanks for the reply here. Its taken a long time for me to migrate this but I ran this for quite some time on 2.7 and never experienced a crash of any sort while on that version. These crashes literally started right after the 2.8 upgrade. I will say that one thing I changed around the same time is the CARP setup. Originally I configured all my NAT VIPs to by CARP VIPs tied to the WAN interface, but changed them to be IP ALIAS type VIPs tied to the WAN CARP interface based on guidance from this forum. These alias VIPs failover together now when the HA WAN CARP address fails to the backup firewall. See below:
The BIOS info is below:
Vendor: American Megatrends Inc.
Version: 2.2
Release Date: Tue Sep 3 2024
Boot Method: BIOSI'm using some WD Blue NvME drives to run this. Again the only time I've had an issue with this is when I did the 2.8 upgrade. It was immediately after the upgrade this started happening. I don't remember the timing but it seems I made the VIP change just before the upgrade. I literally got everything working and was ready to deploy this and the 2.8 upgrade came out. I thought it would be nice to upgrade to the latest available version since this was a brand new deployment, but its done nothing but randomply crash since the upgrade. Does always seem to be shortly after boot.
I'm not sure the process to downgrade to 2.7. I'll poke around for a procedure but if anyone knows the process I would appreciate the guidance. Thanks again for the help.
-
@rfranzke There’s not a direct downgrade; one must reinstall.
-
Also here, it seems that reinstalling packages is a good practice when doing an upgrade. I again did not do this during this upgrade. Any reason to think reinstalling them is a good idea to help try and stabilize this? Do the settings for the various packages stay when you remove and reinstall them? I originally tried PFSense on an old test server I had lying around before purchasing this new HA pair hardware to run it on just to get a base config for PFSense and to try and lern it some with the idea I would backup/restore to the new hardware once I got it. It went horribly and I ended up starting again from scratch on the new hardware. I would like to avoid doing that again if possible for obvious reasons. This was a 2.7.2 config to a 2.7.2 config before the 2.8 upgrade. None of it worked after the config restore. A real mess I'm sorry to say. I had to start completely over after that and months of work down the drain. I am reading that a 2.7.2 downgrade is not officially supported so not sure if I am looking at another start over. Anyay thanks for the help all.
-
@rfranzke The upgrade reinstalls packages because they often have updates for the new OS/libraries. So I expect that won’t help.
Typically a restore to other hardware is fine; one just assigns interfaces.
You typically cannot restore a newer config backwards on an older version. Only forwards.
-
@SteveITS So what I am hearing is I am gonna have to completely start over with this.....again. My experience with the backup/restore process is that it doesn't work that well. I thought PFSense was enterprise ready? This will be my third time building this config from scratch. Not sure I can afford the downtime to mess with this over and over like this when I put it in production. Not a good start.
If anyone has insight as to what this page fault is, and how to stop it without starting over with this I would appreciate it.
-
@rfranzke Usually restores are fine so it's hard to guess what (apparently) went wrong from here.
@stephenw10 may be of assistance with the crashes. Or you could maybe check a FreeBSD hardware compatibility list.
There are a handful of panic fixes in 2.8.1.
-
@SteveITS Thanks for the reply. Being as new as I am with this, I'm not sure how I would get the 2.8.1 version on here. Assuming this is current RC status? You have to be part of the beta team to get access to this or is there a link readily available somewhere? Assuming I have to build a new installer USB stick to get this on there (my UI does not list this RC as available).
I can appreciate trying to figure this out via forum posts but am not beyond spending the coin for a support sub, which I believe requires plus to utilize. As I am still 'trialing' this move to PFSense, I was hoping to get past this issue before fully getting into bed with NetGate for something like that but could be convinced if they could figure this out. We need a new firewall for our org, and I was hoping PFSense was the answer. Not to rant but at this point I would have likely been fired if I had these actually in production with as many times as I have had to 'start over' now. I just cannot believe the answer here is re-install. Makes me feel like this is still just a hobbyists tool. Really want to like this but it's been a fight so far.
Thanks for all replies.
-
@rfranzke On page System > Update there's a choice of branches and you should be able to see the beta. I don't have a 2.x install to check but I've seen others post about it including Netgate so I'd think it should be there? It's a beta not RC.
https://www.reddit.com/r/PFSENSE/comments/1m2g8k2/pfsense_ce_281_beta_now_available/
Just be aware that packages also use that setting so if you change it and don't upgrade, change it back before updating or installing any packages. (see my sig)
Technically I think (assume) you can get the beta from the Netgate Installer but 1) you can just upgrade, and 2) you probably don't need a new Installer as I doubt that would be required...the Installer is separate from the product versions it installs.
-
@SteveITS Yes thanks here I see it now in the UI. I'll give this a try. Thanks for setting me straight.
Can you downgrade from 2.8.1 once you upgrade or are you stuck there running a beta version? What happens when this comes out of beta? Will it just show I'm in the stable branch when it does?
-
So did the 2.8.1 upgrade and let the boxes do what they do. After everything was stable, I shut them both down and brought them up again. Within 5 minutes the backup FW crashed. So unfortunately, the beta version did not fix my issue it seems. I no longer have any option to go back to 2.8.0 so seems I'm stuck with a beta that doesn't fix anything for me. Worth a try I guess but I guess I am relegated to starting over unless someone can sort out what the posted crash reports say. I can't make heads or tails of it.
-
@rfranzke As it is already stated , you can't go back while in beta or rc
When the release comes out you will be able to upgrade to final.It;s sad you have to dig around hardware issues, however pf sense is a complete hardware solution
If you opted for such a solution you would have the complete enterprise ready platform.
Running it on any other platform has some risks.
However there are options.a. Virtualize the whole thing under any hypervisor. Lots of options here.
Then everything is a file and you also get snapshots.
The overhead is negligible, and many have been doing this for years in HA setups.
I'm one of them.
On top of that, pfsense being available on aws and azure also means it runs well under a hypervisor and is more or less supported.b. Spend 120$ for each instance and upgrade to plus.
You will get tac support (low on calories too) and most of all , boot environments
which allows you to go back.Imho virtualization gives you better control than boot environments by design, however knowing your way around running virtualized pf in a ha production env, does require some knowledge, especially in a crisis.
Don't worry too much about going back to 2.8, 2.8,1 beta is quite stable (under kvm) too.
-
I keep forgetting but I believe the
bectlcommand works on CE, it's just that BEs are in the web GUI in Plus. Just gotta make the BE first (sorry). -
@netblues Yes fair point on the hardware bit. Buying the NetGate hardware would likely solve my issues here. Trying to get all this stuff to work on commodity hardware is a challenge, likely more to do with the BSD foundation it sits on than PFSense itself. It's the leg up the Palo Altos and Cisco's of the world have in the enterprise space. I am grateful there is a CE option at all here. The real issue is that I am not sure what to do with this crash report.
Would it be fair to say the support contract would likely get me answers on the crash report or will this boil down to 'start over' while paying for the privilege?
Honestly my real disappointment is not being able to have this thing deployed. Looks like it would work great, but I cannot even get it out of the stable.
Thanks for all the replies.
-
@SteveITS Well good to know for future. Would help with worries about doing upgrades when this finally goes to production. I'll keep it in mind for next upgrade process. Thanks for that tip.
Is there any chance my config for 2.8 would work if I reinstalled to 2.7 and restored it? I guess thinking about it now my only real issue I seemed to have with moving to new hardware was to do with the change in NIC hardware. Could be wrong. I know you said generally newer configs to older versions is a no go but.......
-
@rfranzke Normally no, sorry. They have a table linked on page https://docs.netgate.com/pfsense/en/latest/backup/restore-different-version.html and 2.8.x is config file v24.0.
The files are XML so you could compare them and edit manually. Or make recent changes again.
The time I've run into trouble restoring to different hardware, as I recall now, is by clicking Apply before clicking Save, on the page where you assign interfaces. I don't know if they fixed that. But then in that state pfSense will stop during boot to ask you to reassign interfaces via the console.
-
I prolly have 2.7 backups somewhere that I took before the upgrade that will get me most of the way there. I forget now the status on the various installed packages. Seems like there was a way to have packages installed that are needed as part of the restore but cannot remember if there was some backup tick you had to check when doing the backup to support that. Maybe I am dreaming on that.
So, should I get the base PFSense installed, then packages installed, then restore the config for the correct version? Or should I get base installed, install packages, get HA/sync working, and then restore. Or maybe I can simply install and restore the backups I have to make this work. The backups I have is for everything.
Thanks all for the help. I'm committed to getting this going.