CE 2.9: the first ever release I've had to fall back from
-
You know that saying that your backup strategy is useless if you haven't tested it? Well, today, my Proxmox VE/PBS environment got quite a workout.
For the first time in almost forever, I had to fallback to 2.8.1 from what was otherwise a smooth upgrade to 2.9.
CE 2.9 was just crashing -- hard. Nothing -- and I mean NOTHING in the logs. Just a hard stop and then the boot message after I power off and power on the Proxmox node. I suspect a driver crash that took the entire PVE node with it. I mean a HARD stop -- stuff you shouldn't be seeing in a product this mature.
I'd grown accustomed to rock-solid release upgrades for my detailed but essentially plain-vanilla pfSense configuration. Of course, I'd seen others tortured by upgrades in the past...but no sir, that's not happenin' to me.
Until today.
It's been a long time since pfSense shipped a new feature I've used.
Next time, no bleeding edge for me. Routers gotta route. Firewalls gotta block.
-
@yobyot is it a Celeron by chance?
-
@yobyot said in CE 2.9: the first ever release I've had to fall back from:
It's been a long time since pfSense shipped a new feature I've used.
Next time, no bleeding edge for me. Routers gotta route. Firewalls gotta block.
Firewalls also need to be updated to keep up with security issues.
And since you are in a "less" supported env that offers snapshots, testing and reverting should be super easy, especially when we are talking about a hard stop.
-
Not a Celeron.
It's a Core i7-10810U
-
@netblues Reverting was easy. Proxmox forever.
But something is very much awry with CE 2.9 if, after all these years running reliably and consistently on the same hardware in the same VM, it suddenly takes down the hypervisor to the point that not just the pfSense VM is crashed but all the VMs on that Proxmox node and Proxmox itself are hung, too.
The flaw causes such a hard stop that it requires a power cycle to restore. I want the security fixes, of course. But I was hoping to get them with CE's past consistency -- not a Windows-like experience where security fixes trash the machine.
-
@yobyot said in CE 2.9: the first ever release I've had to fall back from:
But something is very much awry with CE 2.9 if, after all these years running reliably and consistently on the same hardware in the same VM, it suddenly takes down the hypervisor to the point that not just the pfSense VM is crashed but all the VMs on that Proxmox node and Proxmox itself are hung, too.
So, some 'bad program' could download the pfSense Installer, execute it, start pfSense, and takes not only 'that' VM, but the entire layer above it with, this includes all other VM instances with it.
That's pretty huge.I can't claim that "thousands of "2.9.0" are right now running within a proxmox VM", as I have no way of knowing that.
But a VM breaking out of it's cage, is that even a thing ?I can re read your word's and say : "what if the hypervisor crashed ? That would take the VM's with it." Hypervisors can crash ^^
-
But something is very much awry with CE 2.9
This is an opinion stated as fact.
block out log on { ix0 } inet from any to any
block out log on { ix0 } inet6 from any to any -
@yobyot said in CE 2.9: the first ever release I've had to fall back from:
@netblues Reverting was easy. Proxmox forever.
But something is very much awry with CE 2.9 if, after all these years running reliably and consistently on the same hardware in the same VM, it suddenly takes down the hypervisor to the point that not just the pfSense VM is crashed but all the VMs on that Proxmox node and Proxmox itself are hung, too.
The flaw causes such a hard stop that it requires a power cycle to restore. I want the security fixes, of course. But I was hoping to get them with CE's past consistency -- not a Windows-like experience where security fixes trash the machine.
I'd argue the problem is with Proxmox from that description. There is no way a crashing VM should be bringing the whole host down. So I'd wager there is nothing wrong with your CE 2.9, but it now does something new in virtual HW that makes your Proxmox host commit suicide. Start with updating proxmos instead.
-
@tinfoilmatt said in CE 2.9: the first ever release I've had to fall back from:
This is an opinion stated as fact.
In my mind I made a correction, and add the word ' my ' somewhere in that phrase
( I'm not using 2.9.0 but 26.07, and not a VM, so I should probably say nothing ) -
@Gertjan Yup. My reality is fact to me.
-
Well that at least makes you qualified to be the president of the United States.
-
@keyser Good point. But Proxmox was brought to 9.2.10 several weeks before I upgrade to CE 2.9
I get it that this is a pfSense forum -- and I'm a fanboy. The point was that it's my own fault for performing the update early. For something like this, I need to resist being on the bleeding edge. The secondary point was thank goodness for PVE/PBS. I simply restored a known-working backup of 2.8.1 to a new VM, then changed the startup order.
-
@yobyot I have 4 2.9 running on proxmox now since it came out and no crashes or issues.
-
@yobyot said in CE 2.9: the first ever release I've had to fall back from:
But something is very much awry with CE 2.9 if, after all these years running reliably and consistently on the same hardware in the same VM, it suddenly takes down the hypervisor to the point that not just the pfSense VM is crashed but all the VMs on that Proxmox node and Proxmox itself are hung, too.
This should be a top priority for proxmox people to explain. Bringing down a hypervisor isn't common and since
pfsense is not malicious code, this is consider a huge proxmox bug.As for bleeding edge for ce I wouldn't not much blood there either.
Not an rc, not even a beta, and since it is based on 26.07 which is released earlier I wouldn't consider it bleeding at all.On the other hand, try repeating what you did.
Perhaps hypervisor crash was just coincidental and nothing else. -
Mmm, I run in Proxmox and I know many of the devs do too so it seems like it must be some specific combinations of things.
No logs from Proxmox either? I assume it was more that simply consuming all it's assigned CPU cores.
-
@stephenw10 said in CE 2.9: the first ever release I've had to fall back from:
Mmm, I run in Proxmox and I know many of the devs do too so it seems like it must be some specific combinations of things.
No logs from Proxmox either? I assume it was more that simply consuming all it's assigned CPU cores.
Nope...nothing at all. Like I said: whatever is happening it's a real hard stop. The one thing I've failed to mention so far (oversight, not intentional) is that the WAN NIC is dedicated to pfSense via PCI/IOMMU. That's been the case since well before 2.8...dates back several CE releases.
I restored CE 2.8.1 from a PBS backup which runs on the same hardware, same level of Proxmox, same storage, same NIC, same everything -- except the startup/shutdown sequence when Proxmox boots.
That was four days ago and 2.8.1 is running without a hiccup, as it had previously.
I'm mystified.
-
Ah, interesting. What is that NIC? 2.9.0 could be using a newer driver and tickling something. Did it fail immediately on the first boot into 2.9.0?
-
@stephenw10 said in CE 2.9: the first ever release I've had to fall back from:
Ah, interesting. What is that NIC? 2.9.0 could be using a newer driver and tickling something. Did it fail immediately on the first boot into 2.9.0?
No, it didn't fail immediately. First failiure was about 24 hours after upgrade; second about 36.
Output from
lspci -nnon the Proxmox box:01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03) -
Hmm, nothing especially exotic then.
Privacy Policy · Cookie Policy