Warning! Do not update with todays 1/10 snapshot!

louis2

Given wow his positive message, I did start an update from the gui.

I noticed that the FW did not return / did not become active, so I looked ad my kvm-switch. Trying to boot form network since it could not find any valid local startup partion.

So I did force a restart via "Cntr-Alt-Del" and again noticed no valid boot. Did that another time same result.

So .... not ok yet. Notice that redmine has started an "urgent" issue related to this problem (10943), which is not reported as fixed.

Louis

msm

When I upgraded this morning, system was caught in reboot loop trying to load:

FreeBSD 12.2-STABLE 2c7ab6a3c3f(devel-12) pfSense amd64

When I switched to kernel.old it came up with:

2.5.0-DEVELOPMENT (amd64)
built on Sat Oct 03 06:53:59 EDT 2020
FreeBSD 12.2-PRERELEASE

Crashes are here: https://gist.github.com/emes/cba7dfb53e7127f1b122c6045e341983

AKEGEC

Thanks for the info. I almost upgrade it.

kiokoman

@msm i think you have a different problem for kernel.old
e1000_reset_hw_82571
there is a driver problem with your e1000
but maybe the old kernel is incompatible with other upgrades done by pfsense

msm

Yes, was just filing under warnings for others.

You meant to say "new kernel", no?

kiokoman

with new kernel you have boot loop as we have right?
you said that switching to kernel.old lead to a crash dump

w0w

@kiokoman said in Warning! Do not update with todays 1/10 snapshot!:

@w0w
did you reboot twice?

Three times. If it does matter I use ZFS.
On the real hardware I have mirrored ZFS volume. Sometimes after upgrade it comes with old version back, so I need to update this twice sometimes. On the latest snapshots I did it every time today, but I didn't attach any importance to it. May be it's failed really, but booted because of ZFS, dunno. This does not explain why VM version on ZFS but without mirror works fine.

kiokoman

could be a combination of efi and ufs

w0w

Installed snapshot from Sep 16 19:03 in the Oracle VirtualBox VM using BIOS emulation and selecting AUTO UEFI UFS, after installation completed, I've switched to UEFI and then booted the VM and started upgrade from console, option 13, it loaded latest files from 4 oct and installed them without any error. Rebooted 5 times — no problem, am I lucky one or we just missing something?
Here is the screenshot of my VM settings:

Install process:
https://streamable.com/v81y0j
Boot, upgrade and several reboots:
https://streamable.com/50idjx

louis2

I just did a lot of testing using a today's snapshot

First test was an upgrade from the GUI. Not successful, but it did reboot

Second test was a new install from an usb-stick without any config. That did run, but I could not use it (no-config).

Third test was a new install from usb, with my config on a second usb-stick, and that did not work

And I did notice two other problems:

the config file did have wrong entry's according to the parser. I noticed two "dhcpddata" and "dhcpdv6data" both occurring twice! And I think I agree with the parser and deleted the "redundant (different!) entry's.
then there was another issue related to lagg's. As soon as the startup sequence reaches "Configurating LAGG interfaces " it crashed and started again

So it seems that there is more than one issue

Whatever have I stopped testing for now, being back op the older version.

Louis

w0w

Hmmm... it is possible that you have broken config. Netgate team did a lot of changes in the middle of September, adding new features for XML parser and backup. A lot of changes in the code. I have problems with CARP now and there is still something broken. What version do you have now, 27/9?

louis2

A few remarks:

Be aware I do not now what is causing the problems I have.

Given my findings reported a few hours ago, I did decide to have a look what is the next action after Configurating LAGGS
So I did boot and noticed that is Configurating VLANs

Very recent I noticed I had a crashdump when running the 27/9 version, which might be related to laggs as well.
<118>Configuring VLAN interfaces...
<6>vlan0: changing name to 'lagg0.10'
panic: sleeping in an epoch section
cpuid = 2
time = 1601543446
KDB: enter: panic

However, despite that dump the 27/9 version seems to work OK

After the config changes I did do earlier today, related to the xml-parser issues, I did decide to test what happens when I made changes to the lagg.

So, I did remove one of the interfaces from the lagg and assign all vlans related to the lagg to that "now free interface".
=> no issues
=> reboot no issues
=> upgrade to todays snapshot => no issues

Than I did move one of the vlan's back to the lagg
=> crash
=> did recover with the vlan assigned to its previous IO

So may be .... :

there was a boot issue
and a config and/or parser issue
and a lagg issue

So far, my actual findings, not 100% clear yet!

Louis

vesalius

Starting to narrow things down Louis2. Forgive me if I missed this or I am just confounding responses, but this only seems to bite on your setup if you have lagg, but does having a lagg cause the issue on bare-metal and in a VM or just in a VM?

louis2

@vesalius I do no have VM's here so all my findings relate to real hardware.

For info

the LAGG is normally IGB0 plus IGB1
and part of my vlans are assigned to that LAGG
since the LAGG causes trouble I removed IGB0 from the LAGG and moved those VLAN's to IGB0
then I removed the LAGG completely

That is the test config I am using now. The FW seems to work that way

My network is not, since the attached switch does expect a LAGG

So I have to return to a working config tomorrow (the older snapshot and the older config probably)

My main computer is attached to another switch, thats why I can write this reply :)

Louis

Quarkz

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240609

Wonder if it is this bug?

louis2

I would not be surprised
It is OS-related and severe that is for sure.

Netgate should have a look, so I uploaded one of my crash dumps for analyses.

Louis

_rp

I've also had issues with upgrading to the latest snapshots from a previous snapshot, I've had to wipe my device and restore config.

It seems to be fine when upgrading from 2.4.5-p1 to the latest snapshot but I bet it'll break again if I then upgrade to another snapshot.

Sorry I don't have logged what exactly the error was but my google search history points to

"Cannot open /boot/lua/cli
Lua Error loading module, file not found"

If it happens again I'll take a screenshot.

AKEGEC

Guys, it's a bug. I tried to update my bios and use basic settings (disable all turbo boost/overclock). It's still crached.
I think someone added "(" somewhere by accident. ;)

aivxtla

I have pfSense installed bare metal using EFI/GPT/ZFS (no mirroring) currently on an AMD EPYC 3251 SuperMicro build and previously (a few days ago) on an Intel D-1541 based firewall, I have not had issues rolling from one build to another daily since August, I tested extra reboots after upgrades recently after seeing this thread and no issues were seen. I finally did a clean install of the Oct 3 morning snapshot for another issue and still have not run into this crash problem having done upgrades to the latest builds each day since. My network adapter is an Intel X710-T2L and I have no VLANS or LAGG running if that helps. Maybe something to do with specific configurations or hardware?

vesalius

Per Jim Pingle on Redmine seems like netgate knows about the LAGG issue hitting the OP @louis2 and may have a fix for it.
https://redmine.pfsense.org/issues/10956

There is also another EFI bug that needs to be tracked down.
https://redmine.pfsense.org/issues/10943