Warning! Do not update with todays 1/10 snapshot!

louis2

A few remarks:

Be aware I do not now what is causing the problems I have.

Given my findings reported a few hours ago, I did decide to have a look what is the next action after Configurating LAGGS
So I did boot and noticed that is Configurating VLANs

Very recent I noticed I had a crashdump when running the 27/9 version, which might be related to laggs as well.
<118>Configuring VLAN interfaces...
<6>vlan0: changing name to 'lagg0.10'
panic: sleeping in an epoch section
cpuid = 2
time = 1601543446
KDB: enter: panic

However, despite that dump the 27/9 version seems to work OK

After the config changes I did do earlier today, related to the xml-parser issues, I did decide to test what happens when I made changes to the lagg.

So, I did remove one of the interfaces from the lagg and assign all vlans related to the lagg to that "now free interface".
=> no issues
=> reboot no issues
=> upgrade to todays snapshot => no issues

Than I did move one of the vlan's back to the lagg
=> crash
=> did recover with the vlan assigned to its previous IO

So may be .... :

there was a boot issue
and a config and/or parser issue
and a lagg issue

So far, my actual findings, not 100% clear yet!

Louis

vesalius

Starting to narrow things down Louis2. Forgive me if I missed this or I am just confounding responses, but this only seems to bite on your setup if you have lagg, but does having a lagg cause the issue on bare-metal and in a VM or just in a VM?

louis2

@vesalius I do no have VM's here so all my findings relate to real hardware.

For info

the LAGG is normally IGB0 plus IGB1
and part of my vlans are assigned to that LAGG
since the LAGG causes trouble I removed IGB0 from the LAGG and moved those VLAN's to IGB0
then I removed the LAGG completely

That is the test config I am using now. The FW seems to work that way

My network is not, since the attached switch does expect a LAGG

So I have to return to a working config tomorrow (the older snapshot and the older config probably)

My main computer is attached to another switch, thats why I can write this reply :)

Louis

Quarkz

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240609

Wonder if it is this bug?

louis2

I would not be surprised
It is OS-related and severe that is for sure.

Netgate should have a look, so I uploaded one of my crash dumps for analyses.

Louis

_rp

I've also had issues with upgrading to the latest snapshots from a previous snapshot, I've had to wipe my device and restore config.

It seems to be fine when upgrading from 2.4.5-p1 to the latest snapshot but I bet it'll break again if I then upgrade to another snapshot.

Sorry I don't have logged what exactly the error was but my google search history points to

"Cannot open /boot/lua/cli
Lua Error loading module, file not found"

If it happens again I'll take a screenshot.

AKEGEC

Guys, it's a bug. I tried to update my bios and use basic settings (disable all turbo boost/overclock). It's still crached.
I think someone added "(" somewhere by accident. ;)

aivxtla

I have pfSense installed bare metal using EFI/GPT/ZFS (no mirroring) currently on an AMD EPYC 3251 SuperMicro build and previously (a few days ago) on an Intel D-1541 based firewall, I have not had issues rolling from one build to another daily since August, I tested extra reboots after upgrades recently after seeing this thread and no issues were seen. I finally did a clean install of the Oct 3 morning snapshot for another issue and still have not run into this crash problem having done upgrades to the latest builds each day since. My network adapter is an Intel X710-T2L and I have no VLANS or LAGG running if that helps. Maybe something to do with specific configurations or hardware?

vesalius

Per Jim Pingle on Redmine seems like netgate knows about the LAGG issue hitting the OP @louis2 and may have a fix for it.
https://redmine.pfsense.org/issues/10956

There is also another EFI bug that needs to be tracked down.
https://redmine.pfsense.org/issues/10943

louis2

Good news!

Yesterday afternoon "jim/redmine" asked me to test/confirm a fix for the lagg problem.

Triggered by that signal I did upgrade to yesterday evening snapshot. That worked!
This morning I checked my logging's and I did not notice disturbing things.

So

boot issue seems to be solved
lagg issue seems to be solved (I do not know if that is by a temporarily patch or with the final solution)

About my third problem "xml-parser and/or config issues", I did not noticed that problem any more, after I "fixed" my config. I do now what caused those issues and if things have been fixed. I can just advice to monitor the boot/startup messages on the console, in case you have boot/startup problems.

Louis

_rp

OK, I'll give it a test later tonight when I get home and I'll report back.

msm

Boot, lagg and e1000 issues all seem to be resolved with latest build.

I did notice that dhcpv6 client stopped working for me after 10/1 updates, but will look at that separately.

chrcoluk

@Quarkz I doubt pfSense activates invariants.

vesalius

@chrcoluk read the Bug #10956 from links below in redmine. Invarients are mentioned as the cause for that LAGG issue. Not sure on the EFI issue.

maverick_slo

@msm said in Warning! Do not update with todays 1/10 snapshot!:

Boot, lagg and e1000 issues all seem to be resolved with latest build.

I did notice that dhcpv6 client stopped working for me after 10/1 updates, but will look at that separately.

DHCPv6c stopped for me too

_rp

It looks to have resolved the boot issue for me. I'll continue to test and report back if it does fail but I think the boot issue has been resolved.

kiokoman

about the efi part, i found out that using sata instead of scsi controller under esxi solved the boot problem for me
scsi controller is the default for esxi
@w0w posted a screenshot of Virtualbox where Controller is IDE

but there are other thread with similar problem, in any case i suspect something is wrong with freebsd + efi

@ramup https://forum.netgate.com/topic/154769/pfsense-does-not-boot-after-successful-installation/
@insept https://forum.netgate.com/topic/156245/system-freeze-at-usb-boot-secure-boot-off-uefi-mode-only

read https://redmine.pfsense.org/issues/10943

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209821

_rp

@kiokoman Your first link describes a problem with 2.4.5-p1 which the problem in this topic is using a 2.5 snapshot so not the same problem.

My boot issues were also experienced using physical hardware, BIOS settings hadn't changed and the issue only appeared when using recent 2.5 snapshots, version 2.4.5-p1 was fine.

chrcoluk

@vesalius said in Warning! Do not update with todays 1/10 snapshot!:

@chrcoluk read the Bug #10956 from links below in redmine. Invarients are mentioned as the cause for that LAGG issue. Not sure on the EFI issue.

Right ok, surprised that pfSense left that on.

Bob.Dig

@kiokoman said in Warning! Do not update with todays 1/10 snapshot!:

scsi controller is the default for esxi

On every version?