Bricked after Update 2.4.5-p1 to 2.5.2-RELEASE

AWeidner

System:
Atom D525, 2GB RAM
WAN (re0): RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet
LAN (em0): Intel(R) PRO/1000 Legacy Network Connection 1.1.0

I just updated our WAN-Router from 2.4.5-p1 to 2.5.2-RELEASE. The upgrade went through without error messages.

Problems:

Cannot access GUI from LAN
SSH is active on WAN, though i did not activate it (100% sure about that)
Does not route anything
/var/run/php-fpm.core constantly takes up all space on /var (which is only 57MB in size?)
Trying to start 'top' from commandline exits with 'top: sysctl(vm.stats.vm.v_swappgsin...) expected 8, got 4'
the error message 'KLD dummynet.ko: depends on kernel - not available or version mismatch' appears multiple times on the console

What works:

Console menu access from LAN via SSH and on the console itself

I am stumped. I had another pfSense with the same version updated just minnutes before that and it works just fine.

What i tried unsuccessfully:

rerun the update from commandline option 13 - everything is up to date
multiple reboots, including reboot with file system check
commandline options 12 and 16 (16 says it cannot kill a process that is not there)
Added hw.pci.enable_msix="0" to /boot/loader.conf

Any suggestions?

Cool_Corona

@aweidner reimage?

stephenw10

It looks like php is crashing out for some reason there. It's core dumping to /var.
Are you running RAM disks if that's only ~60MB?

If PHP crashes during upgrade then the system state is unknown. It wouldn't have enabled SSH though. Are you sure you didn't have it enabled but blocked on WAN?

I would re-install clean to be sure from there if you can.

Steve

AWeidner

@stephenw10 said in Bricked after Update 2.4.5-p1 to 2.5.2-RELEASE:

It looks like php is crashing out for some reason there. It's core dumping to /var.
Are you running RAM disks if that's only ~60MB?

If PHP crashes during upgrade then the system state is unknown. It wouldn't have enabled SSH though. Are you sure you didn't have it enabled but blocked on WAN?

I would re-install clean to be sure from there if you can.

Steve

Thank you for your reply.
Yes, SSH was enabled before but not allowed on the WAN interface. I also put the web gui on port 8443, which is now also open on the WAN side, though the gui does not open. The ports that should be available to the LAN are now also open on the WAN, for whatever reason.

If i reinstall and restore my config.xml-backup, wouldn't that reintroduce the same errors, like the too small sized ramdisk?

stephenw10

Not necessarily. It might have failed during the upgrade for some entirely different reason. You are just seeing /var exhausted because php is core dumping there.
But I would recommend at least double the default size for RAM disks on x86 anyway. So 80 and 120MB for /tmp and /var.

Steve

AWeidner

@stephenw10 said in Bricked after Update 2.4.5-p1 to 2.5.2-RELEASE:

Not necessarily. It might have failed during the upgrade for some entirely different reason. You are just seeing /var exhausted because php is core dumping there.
But I would recommend at least double the default size for RAM disks on x86 anyway. So 80 and 120MB for /tmp and /var.

Steve

Will try that on monday. I removed the PHP dump file and had disk space available, but the networking side still did not work.

I could live with the gui not working for now. But that it does not do any routing and seemingly ignores the firewall settings bothers me much.

AWeidner

Solution: swapped the device with an APU 2C4 that was still in storage.

Something is amiss with version 2.5.2-RELEASE, the em driver and the Intel Pro1000XT card that is in the device. I made a bare metal install and restarted. The device works fine for a few seconds, maybe up to a minute. If you ping something from the LAN interface (em0), it stops after about a minute and it says "ping: sendto: No buffer space available". If you "ifconfig down" and "ifconfig up" the device, it works again for short period of time.
I suspect something has changed from the em driver used in version 2.4.5 vs the one in 2.5.2 that causes issues with the card used in my device.

stephenw10

Hmm, I've seen one other user reporting issues with em. What's the exact PCI IDs from that? You may have the same card.
There were significant changes to many NIC drivers between 2.4.5 and 2.5.X as they were converted to iflib in FreeBSD 12. em is very widely used though, I have several devices using them here.

Steve

AWeidner

@stephenw10 said in Bricked after Update 2.4.5-p1 to 2.5.2-RELEASE:

Hmm, I've seen one other user reporting issues with em. What's the exact PCI IDs from that? You may have the same card.
There were significant changes to many NIC drivers between 2.4.5 and 2.5.X as they were converted to iflib in FreeBSD 12. em is very widely used though, I have several devices using them here.

Steve

em0@pci0:5:0:0: class=0x020000 card=0x11078086 chip=0x10088086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82544EI Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet

Intel PRO1000XT ethernet adapter.

Additional Testing:
I disabled Hardware Checksum/Hardware TCP Segmentation/Hardware Large Receive Offloading. Now it is working without hickups so far. I reenabled those options and it started to act up again. It appears that one of these offloading mechanisms is the culprit. The man page for the em driver states, that TSO is not available on the chipset this card is based on (82544). I suppose this leaves "checksum offloading" or "hardware large receive offloading" as the source for the problem.

stephenw10

Everything except checksum off loading should be disabled by default so I would look at LRO if you changed that.

Steve

AWeidner

@stephenw10 said in Bricked after Update 2.4.5-p1 to 2.5.2-RELEASE:

Everything except checksum off loading should be disabled by default so I would look at LRO if you changed that.

Steve

I will leave the APU in place. The former device was cobbled together from spare parts anyway (but it worked for years...). Thank you for all the input.