After upgrade from 2.4 to 2.5.1 no Web GUI available

digidax

In /var/log/nginx/error.log there is from today only this: (time stamp is during the update)

2021/06/14 09:30:53 [error] 71232#100869: send() failed (54: Connection reset by peer)
2021/06/14 09:42:02 [alert] 71232#100869: close() socket failed (9: Bad file descriptor)
2021/06/14 09:45:01 [error] 73452#100895: send() failed (54: Connection reset by peer)

Additional now, DNS forwarding hast stoped working for the clients of the lan. Using now public DNS server as work around.

mer

@digidax Hmm. I'd be inclined to reset to factory defaults, reboot, then either try to apply the backup or to redo things one at a time by hand.

digidax

I will install 2.4.5-p1 and the config backup from this version to get back a stable system.

jimp

The output in some of your commands is odd. If I had to guess, I'd say your system actually failed to upgrade and is (was?) not running a matching kernel and base OS.

Your safest path forward is to wipe, reload, and restore a backup.

digidax

Okay, before doing this I found the problem, have 100% used capacity:

[2.5.1-RELEASE][root@edge.localdomain]/root: df -hi
Filesystem            Size    Used   Avail Capacity iused ifree %iused  Mounted on
zroot/ROOT/default    438G    666M    437G     0%     24k  917M    0%   /
devfs                 1.0K    1.0K      0B   100%       0     0  100%   /dev
zroot/tmp             437G    180K    437G     0%      31  917M    0%   /tmp
zroot                 437G     88K    437G     0%       7  917M    0%   /zroot
zroot/var             437G     16M    437G     0%     203  917M    0%   /var
bootpool              1.9G    176M    1.7G     9%     252  3.5M    0%   /bootpool
/dev/md0              3.4M    3.4M   -260K   108%      37   985    4%   /var/run
devfs                 1.0K    1.0K      0B   100%       0     0  100%   /var/dhcpd/dev

why have /dev/md0 used 108%:

[2.5.1-RELEASE][root@edge.localdomain]/root: du -sch /var/run/*
  0B    /var/run/check_reload_status
4.0K    /var/run/cron.pid
4.0K    /var/run/devd.pid
  0B    /var/run/devd.pipe
  0B    /var/run/devd.seqpacket.pipe
  0B    /var/run/dmesg.boot
4.0K    /var/run/dpinger_WAN_PPPOE~83.221.235.65~193.98.112.4.pid
  0B    /var/run/dpinger_WAN_PPPOE~83.221.235.65~193.98.112.4.sock
4.0K    /var/run/expire_accounts.pid
 12K    /var/run/filter_reload_status
4.0K    /var/run/filterlog.pid
4.0K    /var/run/ld-elf.so.hints
4.0K    /var/run/ld-elf32.so.hints
  0B    /var/run/log
  0B    /var/run/logpriv
4.0K    /var/run/nginx-webConfigurator.pid
4.0K    /var/run/ntpd.pid
  0B    /var/run/pcscd.comm
4.0K    /var/run/pcscd.pid
4.0K    /var/run/pfSense_version
4.0K    /var/run/pfSense_version.rc
3.3M    /var/run/php-fpm.core
4.0K    /var/run/php-fpm.pid
  0B    /var/run/php-fpm.socket
4.0K    /var/run/ping_hosts.pid
4.0K    /var/run/pppoe_wan.pid
4.0K    /var/run/snmpd.pid
  0B    /var/run/snmpd.sock
4.0K    /var/run/sshd.pid
4.0K    /var/run/syslog.pid
4.0K    /var/run/unbound.pid
4.0K    /var/run/update_alias_url_data.pid
4.0K    /var/run/updaterrd.sh.pid
  0B    /var/run/utmp
4.0K    /var/run/utx.active
3.4M    total

I have on "/" 437 GB free with 917M inodes.

[2.5.1-RELEASE][root@edge.localdomain]/root: zpool list zroot
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot   452G   690M   451G        -         -     0%     0%  1.00x  ONLINE  -

[2.5.1-RELEASE][root@edge.localdomain]/root: smartctl -a /dev/ada0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 11.3-STABLE amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Black Mobile
Device Model:     WDC WD5000LPLX-21ZNTT0
Serial Number:    WD-WX31A873XYT6
LU WWN Device Id: 5 0014ee 65d37823f
Firmware Version: 01.01A01
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 21 09:55:00 2021 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 6900) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  81) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       5
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       1
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       993
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       5
192 Power-Off_Retract_Count 0x0032   114   114   000    Old_age   Always       -       64761
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       1860326
194 Temperature_Celsius     0x0022   106   097   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

What can I do please?

mer

/dev/md0 is a memory device, not a real file system, so it's not peristent. If you look at your second, there is a corefile from php-fpm; that is why it's full.

The question is why is that crashing? There should be a clue in a log file somewhere.
But you could ssh in, remove the corefile and see if you can manually restart the web interface.

digidax

@mer
removing the core file and rebooting doesn't solve the problem. But I think, the upgrade process was not done for the kernel packages:

[2.5.1-RELEASE][root@edge.localdomain]/root: uname -a
FreeBSD edge.localdomain 11.3-STABLE FreeBSD 11.3-STABLE #243 abf8cba50ce(RELENG_2_4_5): Tue Jun  2 17:53:37 EDT 2020     root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-245/obj/amd64/YNx4Qq3j/build/ce-crossbuild-245/sources/FreeBSD-src/sys/pfSense  amd64
[2.5.1-RELEASE][root@edge.localdomain]/root: pkg info | grep -i kernel
bwi-firmware-kmod-3.130.20     Broadcom AirForce IEEE 802.11 Firmware Kernel Module
pfSense-kernel-pfSense-2.5.1   pfSense kernel (pfSense)

RELENG_2_4_5 != pfSense-kernel-pfSense-2.5.1

Am I right? But:

[2.5.1-RELEASE][root@edge.localdomain]/root: pkg-static version -vL=
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.

How can I force updateing the kernel?

mer

It sounds like the upgrade did not complete.
Forcing it? Not sure.
If this is a Netgate device, the answer may be to open a ticket to get the correct image and reinstall.
If it's not a Netgate device, the answer may be to manually download the 2.5.1 image and reinstall.

Others may have a better answer.

digidax

@mer It's a Intel ATOM Board with a SATA drive.

mer

@digidax Ok, that rules out an offical image from Netgate support, so I agree with @jimp above, I think you're at the point of wipe, reinstall, rebuild with the latest 2.5.1 image.

digidax

@mer
Okay, will wait some days and then do so. Thanks a lot.