PFsense hanging since version 2.4.4

[Taz79]

So... I'm with my first Netgate hardware! :D :D :D

[chrismacmahon]

Thank you for your purchase!

It's a nice unit!

[Taz79]

@chrismacmahon

It seems to be a nice product! :)

5 to 6 MB/s in traffic from 4 IP:s and CPU load at 23%.. Looks good so far :) This is a small network with around 20 devices and max speed is 117Mbit/s so i think its a walk in the park for this hardware :)

[kikigak]

Also have this on Feb2019 from artway and maybe we are on the same boat. :D
Only admin password has been changed at install, cmd reboot 3-times after install then crash happen randomly 40mins, 1hr, 2hrs etc.
Already contact with manufacturer but not very responsive.

Bought here:
https://www.lazada.com.ph/products/-i261594927-s361115350.html

Manufacturer link:
http://artwaytech.com/index.php?s=/Android/detail/id/87

Dump files link:
https://drive.google.com/open?id=1Z1_nkFzruCgFYMQxunjQ2PkcrpyfiI3W

[stephenw10]

db:0:kdb.enter.default>  show pcpu
cpuid        = 3
dynamic pcpu = 0xfffffe026599f380
curthread    = 0xfffff8000c007000: pid 14696 "filterlog"
curpcb       = 0xfffffe022f325cc0
fpcurthread  = 0xfffff8000c007000: pid 14696 "filterlog"
idlethread   = 0xfffff80005217620: tid 100006 "idle: cpu3"
curpmap      = 0xfffff80005f92138
tssp         = 0xffffffff82bb6948
commontssp   = 0xffffffff82bb6948
rsp0         = 0xfffffe022f325cc0
gs32p        = 0xffffffff82bbd1a0
ldt          = 0xffffffff82bbd1e0
tss          = 0xffffffff82bbd1d0
db:0:kdb.enter.default>  bt
Tracing pid 14696 tid 100153 td 0xfffff8000c007000
__mtx_lock_sleep() at __mtx_lock_sleep+0xcd/frame 0xfffffe022f325610
binsfree() at binsfree+0x2bf/frame 0xfffffe022f325660
bqrelse() at bqrelse+0xde/frame 0xfffffe022f325690
ffs_read() at ffs_read+0x281/frame 0xfffffe022f325720
VOP_READ_APV() at VOP_READ_APV+0x7c/frame 0xfffffe022f325750
vn_read() at vn_read+0x186/frame 0xfffffe022f3257d0
vn_io_fault_doio() at vn_io_fault_doio+0x43/frame 0xfffffe022f325830
vn_io_fault1() at vn_io_fault1+0x161/frame 0xfffffe022f325970
vn_io_fault() at vn_io_fault+0x198/frame 0xfffffe022f3259e0
dofileread() at dofileread+0xba/frame 0xfffffe022f325a20
kern_readv() at kern_readv+0x68/frame 0xfffffe022f325a70
sys_read() at sys_read+0x86/frame 0xfffffe022f325ac0
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe022f325bf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe022f325bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800dd2d4a, rsp = 0x7fffffffe678, rbp = 0x7fffffffe690 ---
db:0:kdb.enter.default>  ps

Hmm. possibly a failing drive.

Is the crash report the same every time?

Steve

[kikigak]

Yes crash report is the same every time and I don't know how to interpret crash logs.
How could you possibly say it's a failing drive?
Although SSD mSATA has improper character on BIOS, "SATA PM: ?????? SSD 128GB". I have no extra mSATA yet to replace.
Added more photo on gdrive for reference.

[stephenw10]

There are a number of read functions in the backtrace leading up to the crash which makes me think it's a read error and hence possibly a drive or drive controller issue.
However I would expect some variation. If the backtrace is identical at each crash it's something in software triggering it.

Steve

[kiokoman]

if you can run a system from a usb stick i 'll suggest a memtest and https://www.smartmontools.org/wiki/Download to test the disk
https://www.smartmontools.org/wiki/LiveCDs

[kikigak]

Run memtest as suggested, see 'ref' folder on gdrive.
I've install ClearOS 7.6.0 on 1 of 3 devices we bought, it's up and running 3-days now without crash dump.

[root@xyz-svr-utm ~]# cat /etc/centos-release
ClearOS release 7.6.0 (Final)
[root@xyz-svr-utm ~]# uptime
 10:05:38 up 3 days, 16:46,  2 users,  load average: 0.01, 0.07, 0.08

[kiokoman]

can you disable uefi in the bios?
also i found ppl with similar problem that was able to boot without crash with hw.ibrs_disable=1 and
vm.pmap.pti=0
but this will disable spectre and meltdown mitigation

[kikigak]

Maybe this hardware from ARTWAY is not stable.
Disabled secure boot and set devices to LEGACY only but still UEFI appears on boot devices.

1. Did a clean install choosing non-UEFI USB installer at boot selection (F11) with only admin change at install. This proceeds as usual (large text sizes on installation GUI) but after installation, when USB installer removed, restart then SSD selected on BIOS boot, it says no bootable device. Checking partition says unformatted.

2. Clean install choosing UEFI USB installer at boot selection (F11) with only admin change at install. This proceeds as usual (small text sizes on installation GUI with mouse cursor). After installation, USB installer removed then restart and is working normally.

But on ClearOS it's still up and running 6-days on production without crash dump.

[kikigak]

tried install xcp-ng, it's working fine. setup pfsense vm on xcp-ng and still crash dump.

[kikigak]

replaced ram and now working without crash dump. sigh!

[stephenw10]

Ouch! Nice catch though.