Crash on 2.2.6 on Hyper-V

robwalker

Hi I just logged in to our master firewall node and saw it had crashed and rebooted. The crash report will have been submitted from 51.255.139.252 between midnight and 3AM UTC on 1st May 2016.

There is at least one reference to the syncer process but I'm really not sure I'm looking at the right thing. Pfsense is running as a Hyper-V VM.

robwalker

Sorry the time above is the time of the crash. It will have been submitted between 3 and 5 pm UTC+1 on 3 May 2016 :D

cmb

Crash was in something disk-related. No apparent reason or hints from the backtrace at a quick review.

db:0:kdb.enter.default>  bt
Tracing pid 20 tid 100089 td 0xfffff80003bb0490
softdep_disk_io_initiation() at softdep_disk_io_initiation+0x43f/frame 0xfffffe00003f17e0
ffs_geom_strategy() at ffs_geom_strategy+0x15e/frame 0xfffffe00003f1810
bufwrite() at bufwrite+0x142/frame 0xfffffe00003f1850
vfs_bio_awrite() at vfs_bio_awrite+0x3d1/frame 0xfffffe00003f1900
vop_stdfsync() at vop_stdfsync+0x241/frame 0xfffffe00003f1970
devfs_fsync() at devfs_fsync+0x26/frame 0xfffffe00003f19a0
VOP_FSYNC_APV() at VOP_FSYNC_APV+0xa7/frame 0xfffffe00003f19d0
sched_sync() at sched_sync+0x3ad/frame 0xfffffe00003f1a70
fork_exit() at fork_exit+0x9a/frame 0xfffffe00003f1ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00003f1ab0
--- trap 0, rip = 0, rsp = 0xfffffe00003f1b70, rbp = 0 ---

The storage in Hyper-V local to that system? Often when people hit disk-related crashes in a VM it's because the hypervisor lost connectivity to its NFS/iSCSI/other network attached storage.

robwalker

Thanks for the info, that's much appreciated. The storage is local. 2 SSDs in software RAID 1. I'll have a look in to it further but at first glance there were no errors logged at that time in the host OS. I may have missed something though.

The only other thing that comes to mind is possibly the disks going to sleep or 'spinning down'. Not that they actually spin in this case! I'll check the settings.

cmb

Yeah not likely disappearing disk in that case, that's usually with network-attached storage of some sort. But worth checking. More likely it's either a Hyper-V issue of some sort, or an issue with FreeBSD and Hyper-V. If it doesn't recur, it's probably safe to ignore.

robwalker

We had another crash this morning. I submitted the report a few mins ago from the same IP (few mins after 9am UTC+1)

I'm not sure the cause is the same.

cmb

That one's much different, it's in IPsec. It looks like a known crash there that's confirmed fixed in 2.3.

But if you have > 1 vCPU in your VM, you'll want to change it to 1 vCPU until 2.3.1 to avoid hitting this.
https://redmine.pfsense.org/issues/6296