SG-4860 crashing randomly
-
The SG-4860 started crashing randomly, sometimes every 20 minutes to a couple of hours. To resolve, have to hard reset/reboot. Running latest 2.3.4 release. IP address is 97.68.22.234
Just before the crash happens, the following messages begin to show in the Console:
Timeout on slot 2 port 0
ahcich1: is 00000000 cs 0001fffc ss 0001fffc rs 0001fffc tfd c0 serr 00000000 cmd 0000c217
(ada0:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 8f 8a 12 40 01 00 00 00 00 00
(ada0:ahcich1:0:0:0): CAM status: Command timeout
(ada0:ahcich1:0:0:0): Retrying command
ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich1: Timeout on slot 17 port 0
ahcich1: is 00000000 cs 00020000 ss 00000000 rs 00020000 tfd 80 serr 00000000 cmd 0000d117
(aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich1:0:0:0): CAM status: Command timeout
(aprobe0:ahcich1:0:0:0): Retrying command
ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich1: Timeout on slot 18 port 0
ahcich1: is 00000000 cs 00040000 ss 00000000 rs 00040000 tfd 80 serr 00000000 cmd 0000d217
(aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich1:0:0:0): CAM status: Command timeout
(aprobe0:ahcich1:0:0:0): Error 5, Retries exhaustedTimeout on slot 20 port 0
ahcich1: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 80 serr 00000000 cmd 0000d417
(aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich1:0:0:0): CAM status: Command timeout
(aprobe0:ahcich1:0:0:0): Error 5, Retry was blocked
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <intel ssdsc2bw120h6="" rg21="">s/n CVTR54460424120AGN detached
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=15759245312, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=15759278080, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=15759310848, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=15759343616, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=15759376384, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=49894686720, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=76154929152, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=76154961920, length=32768)]error = 6
/dev: got error 6 while accessing filesystem
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=78780989440, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=78781022208, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=78781284352, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=78781317120, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=84689551360, length=32768)]error = 6
g_vfs_done():ufsid/57165dd214494c66[WRITE(offset=99789242368, length=32768)]error = 6
ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
(aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich1:0:0:0): CAM status: Command timeout
(aprobe0:ahcich1:0:0:0): Retrying command
ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich1: Timeout on slot 22 port 0
ahcich1: is 00000000 cs 00400000 ss 00000000 rs 00400000 tfd 80 serr 00000000 cmd 0000d617
(aprobe0:ahcich1:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich1:0:0:0): CAM status: Command timeout
(aprobe0:ahcich1:0:0:0): Error 5, Retries exhausted
ahcich1: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich1: Poll timeout on slot 23 port 0
ahcich1: is 00000000 cs 00800000 ss 00000000 rs 00800000 tfd 80 serr 00000000 cmd 0000d717
(aprobe0:ahcich1:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich1:0:0:0): CAM status: Command timeout
(aprobe0:ahcich1:0:0:0): Error 5, Retries exhaustedThen there’s a memory dump. After that:
SeaBIOS (version SageBIOS-20160122_083040-localhost.localdomain)
iPXE (http://ipxe.org) 00:14.0 C100 PCI2.10 PnP PMM+7FB7AF80+7FADAF80 C100
iPXE (http://ipxe.org) 00:14.1 C200 PCI2.10 PnP PMM 7FB7AF80 7FADAF80 C200
iPXE (http://ipxe.org) 00:14.2 C300 PCI2.10 PnP PMM 7FB7AF80 7FADAF80 C300
iPXE (http://ipxe.org) 00:14.3 C400 PCI2.10 PnP PMM 7FB7AF80 7FADAF80 C400
Press F12 for boot menu.
Boot failed: not a bootable disk
Booting from ROM…
iPXE (PCI 00:14.0) starting execution...ok
iPXE initialising devices...okiPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXTnet0: 00:08:a2:09:c9:90 using i354-ext on PCI00:14.0 (open)
[Waiting for link-up on net0….............. Down (http://ipxe.org/38086101)
No more network devicesBooting from ROM...
iPXE (PCI 00:14.1) starting execution...ok
iPXE initialising devices...okiPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXTnet1: 00:08:a2:09:c9:91 using i354-ext on PCI00:14.1 (open)](<br />Waiting for link-up on net0................. Down (http://ipxe.org/38086101)<br />No more network devices<br /><br />Booting from ROM... <br />iPXE (PCI 00:14.1) starting execution...ok<br />iPXE initialising devices...ok<br /><br />iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org<br />Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT<br /><br />net1: 00:08:a2:09:c9:91 using i354-ext on PCI00:14.1 (open)<br /> [link]<br /> [link]<br />Waiting for link-up on net1..........<br /><br />No more network devices<br /><br />Booting from ROM... <br />iPXE (PCI 00:14.2) starting execution...ok<br />iPXE initialising devices...ok<br /><br />iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org<br />Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT<br /><br />net2: 00:08:a2:09:c9:92 using i354-ext on PCI00:14.2 (open)<br /> [link]<br /> [link]<br />Waiting for link-up on net2...........<br /><br />iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org<br />Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT<br /><br />net3: 00:08:a2:09:c9:93 using i354-ext on PCI00:14.3 (open)<br /> [link]<br /> [link]<br />Waiting for link-up on net3........<br /><br />Booting from CBFS... <br /><br /><br /> Sortbootorder payload ver 1.1 Sage Electronic Engineering <br />**<br />flash_address = 0x700000<br />==============================================<br />Type lower case letter to move device to top<br />==============================================<br />boot devices<br /><br /> a USB 1<br /> b USB 2<br /> c SATA3 1<br /> d SATA3 2<br /> e msata<br /> f SATA2 1<br /> g SATA2 2<br /> h SATA2 3<br /> i USB-eMMC<br /> j PXE 1<br /> k PXE 2<br /> l PXE 3 <br /> m PXE 4<br />==============================================<br />Type upper case letter to invoke action<br />==============================================<br /> R Restore boot order defaults<br /> E Exit setup with save<br /> X Exit setup without save<br /><br />> <br /><br /><br />Does this mean that a storage device is failing?<br />[/link][/link][/link][/link][/link][/link]) [Waiting for link-up on net1….......
No more network devices
Booting from ROM...
iPXE (PCI 00:14.2) starting execution...ok
iPXE initialising devices...okiPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXTnet2: 00:08:a2:09:c9:92 using i354-ext on PCI00:14.2 (open)](<br />Waiting for link-up on net1..........<br /><br />No more network devices<br /><br />Booting from ROM... <br />iPXE (PCI 00:14.2) starting execution...ok<br />iPXE initialising devices...ok<br /><br />iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org<br />Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT<br /><br />net2: 00:08:a2:09:c9:92 using i354-ext on PCI00:14.2 (open)<br /> [link]<br /> [link]<br />Waiting for link-up on net2...........<br /><br />iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org<br />Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT<br /><br />net3: 00:08:a2:09:c9:93 using i354-ext on PCI00:14.3 (open)<br /> [link]<br /> [link]<br />Waiting for link-up on net3........<br /><br />Booting from CBFS... <br /><br /><br /> Sortbootorder payload ver 1.1 Sage Electronic Engineering <br />**<br />flash_address = 0x700000<br />==============================================<br />Type lower case letter to move device to top<br />==============================================<br />boot devices<br /><br /> a USB 1<br /> b USB 2<br /> c SATA3 1<br /> d SATA3 2<br /> e msata<br /> f SATA2 1<br /> g SATA2 2<br /> h SATA2 3<br /> i USB-eMMC<br /> j PXE 1<br /> k PXE 2<br /> l PXE 3 <br /> m PXE 4<br />==============================================<br />Type upper case letter to invoke action<br />==============================================<br /> R Restore boot order defaults<br /> E Exit setup with save<br /> X Exit setup without save<br /><br />> <br /><br /><br />Does this mean that a storage device is failing?<br />[/link][/link][/link][/link]) [Waiting for link-up on net2…........
iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXTnet3: 00:08:a2:09:c9:93 using i354-ext on PCI00:14.3 (open)](<br />Waiting for link-up on net2...........<br /><br />iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org<br />Features: iSCSI HTTP DNS TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT<br /><br />net3: 00:08:a2:09:c9:93 using i354-ext on PCI00:14.3 (open)<br /> [link]<br /> [link]<br />Waiting for link-up on net3........<br /><br />Booting from CBFS... <br /><br /><br /> Sortbootorder payload ver 1.1 Sage Electronic Engineering <br />**<br />flash_address = 0x700000<br />==============================================<br />Type lower case letter to move device to top<br />==============================================<br />boot devices<br /><br /> a USB 1<br /> b USB 2<br /> c SATA3 1<br /> d SATA3 2<br /> e msata<br /> f SATA2 1<br /> g SATA2 2<br /> h SATA2 3<br /> i USB-eMMC<br /> j PXE 1<br /> k PXE 2<br /> l PXE 3 <br /> m PXE 4<br />==============================================<br />Type upper case letter to invoke action<br />==============================================<br /> R Restore boot order defaults<br /> E Exit setup with save<br /> X Exit setup without save<br /><br />> <br /><br /><br />Does this mean that a storage device is failing?<br />[/link][/link]) [Waiting for link-up on net3….....
Booting from CBFS...
*** Sortbootorder payload ver 1.1 Sage Electronic Engineering *
flash_address = 0x700000
Type lower case letter to move device to top
boot devices
a USB 1
b USB 2
c SATA3 1
d SATA3 2
e msata
f SATA2 1
g SATA2 2
h SATA2 3
i USB-eMMC
j PXE 1
k PXE 2
l PXE 3
m PXE 4Type upper case letter to invoke action
R Restore boot order defaults
E Exit setup with save
X Exit setup without saveDoes this mean that a storage device is failing?](<br />Waiting for link-up on net3........<br /><br />Booting from CBFS... <br /><br /><br /> Sortbootorder payload ver 1.1 Sage Electronic Engineering <br />**<br />flash_address = 0x700000<br />==============================================<br />Type lower case letter to move device to top<br />==============================================<br />boot devices<br /><br /> a USB 1<br /> b USB 2<br /> c SATA3 1<br /> d SATA3 2<br /> e msata<br /> f SATA2 1<br /> g SATA2 2<br /> h SATA2 3<br /> i USB-eMMC<br /> j PXE 1<br /> k PXE 2<br /> l PXE 3 <br /> m PXE 4<br />==============================================<br />Type upper case letter to invoke action<br />==============================================<br /> R Restore boot order defaults<br /> E Exit setup with save<br /> X Exit setup without save<br /><br />> <br /><br /><br />Does this mean that a storage device is failing?<br />)</intel>
-
That would be my guess.
Looks like ada0 should be mSATA. You can either back up the config, remove the mSATA, and reinstall to the eMMC and see if that clears the problem (if you're not using the extra storage of course) or try another mSATA. Doesn't look good for that one.
-
It's the mSATA drive which is an Intel 120GB SSD installed by PfSense when unit was purchased. Replaced it with an old 120GB laptop hard drive, reinstalled the OS, then restored the backup. So far everything seems to be working great so far.
What's lousy is that the unit was still in warranty when this all started. By the time i was able to troubleshoot and figure things out, the warranty has since expired. What's interesting is that the Intel SSD has a five year warranty.
One thing that, from my perspective, which is not fair, to get the image file, one has to pay for support or pay $19.00 for a thumb drive to be sent. Yes, $19.00 is not the end of the world, but waiting even a day to receive the thumb drive is not an option. This has to be up and running now. Since the updates are free, then why can't the image file be free if one can prove ownership of a PfSense device?