PfSense hangs/restarts intermittently...

Gimpymoo

Hello. First off, thanks for looking and any input is appreciated.

I have a custom built appliance, specs as below:

Intel G5400
4GB RAM
SSD
Intel Dual GB NIC

All been running perfectly for around 18 months.

What started to happen initially was that I would lose internet connectiviity and also lose access to the admin page of PfSense. Also unable to access via SSH.

When I checked the box, I would notice the lights on the ethernet ports were on solid, not flashing.

In this instance, restarting/powering the appliance off would get everything working again.

Initially, I had no Display connected to the appliance so did not really know what was going on.

I did an extensive SMART test on the SSD, this passed OK.

I wiped the HDD and did a fresh install of the latest stable build.

What happens now is that the device does not hang but will restart by itself and all is working.

I then connected a display and noticed at the point of the internet not working, there would be a wall of scrolling text and then it would restart.

I have attached the crash report if it means anything to anyone who can decipher it.

Any help would be appreciated, thank you.

pfsense crash.txt

Rico

Sounds like hardware issue, could be bad RAM.

-Rico

stephenw10

Ok, you appear to have numerous nearly identical crashes there that look like:

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff8125800b
stack pointer	        = 0x28:0xfffffe0100cd21e0
frame pointer	        = 0x28:0xfffffe0100cd21e0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (em1 que)
trap number		= 9
panic: general protection fault
cpuid = 0

db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x860580
curthread    = 0xfffff8003b75c000: pid 45978 "unbound"
curpcb       = 0xfffffe01070aea80
fpcurthread  = 0xfffff8003b75c000: pid 45978 "unbound"
idlethread   = 0xfffff800048b3000: tid 100003 "idle: cpu0"
curpmap      = 0xfffff800084c2138
tssp         = 0xffffffff835a32d0
commontssp   = 0xffffffff835a32d0
rsp0         = 0xfffffe01070aea80
gs32p        = 0xffffffff835a9f28
ldt          = 0xffffffff835a9f68
tss          = 0xffffffff835a9f58
tlb gen      = 2210
db:0:kdb.enter.default>  bt
Tracing pid 45978 tid 100155 td 0xfffff8003b75c000
kdb_enter() at kdb_enter+0x3b/frame 0xfffffe01070ade70
vpanic() at vpanic+0x19b/frame 0xfffffe01070aded0
panic() at panic+0x43/frame 0xfffffe01070adf30
trap_pfault() at trap_pfault/frame 0xfffffe01070adf80
trap() at trap+0x5d/frame 0xfffffe01070ae090
calltrap() at calltrap+0x8/frame 0xfffffe01070ae090
--- trap 0x9, rip = 0xffffffff8125800b, rsp = 0xfffffe01070ae160, rbp = 0xfffffe01070ae160 ---
bcmp() at bcmp+0xb/frame 0xfffffe01070ae160
pf_find_state() at pf_find_state+0xad/frame 0xfffffe01070ae1a0
pf_test_state_udp() at pf_test_state_udp+0x11b/frame 0xfffffe01070ae220
pf_test() at pf_test+0x1cbd/frame 0xfffffe01070ae430
pf_check_out() at pf_check_out+0x1d/frame 0xfffffe01070ae450
pfil_run_hooks() at pfil_run_hooks+0x90/frame 0xfffffe01070ae4e0
ip_output() at ip_output+0xa53/frame 0xfffffe01070ae610
udp_send() at udp_send+0xa0c/frame 0xfffffe01070ae6d0
sosend_dgram() at sosend_dgram+0x345/frame 0xfffffe01070ae730
kern_sendit() at kern_sendit+0x1f9/frame 0xfffffe01070ae7e0
sendit() at sendit+0x19e/frame 0xfffffe01070ae830
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe01070ae880
amd64_syscall() at amd64_syscall+0xa86/frame 0xfffffe01070ae9b0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe01070ae9b0
--- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x801f929ea, rsp = 0x7fffdfdfc4e8, rbp = 0x7fffdfdfc530 ---

You imported the config from the old install to the new one?

Were you running 2.5 there? Because:

<118>Loading configuration......
<118>
<118>*******************************************************************************
<118>* WARNING!                                                                    *
<118>* The current configuration has been created with a newer version of pfSense  *
<118>* than this one! This can lead to serious misbehavior and even security       *
<118>* holes! You are urged to either upgrade to a newer version of pfSense or     *
<118>* revert to the default configuration immediately!                            *
<118>*******************************************************************************
<118>
<118>done.

You might want to rebuild the config or test with something very simple for a while.

Steve

Gimpymoo

@rico said in PfSense hangs/restarts intermittently...:

Sounds like hardware issue, could be bad RAM.

-Rico

I did a MemTest and 15 errors were found in single pass.

Gimpymoo

@stephenw10 said in PfSense hangs/restarts intermittently...:

You might want to rebuild the config or test with something very simple for a while.

Have restored to defaults, will leave it "vanilla" for a while.

Rico

Bingo....you need to swap out that bad RAM module.... Fatal trap 9 is screaming bad memory.

-Rico

stephenw10

Yup, if you are seeing any errors at all in a memtest forget any other explanation!

Gimpymoo

Thanks guys.

Should have known to check the RAM.

Have replaced RAM and running PfSense at factory defaults.

Fingers crossed.

Gimpymoo

Well, over 24 hours, and rock solid since replacing RAM.

Even restored the config file from newer pfSense, not skipped a beat.

Thanks for the help guys.