pfsense crashed

ykazari

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x60
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80cc10ef
stack pointer	        = 0x28:0xfffffe0079672e00
frame pointer	        = 0x28:0xfffffe0079672e00
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (em2 taskq)

Gertjan

@ykazari said in pfsense crashed:

Same swap error, system has never had a issue before

swap error ? I vote for hardware issue right away.
Next best : a pretty gore OOM error.

Since when processor instructions = (== pfSEnse) wear out ?
Devices break down. happens all the time.

If you were running Windows on your device, would you thank Microsoft ?

fertig

@gertjan said in pfsense crashed:

swap error ? I vote for hardware issue right away.

we have one R610 in cold reserve and exchanged the hardware of the first failed firewall yesterday and switched back to this master in the evening.
This morning the firewall crashed again.
So we reproduced this issue on 3 absolutly identical Dell R610.
All have 16GB RAM, one 4x1GbitE Broadcom onboard, one 2x1GbitE Intel card, same bios, same lifecycle controller version.

I'm pretty sure, that this is a issue with 2.4.3p1.

Christian

al_no

Just for the possibility to reproduce the error for the development guys, we are had the issue with three similar dell r610 servers with the exact same setup.
These servers do have an LAG Bond with one port on the boradcom and one port on the intel network cards. For both cards it is the first port. The Switches used are Brocade icx6610.

In the logfiles we do see that the bce0 interface does crash and that the firewall will start to do the carp magic and directly return as the network interface will only be unavailable for 4 sek. We did see that some crashes happend two or three times in a row parted with a reboot of the firewall.

Actually we did deactivate the boradcom network interface to quick and dirty solve the problem for us.

Would anybody need the logfiles or how do we procede?

Thanks

Alex

linkdragon70

Hello everyone,

You seem to have the same problem as me. The Pfsense of my company is also on version 2.4.3.

By cons, it is an r310, we use the ports of the motherboard for the network.

Here is the error I receive:

bce0: /builder/ce-243/tmp/FreeBSD-src/sys/bce/if_bce.c(7886): Watchdog timeout occured, resseting!
bce0: Gigabit link up!

The bce0 interface serves as a bgp transit.
The problem happened suddenly.

Does the update to pfsense 2.4.4 fix your problem?

Thank you!

stephenw10

Do you actually see a crash report like the others in this thread or just the Watchdog error in the system log?

Steve

linkdragon70

Hello,

Our Pfsense router starts to display the message as a warning, but the network cuts when the warning is issued. Then after 5-6 warning, the router eventually crashed completely and I need to restart it.

We have also tried on our 2nd interface bc1, and even try to put another network card, but the problem persists. I hesitate to update Pfsense to 2.4.4 to see if the problem comes from there ...

Thank you!

Gabriel

stephenw10

Did it have a crash report though? If so we need to see the panic message and backtrace from it as shown above for those systems.

Steve

linkdragon70

Hello Stephen,

Could you tell me where is the message? Because I'm in shell on the router, the crash is actually freezing it completely and displaying the same thing many times (/builder/ce-243/tmp/FreeBSD-src/sys/bce/if_bce.c(7886) : Watchdog timeout occured, resseting!)

thank you,

Gabriel

stephenw10

You would see it reported as an alert on the dashboard in the GUI.
Any crash reports would be in /var/crash.

Steve

linkdragon70

Thank you Steven,
Finally we virtualize Pfsense with Proxmox.

Gab