Help - My firewall keeps crashing and I don't know why

CaptainHook

Hello,

I am running a custom hardware firewall. Hardware is a Zotact Zbox CI329 Nano. (Intel Celeron NI-4100 4 core 4 thread 1.10 GHz base) 8 GB of DDR4, and a 256GB SSD. When the firewall experiences high bandwidth draw for at least 5 minutes it is prone to loosing its IP from my router. (ISP is Xfinity 800 down 20 up) When I lose connection, I can still access my firewall but if I unplug either WAN or LAN patch cables it 100% craps out and I am unable to get to it without a reboot. Once the connection is lost from the ISP I still have to do a reboot but can either plug the physical plug or reboot from inside the web GUI. Running latest stable version 2.6.0amd and Free BSD version 12.3 stable. I can provide any additional config details as needed.

I am stumped and don't know why this keeps happening.

Thank you in advance for any help.

NollipfSense

@captainhook You'll need to provide system logs: Status > System Logs > System > General.

stephenw10

Is it still accessible at the console directly when this happens? It doesn't sound like it's crashing, more like some network config issue.

Steve

CaptainHook

Log.txt This is all that is in that log file.

stephenw10

This is the most telling line there for me:

Jan 4 21:13:06	kernel		re0: watchdog timeout

There's a good chance re0 eventually stops responding entirely after an event like that. You might need to try the alternative Realtek driver.

Steve

CaptainHook

Ok, is there an easy way to do that without resetting my firewall or should I just factory it and do a whole new install on the hardware?

And thank you for you help.

stephenw10

No need to reinstall, do this: https://forum.netgate.com/post/1072719

Have a plan to roll back though. If, for some reason, your NICs don't like that driver you'll need OOB access to remove it.

Steve

CaptainHook

Hello Sir,

I was a little slow on being able to apply the commands as I am still learning. I finally got it installed today for the package that you referenced. I do not know how to check the status of the driver but the command did run without error.

Thank you for your help and I will let you know if the problem persists.

Thank you,
Captain Hook

stephenw10

After rebooting you should see the new driver version reported in the boot log if it is loading.

CaptainHook

Well, that didn't seem to work. Last night it crashed again. I attempted to get into the GUI, and it allowed me to sign in but then the webpage froze, and I had to pull the plug for a hard reboot. I have several text docs that I will try to upload when I make it to work and make sure all sensitive data is cleaned up from them.

stephenw10

When you say 'crashed' is it actually crashing? Do you see a crash report after rebooting?

Or does it just stop responding? Still active at the console?

CaptainHook

@stephenw10 I got one when I did a Reroot of the system but when these glitches happen it does not seem to generate a report. It just kind of freezes and won't come back. Crashing might not be the correct term for it in the technical aspect. I have almost finished up with the logs I have and will be uploading what I have in a few.

Thank you for your continued help.

stephenw10

Were you able to confirm the alternative driver is loading from the boot log?

CaptainHook

@stephenw10 I was not. I did my best to try and find it but unfortunately I was unable to confirm that it changed.

Here are the 4 files I have regarding the logs from the firewall.Routing Edited.txt info.0 General Edit.txt Gateways edit.txt

I will not be able to make it home till 6:30 Central Standard time to look at any information on the device as I do not have a way into it outside of my house.

Thank you,
Thomas Hook

stephenw10

If the driver is loading the boot log should contain an entry like:

re0: version:1.97.00

Your log doesn't appear to have that and the gateway issues coinside with the watchdog timeouts.

So I would make sure that driver is loading before doing anything else.

Steve

CaptainHook

@stephenw10 Hello sir,

I will try and find the specified bit of information now. I did get a crash log today when I signed in.

crash.txt

stephenw10

That looks like a UFS issue, it's trying to remount after shutting down:

<118>pfSense is now shutting down ...
<118>
<6>pflog0: promiscuous mode disabled
Trying to mount root from ufs:/dev/ufsid/628e817812c330cd [rw,noatime]...
panic: vm_fault: fault on nofault entry, addr: 0xffffffff83d93000
cpuid = 2
time = 1673396073
KDB: enter: panic

You should run a filesystem check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

You can see from logs there though that it isn't running the alternative driver:

re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xa1204000-0xa1204fff,0xa1200000-0xa1203fff at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: turning off MSI enable bit.
re0: ASPM disabled
re0: Chip rev. 0x4c000000
re0: MAC rev. 0x00000000

Steve

CaptainHook

@stephenw10 Hello sir,

Ran a reboot with file system check. attempted to do the driver install again and this is what I got. I do apologies as I am not that adept and linux based command line nor SSH The device is in a tricky spot due to space constraints and if I still am unable to get this to use the new driver I will have to try to take it down to do the single user file system check in the linked document.

stephenw10

Did you also run the lines to add the loader.conf.local values?

https://forum.netgate.com/post/1072719

CaptainHook

@stephenw10 Hello Sir,

I had to end up getting the device down and bringing it into my work and having my boss help me as I was unable to get the new driver loaded. He helped me with the command line and we believe that we finally have the driver installed. I will have to wait till I get home to see if the problem persists.

Thank you again for you time and help!