Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Help - My firewall keeps crashing and I don't know why

    Scheduled Pinned Locked Moved Hardware
    21 Posts 3 Posters 2.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • CaptainHookC
      CaptainHook
      last edited by

      Hello,

      I am running a custom hardware firewall. Hardware is a Zotact Zbox CI329 Nano. (Intel Celeron NI-4100 4 core 4 thread 1.10 GHz base) 8 GB of DDR4, and a 256GB SSD. When the firewall experiences high bandwidth draw for at least 5 minutes it is prone to loosing its IP from my router. (ISP is Xfinity 800 down 20 up) When I lose connection, I can still access my firewall but if I unplug either WAN or LAN patch cables it 100% craps out and I am unable to get to it without a reboot. Once the connection is lost from the ISP I still have to do a reboot but can either plug the physical plug or reboot from inside the web GUI. Running latest stable version 2.6.0amd and Free BSD version 12.3 stable. I can provide any additional config details as needed.

      I am stumped and don't know why this keeps happening.

      Thank you in advance for any help.

      NollipfSenseN CaptainHookC 2 Replies Last reply Reply Quote 0
      • NollipfSenseN
        NollipfSense @CaptainHook
        last edited by

        @captainhook You'll need to provide system logs: Status > System Logs > System > General.

        pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
        pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

        1 Reply Last reply Reply Quote 2
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Is it still accessible at the console directly when this happens? It doesn't sound like it's crashing, more like some network config issue.

          Steve

          1 Reply Last reply Reply Quote 0
          • CaptainHookC
            CaptainHook @CaptainHook
            last edited by

            Log.txt This is all that is in that log file.

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              This is the most telling line there for me:

              Jan 4 21:13:06	kernel		re0: watchdog timeout
              

              There's a good chance re0 eventually stops responding entirely after an event like that. You might need to try the alternative Realtek driver.

              Steve

              1 Reply Last reply Reply Quote 0
              • CaptainHookC
                CaptainHook
                last edited by

                Ok, is there an easy way to do that without resetting my firewall or should I just factory it and do a whole new install on the hardware?

                And thank you for you help.

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  No need to reinstall, do this: https://forum.netgate.com/post/1072719

                  Have a plan to roll back though. If, for some reason, your NICs don't like that driver you'll need OOB access to remove it.

                  Steve

                  1 Reply Last reply Reply Quote 0
                  • CaptainHookC
                    CaptainHook
                    last edited by

                    Hello Sir,

                    I was a little slow on being able to apply the commands as I am still learning. I finally got it installed today for the package that you referenced. I do not know how to check the status of the driver but the command did run without error.

                    Thank you for your help and I will let you know if the problem persists.

                    Thank you,
                    Captain Hook

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      After rebooting you should see the new driver version reported in the boot log if it is loading.

                      1 Reply Last reply Reply Quote 0
                      • CaptainHookC
                        CaptainHook
                        last edited by

                        Well, that didn't seem to work. Last night it crashed again. I attempted to get into the GUI, and it allowed me to sign in but then the webpage froze, and I had to pull the plug for a hard reboot. I have several text docs that I will try to upload when I make it to work and make sure all sensitive data is cleaned up from them.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          When you say 'crashed' is it actually crashing? Do you see a crash report after rebooting?

                          Or does it just stop responding? Still active at the console?

                          CaptainHookC 1 Reply Last reply Reply Quote 0
                          • CaptainHookC
                            CaptainHook @stephenw10
                            last edited by

                            @stephenw10 I got one when I did a Reroot of the system but when these glitches happen it does not seem to generate a report. It just kind of freezes and won't come back. Crashing might not be the correct term for it in the technical aspect. I have almost finished up with the logs I have and will be uploading what I have in a few.

                            Thank you for your continued help.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Were you able to confirm the alternative driver is loading from the boot log?

                              CaptainHookC 1 Reply Last reply Reply Quote 0
                              • CaptainHookC
                                CaptainHook @stephenw10
                                last edited by

                                @stephenw10 I was not. I did my best to try and find it but unfortunately I was unable to confirm that it changed.

                                Here are the 4 files I have regarding the logs from the firewall.Routing Edited.txt info.0 General Edit.txt Gateways edit.txt

                                I will not be able to make it home till 6:30 Central Standard time to look at any information on the device as I do not have a way into it outside of my house.

                                Thank you,
                                Thomas Hook

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by stephenw10

                                  If the driver is loading the boot log should contain an entry like:

                                  re0: version:1.97.00
                                  

                                  Your log doesn't appear to have that and the gateway issues coinside with the watchdog timeouts.

                                  So I would make sure that driver is loading before doing anything else.

                                  Steve

                                  CaptainHookC 1 Reply Last reply Reply Quote 0
                                  • CaptainHookC
                                    CaptainHook @stephenw10
                                    last edited by

                                    @stephenw10 Hello sir,

                                    I will try and find the specified bit of information now. I did get a crash log today when I signed in.

                                    crash.txt

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      That looks like a UFS issue, it's trying to remount after shutting down:

                                      <118>pfSense is now shutting down ...
                                      <118>
                                      <6>pflog0: promiscuous mode disabled
                                      Trying to mount root from ufs:/dev/ufsid/628e817812c330cd [rw,noatime]...
                                      panic: vm_fault: fault on nofault entry, addr: 0xffffffff83d93000
                                      cpuid = 2
                                      time = 1673396073
                                      KDB: enter: panic
                                      

                                      You should run a filesystem check:
                                      https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

                                      You can see from logs there though that it isn't running the alternative driver:

                                      re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xa1204000-0xa1204fff,0xa1200000-0xa1203fff at device 0.0 on pci1
                                      re0: Using 1 MSI-X message
                                      re0: turning off MSI enable bit.
                                      re0: ASPM disabled
                                      re0: Chip rev. 0x4c000000
                                      re0: MAC rev. 0x00000000
                                      

                                      Steve

                                      CaptainHookC 1 Reply Last reply Reply Quote 0
                                      • CaptainHookC
                                        CaptainHook @stephenw10
                                        last edited by

                                        @stephenw10 Hello sir,

                                        Ran a reboot with file system check. attempted to do the driver install again and this is what I got. I do apologies as I am not that adept and linux based command line nor SSH The device is in a tricky spot due to space constraints and if I still am unable to get this to use the new driver I will have to try to take it down to do the single user file system check in the linked document.

                                        e55aa4fc-4423-4da9-8194-c990190d21fb-image.png

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Did you also run the lines to add the loader.conf.local values?

                                          https://forum.netgate.com/post/1072719

                                          CaptainHookC 1 Reply Last reply Reply Quote 0
                                          • CaptainHookC
                                            CaptainHook @stephenw10
                                            last edited by

                                            @stephenw10 Hello Sir,

                                            I had to end up getting the device down and bringing it into my work and having my boss help me as I was unable to get the new driver loaded. He helped me with the command line and we believe that we finally have the driver installed. I will have to wait till I get home to see if the problem persists.

                                            Thank you again for you time and help!

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.