Need some help. Random pfSense crashes.
-
It's likely it isn't actually crashing the first time it fails. The filesystem panics are probably due to resettting it after whatever the initial issue. What you need to do is find out what that is.
Is it still responsive at the console when the network fails?
Can it still connect out from there?
Look at the message buffer and system logs at the console and see what it's showing.
If the WAN is link-flapping that will create a lot of logs by itself which is unhelpful.
Some Realtek NICs behave badly with the default FreeBSD driver that ships in pfSense. There is an alternative driver you can try by loading it as a kernel module. See for example:
https://forum.netgate.com/post/1064399Normally I would not recommend doing that unless you can definitely see the re NIC(s) failing with the default driver. Typically they might throw some watchdog errors and just stop passing traffic.
Steve
-
@stephenw10
Man!Ok. The console is a great idea. I should’ve thought of it!
Alas, where the internet comes into the house is a good 30 feet from the nearest hdmi video source. I don’t have any HDMI cords that long. My 2 usb ports are also occupied, so no keyboard access either.I’m not trying to be difficult, I swear.
I do have a generic HDMI USB capture card, and a USB Hub. I can connect my laptop to the HDMI out and see what happens.
-
Can you still SSH into it? Or ping it even when this happens?
-
@stephenw10
I can ping it from the LAN port, yes. Although I tried to ssh into it, I realized later that I had changed the SSH port and had forgotten. As such, I’ll need to wait until the next time the issue surfaces to retest. -
Yes, if you can still SSH into it that makes it much easier to find out what's happening.
-
Ok, so. Some progress, I hope.
The first time it happened (earlier today), I connected my laptop, and I couldn’t access the system at all. Couldn’t ping, and couldn’t ssh, nothing.
It happened again just now, at about 21:50 PDT-7. This time, I could ssh in. Alas, I don’t know enough of the inner workings to know for what I should be looking.
Restarting the Web Configurator from the main menu did cause cause the network to come back up, but only for about 3 seconds, and then it was back down again. Selecting the last option (16?) yielded the same result.
I checked the nginx logs, but they were empty. Nginx.log showed the server starting up, but nothing else. Error.log was 0 bytes in size.
I ultimately did a restart with FS check.
Aaaaand, it crashed again after less than 15 minutes! Grrrr!
-
Forgot to mention. When I do get a crash report now, the reason mentions a “page fault in kernel mode”, or some such thing.
-
That's potentially more useful that the filesystem fault. The console message buffer contents and back trace may show us something if you have that crash report.
The main system log is where I would start looking if you're able to SSH in.
Also try to check what is or isn't working. Can you ping out from the console? To LAN clients? To external hosts? By IP and by FQDN?
Steve
-
@stephenw10
Most recent crash around 1 AM (01:00 PDT-7). I was asleep. By 6 AM (06:00), I had no access.Crash report from last night attached.textdump.txt
-
Backtrace:
db:0:kdb.enter.default> bt Tracing pid 16 tid 100070 td 0xfffff80005951740 kdb_enter() at kdb_enter+0x37/frame 0xfffffe000059c500 vpanic() at vpanic+0x197/frame 0xfffffe000059c550 panic() at panic+0x43/frame 0xfffffe000059c5b0 trap_fatal() at trap_fatal+0x391/frame 0xfffffe000059c610 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe000059c660 trap() at trap+0x286/frame 0xfffffe000059c770 calltrap() at calltrap+0x8/frame 0xfffffe000059c770 --- trap 0xc, rip = 0xffffffff80d6f3f7, rsp = 0xfffffe000059c840, rbp = 0xfffffe000059c8c0 --- __mtx_lock_sleep() at __mtx_lock_sleep+0xd7/frame 0xfffffe000059c8c0 ieee80211_node_psq_drain() at ieee80211_node_psq_drain+0x108/frame 0xfffffe000059c910 node_cleanup() at node_cleanup+0x65/frame 0xfffffe000059c940 node_free() at node_free+0x25/frame 0xfffffe000059c960 ieee80211_tx_complete() at ieee80211_tx_complete+0x8c/frame 0xfffffe000059c990 rtwn_bulk_tx_callback() at rtwn_bulk_tx_callback+0x78/frame 0xfffffe000059c9d0 usbd_callback_wrapper() at usbd_callback_wrapper+0x7c6/frame 0xfffffe000059ca30 usb_command_wrapper() at usb_command_wrapper+0xb5/frame 0xfffffe000059ca50 usb_callback_proc() at usb_callback_proc+0xc8/frame 0xfffffe000059ca70 usb_process() at usb_process+0x116/frame 0xfffffe000059cab0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000059caf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000059caf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 ---Panic:
Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x410 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80d6f3f7 stack pointer = 0x28:0xfffffe000059c840 frame pointer = 0x28:0xfffffe000059c8c0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 16 (usbus0) trap number = 12 panic: page fault cpuid = 1 time = 1665380436 KDB: enter: panicThat is in the rtwn(4) driver. You have a Realtek USB wifi device attached. Try removing it.
You also have a USB Ethernet device attached. You should remove that too at least until you have proven it is stable with only the onboard NICs.Steve
-
Interesting. The Realtek Card provides internet to one of the renters here. The USB Ethernet is my LAN Port, which I only use for admin access. It would take me a while to reconfigure my VLANs to use LAN and switch LAN to be the Internal NIC.
Still, none of the other Renters are awake yet, so I suppose I could use the console. I'd just have to shutdown the system and connect to my TV temporarily.
Ok. I'm on it. Thank you so much Steven!
-
USB Ethernet using the axe(4) driver is relatively reliable. But that's not saying much for USB Ethernet!
I would at least remove or disable the USB wifi card though. That's where the crash is happening, it is not stable.Steve
-
Well, it’s been up for a few hours now. No hiccoughs yet.
I was trying to use an Alfa brand WiFi card with the notorious RTL-8821AU. I knew this device has numerous issues in Monitor mode under Linux, but I’ve never tried using it as an AP there. I was looking for a working WiFi card for the one renter, and was pleasantly surprised to find that particular card “working”. None of my others did, and that one has 2x 6 dbi antennas. Little did I know…
That said, I managed to connect an old Apple Airport Express that I had laying around, in place of the Alfa Card. That particular renter has only 1 device: An Xbox One, which is capable of connecting at 5GHz, so I always connect a 5 GHz only signal. 2.4 GHz airwaves are crowded enough.
On the one hand, I used up the last spot on my managed router. On the other hand, 3 of the ports are connected to PoE cameras with separate injectors. I can definitely streamline that with a PoE switch later on. Free up 2 ports on my main switch, and eliminate 2 blocky wall-plugs. Win-win!
Thanks again StevenW10!
-
@aaronouthier said in Need some help. Random pfSense crashes.:
I was trying to use an Alfa brand WiFi card with the notorious RTL-8821AU. I knew this device has numerous issues in Monitor mode under Linux, but I’ve never tried using it as an AP there. I was looking for a working WiFi card for the one renter, and was pleasantly surprised to find that particular card “working”. None of my others did, and that one has 2x 6 dbi antennas. Little did I know…
I would steer clear of WiFi in BSD.
-
Duely-noted.
