Need some help. Random pfSense crashes.
-
Mmm, that's not the normal bad ufs filesystem panic:
dev = ufsid/4effa82c0d8b10f4, block = 6903583, fs = / panic: ffs_blkfree_cg: freeing free frag cpuid = 3 time = 1665104129 KDB: enter: panic panic.txt0600004114317674401 7142 ustarrootwheelffs_blkfree_cg: freeing free fragversion.txt0600007514317674401 7544 ustarrootwheelFreeBSD 12.3-STABLE RELENG_2_6_0-n226742-1285d6d205f pfSenseBut yeah, first reinstall clean. Are you running from eMMC?
-
Correct, running from eMMC.
System hasn’t crashed since running the fs check, but it’s early yet. Now, I need to find a piece of wood to go knock on… -
Ok now, same system, same symptoms, but no relief in sight!
I have wiped the emmc now and reinstalled the OS. An hour later, network access came to a grinding halt.
I did a wireshark trace on both LAN, and on a USB ethernet adapter with a LAN Tap (Hak5 PlunderBug) connected between my opt1 and my managed switch. For reference, opt1 has my vlans going through it, and LAN I normally keep as a dangling ethernet wire for a “failsafe”. You know, in case I do something stupid and mess up my config so badly I can’t access my system via any of my Vlans.
A regular Nmap scan on my LAN port (nmap <ip addr>) reported ports 80 and 443 open, however, nothing seems to be actively listening on port 443, and connecting to port 80 gives an nginx http 500 error with the words “bad gateway” written in large letters. I am able to ping the LAN port, an I am getting a DHCP lease upon disconnecting and reconnecting ethernet.
Also of note: the WAN port seems to be cycling the link light off-on-off-on-etc. every few seconds.
Also, no crash report is being generated any more.
For privacy reasons, I do not wish to post the wireshark dumps in public. If a mod wishes to take a look I can DM them a link to my Nextcloud, or if there is a support email I should contact, let me know.
Is there a way to capture the live log files to, say, a flash drive or something? I think if we had the logs at the time of “death”, it would be very helpful!
-
@aaronouthier
According to this :
https://www.zimaboard.com/zimaboard/productYou have Realtek : 2 x Realtek 8111H - Lan adapters.
And you have : 2 x 6Gb SATA Ports.I'd try to :
1:
Run a serious Memory check - Can be found on a Linux Live BootStick.2:
Find/load the Alternate Realtek Driver3:
Switch to a SATA Disk./Bingo
-
@bingo600 said in Need some help. Random pfSense crashes.:
I'd try to :
1:
Run a serious Memory check - Can be found on a Linux Live BootStick.2:
Find/load the Alternate Realtek Driver3:
Switch to a SATA Disk.-
I can do that.
-
Never heard of such a thing. To the best of my knowledge, you can’t load/run custom software on a pfSense box.
-
I am fresh out of spare SATA disks, and out of money. Also, I don’t need much storage space. Finally, it was quite tricky to wall-mount my ZimaBoard. If I were to install a SATA Drive, it would be hanging/dangling from the bottom of the Zimaboard. I don’t think that would be good. Both of my USB ports are in use, so a flash drive is not possible either.
-
-
It's likely it isn't actually crashing the first time it fails. The filesystem panics are probably due to resettting it after whatever the initial issue. What you need to do is find out what that is.
Is it still responsive at the console when the network fails?
Can it still connect out from there?
Look at the message buffer and system logs at the console and see what it's showing.
If the WAN is link-flapping that will create a lot of logs by itself which is unhelpful.
Some Realtek NICs behave badly with the default FreeBSD driver that ships in pfSense. There is an alternative driver you can try by loading it as a kernel module. See for example:
https://forum.netgate.com/post/1064399Normally I would not recommend doing that unless you can definitely see the re NIC(s) failing with the default driver. Typically they might throw some watchdog errors and just stop passing traffic.
Steve
-
@stephenw10
Man!Ok. The console is a great idea. I should’ve thought of it!
Alas, where the internet comes into the house is a good 30 feet from the nearest hdmi video source. I don’t have any HDMI cords that long. My 2 usb ports are also occupied, so no keyboard access either.I’m not trying to be difficult, I swear.
I do have a generic HDMI USB capture card, and a USB Hub. I can connect my laptop to the HDMI out and see what happens.
-
Can you still SSH into it? Or ping it even when this happens?
-
@stephenw10
I can ping it from the LAN port, yes. Although I tried to ssh into it, I realized later that I had changed the SSH port and had forgotten. As such, I’ll need to wait until the next time the issue surfaces to retest. -
Yes, if you can still SSH into it that makes it much easier to find out what's happening.
-
Ok, so. Some progress, I hope.
The first time it happened (earlier today), I connected my laptop, and I couldn’t access the system at all. Couldn’t ping, and couldn’t ssh, nothing.
It happened again just now, at about 21:50 PDT-7. This time, I could ssh in. Alas, I don’t know enough of the inner workings to know for what I should be looking.
Restarting the Web Configurator from the main menu did cause cause the network to come back up, but only for about 3 seconds, and then it was back down again. Selecting the last option (16?) yielded the same result.
I checked the nginx logs, but they were empty. Nginx.log showed the server starting up, but nothing else. Error.log was 0 bytes in size.
I ultimately did a restart with FS check.
Aaaaand, it crashed again after less than 15 minutes! Grrrr!
-
Forgot to mention. When I do get a crash report now, the reason mentions a “page fault in kernel mode”, or some such thing.
-
That's potentially more useful that the filesystem fault. The console message buffer contents and back trace may show us something if you have that crash report.
The main system log is where I would start looking if you're able to SSH in.
Also try to check what is or isn't working. Can you ping out from the console? To LAN clients? To external hosts? By IP and by FQDN?
Steve
-
@stephenw10
Most recent crash around 1 AM (01:00 PDT-7). I was asleep. By 6 AM (06:00), I had no access.Crash report from last night attached.textdump.txt
-
Backtrace:
db:0:kdb.enter.default> bt Tracing pid 16 tid 100070 td 0xfffff80005951740 kdb_enter() at kdb_enter+0x37/frame 0xfffffe000059c500 vpanic() at vpanic+0x197/frame 0xfffffe000059c550 panic() at panic+0x43/frame 0xfffffe000059c5b0 trap_fatal() at trap_fatal+0x391/frame 0xfffffe000059c610 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe000059c660 trap() at trap+0x286/frame 0xfffffe000059c770 calltrap() at calltrap+0x8/frame 0xfffffe000059c770 --- trap 0xc, rip = 0xffffffff80d6f3f7, rsp = 0xfffffe000059c840, rbp = 0xfffffe000059c8c0 --- __mtx_lock_sleep() at __mtx_lock_sleep+0xd7/frame 0xfffffe000059c8c0 ieee80211_node_psq_drain() at ieee80211_node_psq_drain+0x108/frame 0xfffffe000059c910 node_cleanup() at node_cleanup+0x65/frame 0xfffffe000059c940 node_free() at node_free+0x25/frame 0xfffffe000059c960 ieee80211_tx_complete() at ieee80211_tx_complete+0x8c/frame 0xfffffe000059c990 rtwn_bulk_tx_callback() at rtwn_bulk_tx_callback+0x78/frame 0xfffffe000059c9d0 usbd_callback_wrapper() at usbd_callback_wrapper+0x7c6/frame 0xfffffe000059ca30 usb_command_wrapper() at usb_command_wrapper+0xb5/frame 0xfffffe000059ca50 usb_callback_proc() at usb_callback_proc+0xc8/frame 0xfffffe000059ca70 usb_process() at usb_process+0x116/frame 0xfffffe000059cab0 fork_exit() at fork_exit+0x7e/frame 0xfffffe000059caf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000059caf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 ---Panic:
Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x410 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80d6f3f7 stack pointer = 0x28:0xfffffe000059c840 frame pointer = 0x28:0xfffffe000059c8c0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 16 (usbus0) trap number = 12 panic: page fault cpuid = 1 time = 1665380436 KDB: enter: panicThat is in the rtwn(4) driver. You have a Realtek USB wifi device attached. Try removing it.
You also have a USB Ethernet device attached. You should remove that too at least until you have proven it is stable with only the onboard NICs.Steve
-
Interesting. The Realtek Card provides internet to one of the renters here. The USB Ethernet is my LAN Port, which I only use for admin access. It would take me a while to reconfigure my VLANs to use LAN and switch LAN to be the Internal NIC.
Still, none of the other Renters are awake yet, so I suppose I could use the console. I'd just have to shutdown the system and connect to my TV temporarily.
Ok. I'm on it. Thank you so much Steven!
-
USB Ethernet using the axe(4) driver is relatively reliable. But that's not saying much for USB Ethernet!
I would at least remove or disable the USB wifi card though. That's where the crash is happening, it is not stable.Steve
-
Well, it’s been up for a few hours now. No hiccoughs yet.
I was trying to use an Alfa brand WiFi card with the notorious RTL-8821AU. I knew this device has numerous issues in Monitor mode under Linux, but I’ve never tried using it as an AP there. I was looking for a working WiFi card for the one renter, and was pleasantly surprised to find that particular card “working”. None of my others did, and that one has 2x 6 dbi antennas. Little did I know…
That said, I managed to connect an old Apple Airport Express that I had laying around, in place of the Alfa Card. That particular renter has only 1 device: An Xbox One, which is capable of connecting at 5GHz, so I always connect a 5 GHz only signal. 2.4 GHz airwaves are crowded enough.
On the one hand, I used up the last spot on my managed router. On the other hand, 3 of the ports are connected to PoE cameras with separate injectors. I can definitely streamline that with a PoE switch later on. Free up 2 ports on my main switch, and eliminate 2 blocky wall-plugs. Win-win!
Thanks again StevenW10!
-
@aaronouthier said in Need some help. Random pfSense crashes.:
I was trying to use an Alfa brand WiFi card with the notorious RTL-8821AU. I knew this device has numerous issues in Monitor mode under Linux, but I’ve never tried using it as an AP there. I was looking for a working WiFi card for the one renter, and was pleasantly surprised to find that particular card “working”. None of my others did, and that one has 2x 6 dbi antennas. Little did I know…
I would steer clear of WiFi in BSD.
-
Duely-noted.
