Need some help. Random pfSense crashes.

aaronouthier

@bingo600 said in Need some help. Random pfSense crashes.:

I'd try to :
1:
Run a serious Memory check - Can be found on a Linux Live BootStick.

2:
Find/load the Alternate Realtek Driver

3:
Switch to a SATA Disk.

I can do that.
Never heard of such a thing. To the best of my knowledge, you can’t load/run custom software on a pfSense box.
I am fresh out of spare SATA disks, and out of money. Also, I don’t need much storage space. Finally, it was quite tricky to wall-mount my ZimaBoard. If I were to install a SATA Drive, it would be hanging/dangling from the bottom of the Zimaboard. I don’t think that would be good. Both of my USB ports are in use, so a flash drive is not possible either.

stephenw10

It's likely it isn't actually crashing the first time it fails. The filesystem panics are probably due to resettting it after whatever the initial issue. What you need to do is find out what that is.

Is it still responsive at the console when the network fails?

Can it still connect out from there?

Look at the message buffer and system logs at the console and see what it's showing.

If the WAN is link-flapping that will create a lot of logs by itself which is unhelpful.

Some Realtek NICs behave badly with the default FreeBSD driver that ships in pfSense. There is an alternative driver you can try by loading it as a kernel module. See for example:
https://forum.netgate.com/post/1064399

Normally I would not recommend doing that unless you can definitely see the re NIC(s) failing with the default driver. Typically they might throw some watchdog errors and just stop passing traffic.

Steve

aaronouthier

@stephenw10
Man!

Ok. The console is a great idea. I should’ve thought of it!
Alas, where the internet comes into the house is a good 30 feet from the nearest hdmi video source. I don’t have any HDMI cords that long. My 2 usb ports are also occupied, so no keyboard access either.

I’m not trying to be difficult, I swear.

I do have a generic HDMI USB capture card, and a USB Hub. I can connect my laptop to the HDMI out and see what happens.

stephenw10

Can you still SSH into it? Or ping it even when this happens?

aaronouthier

@stephenw10
I can ping it from the LAN port, yes. Although I tried to ssh into it, I realized later that I had changed the SSH port and had forgotten. As such, I’ll need to wait until the next time the issue surfaces to retest.

stephenw10

Yes, if you can still SSH into it that makes it much easier to find out what's happening.

aaronouthier

Ok, so. Some progress, I hope.

The first time it happened (earlier today), I connected my laptop, and I couldn’t access the system at all. Couldn’t ping, and couldn’t ssh, nothing.

It happened again just now, at about 21:50 PDT-7. This time, I could ssh in. Alas, I don’t know enough of the inner workings to know for what I should be looking.

Restarting the Web Configurator from the main menu did cause cause the network to come back up, but only for about 3 seconds, and then it was back down again. Selecting the last option (16?) yielded the same result.

I checked the nginx logs, but they were empty. Nginx.log showed the server starting up, but nothing else. Error.log was 0 bytes in size.

I ultimately did a restart with FS check.

Aaaaand, it crashed again after less than 15 minutes! Grrrr!

aaronouthier

Forgot to mention. When I do get a crash report now, the reason mentions a “page fault in kernel mode”, or some such thing.

stephenw10

That's potentially more useful that the filesystem fault. The console message buffer contents and back trace may show us something if you have that crash report.

The main system log is where I would start looking if you're able to SSH in.

Also try to check what is or isn't working. Can you ping out from the console? To LAN clients? To external hosts? By IP and by FQDN?

Steve

aaronouthier

@stephenw10
Most recent crash around 1 AM (01:00 PDT-7). I was asleep. By 6 AM (06:00), I had no access.

Crash report from last night attached.textdump.txt

stephenw10

Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 16 tid 100070 td 0xfffff80005951740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe000059c500
vpanic() at vpanic+0x197/frame 0xfffffe000059c550
panic() at panic+0x43/frame 0xfffffe000059c5b0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe000059c610
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe000059c660
trap() at trap+0x286/frame 0xfffffe000059c770
calltrap() at calltrap+0x8/frame 0xfffffe000059c770
--- trap 0xc, rip = 0xffffffff80d6f3f7, rsp = 0xfffffe000059c840, rbp = 0xfffffe000059c8c0 ---
__mtx_lock_sleep() at __mtx_lock_sleep+0xd7/frame 0xfffffe000059c8c0
ieee80211_node_psq_drain() at ieee80211_node_psq_drain+0x108/frame 0xfffffe000059c910
node_cleanup() at node_cleanup+0x65/frame 0xfffffe000059c940
node_free() at node_free+0x25/frame 0xfffffe000059c960
ieee80211_tx_complete() at ieee80211_tx_complete+0x8c/frame 0xfffffe000059c990
rtwn_bulk_tx_callback() at rtwn_bulk_tx_callback+0x78/frame 0xfffffe000059c9d0
usbd_callback_wrapper() at usbd_callback_wrapper+0x7c6/frame 0xfffffe000059ca30
usb_command_wrapper() at usb_command_wrapper+0xb5/frame 0xfffffe000059ca50
usb_callback_proc() at usb_callback_proc+0xc8/frame 0xfffffe000059ca70
usb_process() at usb_process+0x116/frame 0xfffffe000059cab0
fork_exit() at fork_exit+0x7e/frame 0xfffffe000059caf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000059caf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Panic:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address	= 0x410
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d6f3f7
stack pointer	        = 0x28:0xfffffe000059c840
frame pointer	        = 0x28:0xfffffe000059c8c0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 16 (usbus0)
trap number		= 12
panic: page fault
cpuid = 1
time = 1665380436
KDB: enter: panic

That is in the rtwn(4) driver. You have a Realtek USB wifi device attached. Try removing it.
You also have a USB Ethernet device attached. You should remove that too at least until you have proven it is stable with only the onboard NICs.

Steve

aaronouthier

Interesting. The Realtek Card provides internet to one of the renters here. The USB Ethernet is my LAN Port, which I only use for admin access. It would take me a while to reconfigure my VLANs to use LAN and switch LAN to be the Internal NIC.

Still, none of the other Renters are awake yet, so I suppose I could use the console. I'd just have to shutdown the system and connect to my TV temporarily.

Ok. I'm on it. Thank you so much Steven!

stephenw10

USB Ethernet using the axe(4) driver is relatively reliable. But that's not saying much for USB Ethernet!
I would at least remove or disable the USB wifi card though. That's where the crash is happening, it is not stable.

Steve

aaronouthier

@stephenw10

Well, it’s been up for a few hours now. No hiccoughs yet.

I was trying to use an Alfa brand WiFi card with the notorious RTL-8821AU. I knew this device has numerous issues in Monitor mode under Linux, but I’ve never tried using it as an AP there. I was looking for a working WiFi card for the one renter, and was pleasantly surprised to find that particular card “working”. None of my others did, and that one has 2x 6 dbi antennas. Little did I know…

That said, I managed to connect an old Apple Airport Express that I had laying around, in place of the Alfa Card. That particular renter has only 1 device: An Xbox One, which is capable of connecting at 5GHz, so I always connect a 5 GHz only signal. 2.4 GHz airwaves are crowded enough.

On the one hand, I used up the last spot on my managed router. On the other hand, 3 of the ports are connected to PoE cameras with separate injectors. I can definitely streamline that with a PoE switch later on. Free up 2 ports on my main switch, and eliminate 2 blocky wall-plugs. Win-win!

Thanks again StevenW10!

rcoleman-netgate

@aaronouthier said in Need some help. Random pfSense crashes.:

I was trying to use an Alfa brand WiFi card with the notorious RTL-8821AU. I knew this device has numerous issues in Monitor mode under Linux, but I’ve never tried using it as an AP there. I was looking for a working WiFi card for the one renter, and was pleasantly surprised to find that particular card “working”. None of my others did, and that one has 2x 6 dbi antennas. Little did I know…

I would steer clear of WiFi in BSD.

aaronouthier

@rcoleman-netgate

Duely-noted.