pfSense constantly crashing

deanfourie

@gertjan I get this multiple time, like almost 50 times. I am also having huge issues with ntopng plugins crashing I should add.

94319 root         20    0   304M   250M uwait    2   0:00   0.10% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i em0 -i ovpnc1 -i ue0 --dns-mode 0 --local-networks 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8{ntopng}
    0 root        -16    -     0B   704K swapin   0   0:16   0.00% [kernel{swapper}]
94319 root         20    0   304M   250M uwait    0   0:03   0.00% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i em0 -i ovpnc1 -i ue0 --dns-mode 0 --local-networks 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8{ntopng}
94319 root         20    0   304M   250M uwait    1   0:02   0.00% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i em0 -i ovpnc1 -i ue0 --dns-mode 0 --local-networks 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8{ntopng}
94319 root         23    0   304M   250M uwait    1   0:01   0.00% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i em0 -i ovpnc1 -i ue0 --dns-mode 0 --local-networks 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8{ntopng}
94319 root         20    0   304M   250M nanslp   3   0:01   0.00% /usr/local/bin/ntopng -d /var/db/ntopng -G /var/run/ntopng.pid -s -e -w 0 -W 3000 -i em0 -i ovpnc1 -i ue0 --dns-mode 0 --local-networks 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8{ntopng}

deanfourie

I cant figure out why everything keeps crashing. I'm starting to lean toward a kernel panic. Only thing that makes sense at this point.

stephenw10

I assume you never see a crash report after it reboots?

Can you log the console output?

Nothing shown at all looks more like a hardware issue. ntopng trying to start twice is common and doesn't usually cause a problem. What do you mean by 'ntopng plugins'?

If I had to guess what the issue is I'd say it's because you're running a Realtek USB NIC.

Steve

Gertjan

@stephenw10 said in pfSense constantly crashing:

Realtek USB NIC.

Nice catch !

deanfourie

@stephenw10 Correct, there appears to be no crash report.

@Gertjan and yes correct, a Realtek USB NIC. Could this really cause something so catastrophic like a kernel panic?

deanfourie

This morning again, it has crashed. I have internet still but cannot reach any of the web services.

Not sure at this stage if its a kernel panic or some kind on internal ddos on the web services.

Can confirm there is a active listening socket on port 80, just cannot connect.

Starting Nmap 7.92 ( https://nmap.org ) at 2021-10-13 09:10 New Zealand Daylight Time

NSE: Loaded 155 scripts for scanning.

NSE: Script Pre-scanning.

Initiating NSE at 09:10

Completed NSE at 09:10, 0.00s elapsed

Initiating NSE at 09:10

Completed NSE at 09:10, 0.00s elapsed

Initiating NSE at 09:10

Completed NSE at 09:10, 0.00s elapsed

Initiating ARP Ping Scan at 09:10

Scanning 172.16.101.1 [1 port]

Completed ARP Ping Scan at 09:10, 0.06s elapsed (1 total hosts)

Initiating Parallel DNS resolution of 1 host. at 09:10

Completed Parallel DNS resolution of 1 host. at 09:10, 0.05s elapsed

Initiating SYN Stealth Scan at 09:10

Scanning 172.16.101.1 [1000 ports]

Discovered open port 80/tcp on 172.16.101.1

Discovered open port 443/tcp on 172.16.101.1

Discovered open port 53/tcp on 172.16.101.1

Discovered open port 3000/tcp on 172.16.101.1

Completed SYN Stealth Scan at 09:10, 4.95s elapsed (1000 total ports)

Initiating Service scan at 09:10

stephenw10

If you have internet then pfSense has not crashed.

If there was a kernel panic it would almost always create a crash report. The only time it doesn't is when the panic is caused by a driver failure of some sort so it can't write out the report.
Usually if there's no crash report it's because the device hard rebooted which is probably hardware failure.
The only other thing is if you have RAM disks enabled that are lost on reboot it will lose and crash data.

Have you actually seen a kernel panic?

If you can't connect to the webgui try resetting php and the webconfigurator at the console, options 16 then 11 in the menu.

It looks like you don't have SSH enabled so that's the first thing I would so. Then you can access it that way.

Steve

deanfourie

@stephenw10 Ok, I also have DHCP discovers sent out with no reply offers.

deanfourie

@stephenw10 What logs should I pull from pfSense?

stephenw10

The main system log would contain any errors if anything is logged.

Gertjan

@deanfourie said in pfSense constantly crashing:

Could this really cause something so catastrophic like a kernel panic?

I adivse you to make this a priority task :
Have a look at what's been said about 'realtek' for 'serious' applications like routers.
I've no solid proof, but their is this common knowledge that you should stay away from this brand, just to be on the safe side.
Realtek over USB ? That's like playing russish roulette with 5 bullets in the 6 chambres, instead of one bullet.
Ethernet over USB : that's just a big nono in your situation. If it works, ok, good for you. But that kind of hardware should be removed if you suspect issues.

So : first go native, classic bare bone : a device with two (or more) real NIC's. test drive that. If still issues, then you know the device (drive or motherboard or power) has an issue.
Don't do tests with realtek or USB NICs nearby.