Can't access the GUI after a reboot

[bazzacad]

I was logged into my XG-7100 1U (22.05-RELEASE) last night & I notice the CPU usage was high so I rebooted it. After the reboot, I could no longer log into the GUI. I get a time-out error. The firewall seems to be running fine, just not the GUI. I can SSH into the box & have physical access. I've tried most of the suggestions from here:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/connect-to-gui.html
and
https://docs.netgate.com/pfsense/en/latest/troubleshooting/locked-out.html
But still no luck. I tried the following:
Restarted the webConfigurator & PHP-FPM from the council
HTTP & HTTPS
Added the EasyRule
SSH Tunneling with Putty
When I run sockstat | grep nginx I don't get any results back.

Any suggestions on how I can gain access to the GUI again?

[stephenw10]

After you have attempted to restart the webconfgurator from the console menu chech the system log to see what errors are shown: cat /var/log/system.log You can also try the nginx log in /var/log/nginx.log

Steve

[bazzacad]

@stephenw10 Thanks for the reply. I was able to get back into the GUI now. But I think I still have issues. From the nginx.log, I found that nginx was failing because /var/run was full.
I moved the large php-cgi.core to the /tmp folder & then rebooted again & I was able to get back into the GUI. But /dev & /var/dhcp/dev are still 100% full after the reboot.

And now the state table size & CPU usage seem really high & no one is logged in right now.

[SteveITS]

@bazzacad said in Can't access the GUI after a reboot:

/dev & /var/dhcp/dev are still 100% full

That's normal: https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-usage.html

re: states, that would imply a lot of connections. Any chance you're having a DoS?

[bazzacad]

@steveits Thanks again, after posting & looking at the states some more, I realized I had a network monitoring service running in the background on my workstation. I've now turned that off & the states are now back to 0%-1%.

Was the large php-cgi.core file and filling up /var/run normial?

[SteveITS]

@bazzacad The .core is a core dump indicating the PHP process crashed. Unless you're planning to investigate further you can delete it. (I would just delete it)

I see /var/run is tmpfs so do you have a 4 MB RAM disk set in System/Advanced/Misc.? That is super small. The 7100 has 8 GB so I'd go with maybe 1 GB for each /tmp and /var if that's what you wanted.

Plus, I don't recall seeing /var/run mounted separately...? I have only installs with RAM disks to look at, at the moment, but /var/run isn't its own mount on either router I checked.

[bazzacad]

@steveits Here is what I have, I think this was the default.

[bazzacad]

OK I've changed them to 1GB. now

[SteveITS]

@bazzacad Off is the default. You might try to find another df -h on the forums. All I was saying was 4 MB seemed odd but I don’t have a comparison.

[stephenw10]

/var/run is always 4MB, that's the expected size. It should only ever have PID files in there so 4MB is plenty.

I'm not sure how the core dump ended up in there, it shouldn't have been put there.

I would disable ram disks again unless you have a good reason to use them.

Steve

[bazzacad]

@stephenw10 Hmmm, I decreased /var to 128MB, and it's already 75% full again. How can I turn off the RAM disk, but still adjust the size of /var?

[SteveITS]

@bazzacad /var is not normally set separately, except apparently /var/run. You can just uncheck the "use RAM disks" option above to not use a RAM disk. Otherwise you have to set a size, and it needs to be big enough for whatever pfSense needs, which will vary by logging and package usage.

Sorry for going down the wrong path on /var/run, I didn't remember that was tmpfs and while I did try to search for a "df -h" output online I didn't find one with that.

[stephenw10]

Yeah if you just disable RAM disks there's no restriction on the size of /var or /tmp.