WebGUI and SSH not functional
-
Greetings,
I had the firewall lock up a few days ago and I couldn't SSH in, couldn't get into the WebConfigurator and it was running headless, so I hard rebooted it. Seemed to come back up fine and routing, firewall and other functions worked (DHCP/DHCPv6/etc.). Afterwards though I cannot access the WebGUI or SSH, at all - just times out.
I've tired disabling pfctl, loading the debug ruleset, even loading a saved config from the backups after hooking up a monitor and keyboard. I've tried graceful reboots and a variety of things…
I'm running 2.4.2 currently. I've tried restarting WebConfigurator, restarting PHP-FHP... when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:
nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)
Any ideas? I'd prefer to not delete and redo as it's got a ton of work on the IPv6-PD and the reservations. haha
Thanks!
-
Hi,
@McFly80:… when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:
nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)
Any ideas?Well : what about this one : no more memory !?
Can you develop that one ? Like how much installed ? What other memory eaters (also called packages), etc.Btw : No GUI (we know why) and no SSH (the ssh also abandoned, check log for reason) so you are using the console access. Run "top" to see more info.
-
I totally agree it's "possible" but it's got 8GB of memory on a Core i5… I've got 5450MB free currently on top.
Even for CPU use, when I run top, it's the highest user of resources. Swap has 16GB and 16GB free. :)
-
Great, no apparent memory issue.
Next focus :
Who are the zombies on your system - I've none. Dead nginx instances ?
Easy to check also : stop all processes that are activated by packages. Also : you have only access to the console so it's more a manual operation to "stop" package so the won't restart on reboot. Visit /usr/local/etc/rc.d and do some clean up (copy the related script files out of the way, for example, move them to /root/) and restart - see if the GUI comes up now.My "top" command :
Mem: 34M Active, 280M Inact, 340M Wired, 184M Buf, 1288M Free
(I have pfSense running on 4Gb)
Your "Buf" size is 3 times smaller then mine ? (I don't know what "Buf" really is, except that it is reserved work kernel space for communication)Btw : this is what I should do with my setup - not some sort of "you should do this and all will be fine".
-
Hi there,
Checked it out - they're all bandwidthd processes that are zombies… 8 of 'em.
I'll have to check the startup and logging, kinda slow.
Thanks!
-
…. all bandwidthd processes ...
I'll bet that bandwidthd as allocating all internal "Buf" (limited !) memory. With a final result that the GUI web server (nginx) would (re) start anymore.
edit : have a look at this https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
-
Well, I've tried these buffer changes… no luck.
Why can't I ssh in either? Why would a lockup break SSH also? It's like it reordered the firewall rules and locked me out.
I'm trying to manually remove rules on this now... but I'll tell you - I didn't go in and heavily edit things on the install - so if running a few downloads of Linux ISO torrents does this - I'm at a loss how people don't see some crazy issues in production environments. lol
-
What about a simple console access - and goto default.
I'll bet all will be fine and up afterwards.Then add your settings, rules, etc, and make a pause between each step - test.
As soon as something breaks you will know precisely what not to do, and you have the console to go one step back.
-
"It's like it reordered the firewall rules and locked me out."
There is a specific rule to prevent that - the antilockout rule that allow the port the gui listens on and the ssh port.. Did you disable this rule?
Did you create a rule in floating that happens before interface rules that overrode the antilockout rule? Are you coming from a different network and not the lan network that doesn't have the antilock out rule?
