WebGUI and SSH not functional

McFly80 · Dec 27, 2017, 5:37 PM

Greetings,

I had the firewall lock up a few days ago and I couldn't SSH in, couldn't get into the WebConfigurator and it was running headless, so I hard rebooted it. Seemed to come back up fine and routing, firewall and other functions worked (DHCP/DHCPv6/etc.). Afterwards though I cannot access the WebGUI or SSH, at all - just times out.

I've tired disabling pfctl, loading the debug ruleset, even loading a saved config from the backups after hooking up a monitor and keyboard. I've tried graceful reboots and a variety of things…

I'm running 2.4.2 currently. I've tried restarting WebConfigurator, restarting PHP-FHP... when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:

nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)

Any ideas? I'd prefer to not delete and redo as it's got a ton of work on the IPv6-PD and the reservations. haha

Thanks!

Gertjan · Dec 27, 2017, 5:47 PM

Hi,
@McFly80:

… when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:
nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)
Any ideas?

Well : what about this one : no more memory !?
Can you develop that one ? Like how much installed ? What other memory eaters (also called packages), etc.

Btw : No GUI (we know why) and no SSH (the ssh also abandoned, check log for reason) so you are using the console access. Run "top" to see more info.

McFly80 · Dec 27, 2017, 8:01 PM

I totally agree it's "possible" but it's got 8GB of memory on a Core i5… I've got 5450MB free currently on top.

Even for CPU use, when I run top, it's the highest user of resources. Swap has 16GB and 16GB free. :)

Gertjan · Dec 27, 2017, 8:47 PM

Great, no apparent memory issue.

Next focus :
Who are the zombies on your system - I've none. Dead nginx instances ?
Easy to check also : stop all processes that are activated by packages. Also : you have only access to the console so it's more a manual operation to "stop" package so the won't restart on reboot. Visit /usr/local/etc/rc.d and do some clean up (copy the related script files out of the way, for example, move them to /root/) and restart - see if the GUI comes up now.

My "top" command :

Mem: 34M Active, 280M Inact, 340M Wired, 184M Buf, 1288M Free

(I have pfSense running on 4Gb)
Your "Buf" size is 3 times smaller then mine ? (I don't know what "Buf" really is, except that it is reserved work kernel space for communication)

Btw : this is what I should do with my setup - not some sort of "you should do this and all will be fine".

McFly80 · Dec 27, 2017, 9:34 PM

Hi there,

Checked it out - they're all bandwidthd processes that are zombies… 8 of 'em.

I'll have to check the startup and logging, kinda slow.

Thanks!

Gertjan · Dec 28, 2017, 2:28 PM

@McFly80:

…. all bandwidthd processes ...

I'll bet that bandwidthd as allocating all internal "Buf" (limited !) memory. With a final result that the GUI web server (nginx) would (re) start anymore.

edit : have a look at this https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

McFly80 · Jan 2, 2018, 8:11 AM

Well, I've tried these buffer changes… no luck.

Why can't I ssh in either? Why would a lockup break SSH also? It's like it reordered the firewall rules and locked me out.

I'm trying to manually remove rules on this now... but I'll tell you - I didn't go in and heavily edit things on the install - so if running a few downloads of Linux ISO torrents does this - I'm at a loss how people don't see some crazy issues in production environments. lol

Gertjan · Jan 2, 2018, 10:37 AM

What about a simple console access - and goto default.
I'll bet all will be fine and up afterwards.

Then add your settings, rules, etc, and make a pause between each step - test.

As soon as something breaks you will know precisely what not to do, and you have the console to go one step back.

johnpoz · Jan 2, 2018, 10:53 AM

"It's like it reordered the firewall rules and locked me out."

There is a specific rule to prevent that - the antilockout rule that allow the port the gui listens on and the ssh port.. Did you disable this rule?

Did you create a rule in floating that happens before interface rules that overrode the antilockout rule? Are you coming from a different network and not the lan network that doesn't have the antilock out rule?