WebGUI and SSH not functional



  • Greetings,

    I had the firewall lock up a few days ago and I couldn't SSH in, couldn't get into the WebConfigurator and it was running headless, so I hard rebooted it. Seemed to come back up fine and routing, firewall and other functions worked (DHCP/DHCPv6/etc.). Afterwards though I cannot access the WebGUI or SSH, at all - just times out.

    I've tired disabling pfctl, loading the debug ruleset, even loading a saved config from the backups after hooking up a monitor and keyboard. I've tried graceful reboots and a variety of things…

    I'm running 2.4.2 currently. I've tried restarting WebConfigurator, restarting PHP-FHP... when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:

    nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)

    Any ideas? I'd prefer to not delete and redo as it's got a ton of work on the IPv6-PD and the reservations. haha

    Thanks!



  • Hi,
    @McFly80:

    … when I try to restart WebConfig it gives me an error about nginx being out of buffer space but it continues after reboots:
    nginx: [emerg] socket() 0.0.0.0:443 failed (105: No buffer space available)
    Any ideas?

    Well : what about this one : no more memory !?
    Can you develop that one ? Like how much installed ? What other memory eaters (also called packages), etc.

    Btw : No GUI (we know why) and no SSH (the ssh also abandoned, check log for reason) so you are using the console access. Run "top" to see more info.



  • I totally agree it's "possible" but it's got 8GB of memory on a Core i5… I've got 5450MB free currently on top.

    Even for CPU use, when I run top, it's the highest user of resources. Swap has 16GB and 16GB free. :)




  • Great, no apparent memory issue.

    Next focus :
    Who are the zombies on your system - I've none. Dead nginx instances ?
    Easy to check also : stop all processes that are activated by packages. Also : you have only access to the console so it's more a manual operation to "stop" package so the won't restart on reboot. Visit  /usr/local/etc/rc.d and do some clean up (copy the related script files out of the way, for example, move them to /root/) and restart - see if the GUI comes up now.

    My "top" command :

    Mem: 34M Active, 280M Inact, 340M Wired, 184M Buf, 1288M Free

    (I have pfSense running on 4Gb)
    Your "Buf" size is 3 times smaller then mine ? (I don't know what "Buf" really is, except that it is reserved work kernel space for communication)

    Btw : this is what I should do with my setup - not some sort of "you should do this and all will be fine".



  • Hi there,

    Checked it out - they're all bandwidthd processes that are zombies… 8 of 'em.

    I'll have to check the startup and logging, kinda slow.

    Thanks!



  • @McFly80:

    …. all bandwidthd processes ...

    I'll bet that bandwidthd as allocating all internal "Buf" (limited !) memory. With a final result that the GUI web server (nginx) would (re) start anymore.

    edit : have a look at this https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards



  • Well, I've tried these buffer changes… no luck.

    Why can't I ssh in either? Why would a lockup break SSH also? It's like it reordered the firewall rules and locked me out.

    I'm trying to manually remove rules on this now... but I'll tell you - I didn't go in and heavily edit things on the install - so if running a few downloads of Linux ISO torrents does this - I'm at a loss how people don't see some crazy issues in production environments. lol



  • What about a simple console access - and goto default.
    I'll bet all will be fine and up afterwards.

    Then add your settings, rules, etc, and make a pause between each step - test.

    As soon as something breaks you will know precisely what not to do, and you have the console to go one step back.


  • LAYER 8 Global Moderator

    "It's like it reordered the firewall rules and locked me out."

    There is a specific rule to prevent that - the antilockout rule that allow the port the gui listens on and the ssh port.. Did you disable this rule?

    Did you create a rule in floating that happens before interface rules that overrode the antilockout rule?  Are you coming from a different network and not the lan network that doesn't have the antilock out rule?


Log in to reply