webgui freezes after upgrading to 2.9
-
all was fine with 2.8.x.
it glitched when going thru the update process and i had to do quite a bit to get it all working properly again.ever since though, there seems to be an issue with suricata. when suricata was enabled, the only way i could get to the webgui was if i ssh'd in and rebooted. then i'd have access for about a minute or two, maybe 3 before it froze. restarting fpm-php and the webgui from ssh didn't do anything. disabling suricata would allow me back in.
my system is based on the following
4 core xeon (one of the low power ones (e3 i believe.. similar to the 7th gen intels)
supermicro x11ssh-f
32gigs of ecc ram
intel i350t2-v2 nic.just to add to it, i don't lose internet connectivity and my speed is normal. i just can't get to the webgui. closing out of the web browser and trying to get back in, it just fails to connect.
as far as troubleshooting goes, i uninstalled suricata, shut down for a couple of minutes, restarted, and all was fine.. i ran the installer (for suricata) and within 5 minutes or so of activating the service, it locked up again.
since i have more than enough ram, i believe i gave php 4Gigs of ram? if i remember correctly? more than enough..
lastly, the only way to get back into the webgui is if i either ssh in and killall -9 suricata (and i have to include the -9, without it, even killall suricata doesn't work).
or, i have to reboot from the command line and quickly web into the webgui and disable the suricata service before it freezes again.Thanks!
-
wow.. no one??
-
How do you have Suricata configured for blocking? Is it set to use Legacy Mode or Inline Mode?
Is your webgui connection to pfSense over the same physical port as the rest of your LAN or do you have a dedicated management interface configured? My initial guess from your description is that Suricata and your NIC don't play well together, but you say you don't lose internet connectivity. That would seem to invalidate my supposition unless you have separate interfaces in play,
-
@bmeeks its in inline mode and i web in thru the lan, not a separate management port.
as far as configuration, it's been this way for years and never an issue. as far as i can tell suricata isn't crashing. so even with the webgui frozen, all is working properly, just the web gui freezes now.
-
@jc1976 said in webgui freezes after upgrading to 2.9:
@bmeeks its in inline mode and i web in thru the lan, not a separate management port.
as far as configuration, it's been this way for years and never an issue. as far as i can tell suricata isn't crashing. so even with the webgui frozen, all is working properly, just the web gui freezes now.
With Inline IPS Mode, your particular PC that you are using to access the GUI is having that traffic dropped by Suricata. That is probably from having the Suricata EVENTS rules all enabled and set to DROP.
Short version is you appear to be experiencing a false positive DROP of traffic from your PC when accessing the webgui of pfSense. That is caused by one or more of the rules you have enabled. Perhaps a rule got a recent automatic update from the Suricata rules update process and now that rule is triggering on a false positive.
Get in via SSH and kill the Suricata processes or run this command from the shell:
/usr/local/etc/rc.d/suricata.sh stopThen go into the GUI and change Suricata to non-blocking mode. You can also, then, examine the ALERTS tab entries to see which rule triggered to drop traffic from your PC. When you identify the rule (or rules) that triggered on your PC's IP address, disable those rules or change their action from DROP to ALERT.
When using Inline IPS Mode blocking, there is no Pass List to act as a safety valve like exists with Legacy Blocking Mode. An overly aggressive rule set with everything set to DROP could result in effectively locking you out from the GUI.
-
ok.. eventhough nothing has changed other than the pfsense version?
before the upgrade suricata was on the latest stable version and i've never had an issue with it.
you're saying that with the pfsense update, now a suricata rule is blocking the webgui?
-
@jc1976 said in webgui freezes after upgrading to 2.9:
ok.. eventhough nothing has changed other than the pfsense version?
before the upgrade suricata was on the latest stable version and i've never had an issue with it.
you're saying that with the pfsense update, now a suricata rule is blocking the webgui?
Do you know that Suricata automatically downloads new updated rules from the sources you have enabled? This and the update interval are configured on the GLOBAL SETTINGS tab. I suspect you have automatic updates enabled. That means on any given day or week (depending on the chosen rule update interval) things can change in the Suricata enforcement realm. That includes new or updated rules that now false-positive trigger where last week they did not.
So, from your description of the problem, I doubt it has anything to do with pfSense itself and is instead the result of a recent Suricata rules update that may have by happenstance coincided with your recent pfSense update.
No matter that relationship, though, your first task is to get to the Suricata GUI and carefully examine all the entries shown on the ALERTS tab. See which rule or rules recently triggered a DROP on the IP address of the PC you are trying to access the GUI from. That rule (or rules) is the problem and not any update to pfSense itself.
I don't know your experience level with an IPS/IDS, but they are not "install and forget about it" systems. They require constant vigilance by the admin to both keep them relevant with rules updates and weed out potential false positives. And anytime something that "was working" and suddenly "is not working" when an IPS/IDS is installed, you should immediately consider the IPS/IDS the culprit and examine the alert logs to see what it is doing. Rules need periodic review to ensure they are adequate for the current threat environment.
I am the creator of the Suricata package for pfSense and I maintained it for many years. Last year I retired from maintenance of the Suricata and Snort packages on pfSense. I have not run nor tested Suricata on the 2.9.0 CE release (nor even on pfSense Plus). So far as I know it continues to function well, but I have no verification other than the several thousand pfSense users. I assume this forum would be filled with posts if Suricata was not working for those users on 2.9.0 CE.
For what it's worth, Netgate has apparently chosen to natively support the new Snort3 system in pfSense Plus as part of the Nexus package. That setup, as of now, is not available in CE. So, I don't know what the plans are for IDS/IPS in CE going forward.
-
B bmeeks referenced this topic
-
You may have hit this issue at upgrade when it re-installs Suricata: https://redmine.pfsense.org/issues/16808
-
It's possibly that bug, but the default EVENTS rules distributed with Suricata are all set to ALERT for the action and not DROP. Since the OP states they are using Inline IPS Mode, only rules specifically set to DROP would be causing blocking.
No rules from any ruleset source are distributed with DROP as the action. They are all distributed with ALERT as the default action. It takes specific action by the admin to change rule actions to DROP. That is done by either using the SID MGMT tab feature, manually changing the action on the RULES or ALERTS tabs, or selecting an IPS Policy (but this is only available for Snort rules when using those in Suricata).
-
Not to mention #16808 is hardly a bug.
-
@tinfoilmatt said in webgui freezes after upgrading to 2.9:
Not to mention #16808 is hardly a bug.
Yeah, the issue described in that bug report is a side effect of an attempt to be sure any new EVENTS rules distributed with periodic Suricata source code updates get added to pfSense. In hindsight I probably could have put some additional intelligence in that section of the post-installation script that added only rules that were missing from BOTH the "enabled" and "manually disabled" rules. The various EVENTS rules are distributed by default in the Suricata source code and get reinstalled with each update or installation of the binary package.
But in almost all cases those EVENTS rules should never be enabled for blocking (or dropping). They are designed to be "information only" type rules and don't necessarily indicate malicious traffic.
-
i've been using suricata for years. i know it's not set it and forget it, but if i ever had an issue in the past, it was always blocking outside webpages, never the pfsense webgui.
i uninstalled it, shut everything down for a few minutes, restarted the firewall and let it all idle for a while without any issue. then i ran the suricata installer.. all seemed fine for several minutes, and then it froze again.
-
@jc1976 said in webgui freezes after upgrading to 2.9:
i've been using suricata for years. i know it's not set it and forget it, but if i ever had an issue in the past, it was always blocking outside webpages, never the pfsense webgui.
i uninstalled it, shut everything down for a few minutes, restarted the firewall and let it all idle for a while without any issue. then i ran the suricata installer.. all seemed fine for several minutes, and then it froze again.
I've told you the first step yet you have not done that yet. You need to examine the alert log accessible via the ALERTS tab and see which rule is actually causing the dropped traffic. Post back a screenshot of the ALERTS tab.
When you say "Internet connectivity is unaffected", do you mean that from the same machine you can browse the web even when access attempts to the pfSense web GUI fail?
Reinstalling and rebooting are not the path to the solution. You need to see what is actually happening. Is it a blocking (false positive) rule or not?
-
I updated to the latest stable version of suricata whenever it came out and had no issues with it. this probably only occurred after updating pfsnese to 2.9.
-
@jc1976 said in webgui freezes after upgrading to 2.9:
I updated to the latest stable version of suricata whenever it came out and had no issues with it. this probably only occurred after updating pfsnese to 2.9.
Don't get distracted by the past and any updates. First determine if a rule is triggering and causing the problem. If zero rules are triggering dropped traffic from your PC, then we can move on to perhaps another possible cause. But until you can absolutely rule out a triggering DROP rule as the source of the lockout, then any other steps are wasted effort.
-
how will i be able to tell if a rule is causing the problem? if the webgui locks up and i lose access to it, what should i do and look for at that point?
-
also, isn't there a way to add an ip address or url that is allowed to bypass all scanning?
i tried and iptv service a little while back and suricata was blocking it because the feeds were all unencrypted. i remember being able to put a url in somewhere, basically whitelist the provider so that suricata would ignore anything coming from that address..
-
@jc1976 said in webgui freezes after upgrading to 2.9:
how will i be able to tell if a rule is causing the problem? if the webgui locks up and i lose access to it, what should i do and look for at that point?
Login to the firewall with an SSH session. Stop or kill the Suricata processes. This command executed from a shell prompt should stop all running Suricata processes:
/usr/local/etc/rc.d/suricata.sh stopIf that does not work, then use the
kill -9method you mentioned in an earlier post. That should let you log into the GUI at that point according to your previous post.Once in the GUI, then go to the Suricata ALERTS tab and then examine all the entries shown on that tab. Those will be the rules that triggered. Unless you wiped the log, it should have a good bit of history in it going back several days or weeks. The default sort is "most recent alerts first". Find any alerts containing the IP address of your PC. See if the alert time shown for the logged event matches the time interval when you were locked out of the GUI. Any match, and you have found the culprit.
You can post a screen capture of the ALERTS tab along with your PC's IP address (I assume it's a private RFC1918 address). From the capture I may be able to identify the faulting rule if that's the cause. Just go to SERVICES ... SURICATA in the pfSense menu and then click the ALERTS tab to see all the triggered rule alerts. You can choose an interface from the drop-down selector on that tab.
-
@jc1976 said in webgui freezes after upgrading to 2.9:
also, isn't there a way to add an ip address or url that is allowed to bypass all scanning?
i tried and iptv service a little while back and suricata was blocking it because the feeds were all unencrypted. i remember being able to put a url in somewhere, basically whitelist the provider so that suricata would ignore anything coming from that address..
When using Inline IPS Mode, you can create a custom PASS rule with a specific IP address. PASS rules are evaluated first and any match bypasses all other rule comparisons. Not sure what you mean by URL, though. PASS rules only work with IP addresses.
-
@bmeeks i would run top from my ssh session and suircata would be at the top of the list, monopolizing 1 core at 97%..
dunno if that means anything to you..
Privacy Policy · Cookie Policy