webgui freezes after upgrading to 2.9
-
@jc1976 said in webgui freezes after upgrading to 2.9:
@bmeeks its in inline mode and i web in thru the lan, not a separate management port.
as far as configuration, it's been this way for years and never an issue. as far as i can tell suricata isn't crashing. so even with the webgui frozen, all is working properly, just the web gui freezes now.
With Inline IPS Mode, your particular PC that you are using to access the GUI is having that traffic dropped by Suricata. That is probably from having the Suricata EVENTS rules all enabled and set to DROP.
Short version is you appear to be experiencing a false positive DROP of traffic from your PC when accessing the webgui of pfSense. That is caused by one or more of the rules you have enabled. Perhaps a rule got a recent automatic update from the Suricata rules update process and now that rule is triggering on a false positive.
Get in via SSH and kill the Suricata processes or run this command from the shell:
/usr/local/etc/rc.d/suricata.sh stopThen go into the GUI and change Suricata to non-blocking mode. You can also, then, examine the ALERTS tab entries to see which rule triggered to drop traffic from your PC. When you identify the rule (or rules) that triggered on your PC's IP address, disable those rules or change their action from DROP to ALERT.
When using Inline IPS Mode blocking, there is no Pass List to act as a safety valve like exists with Legacy Blocking Mode. An overly aggressive rule set with everything set to DROP could result in effectively locking you out from the GUI.
-
ok.. eventhough nothing has changed other than the pfsense version?
before the upgrade suricata was on the latest stable version and i've never had an issue with it.
you're saying that with the pfsense update, now a suricata rule is blocking the webgui?
-
@jc1976 said in webgui freezes after upgrading to 2.9:
ok.. eventhough nothing has changed other than the pfsense version?
before the upgrade suricata was on the latest stable version and i've never had an issue with it.
you're saying that with the pfsense update, now a suricata rule is blocking the webgui?
Do you know that Suricata automatically downloads new updated rules from the sources you have enabled? This and the update interval are configured on the GLOBAL SETTINGS tab. I suspect you have automatic updates enabled. That means on any given day or week (depending on the chosen rule update interval) things can change in the Suricata enforcement realm. That includes new or updated rules that now false-positive trigger where last week they did not.
So, from your description of the problem, I doubt it has anything to do with pfSense itself and is instead the result of a recent Suricata rules update that may have by happenstance coincided with your recent pfSense update.
No matter that relationship, though, your first task is to get to the Suricata GUI and carefully examine all the entries shown on the ALERTS tab. See which rule or rules recently triggered a DROP on the IP address of the PC you are trying to access the GUI from. That rule (or rules) is the problem and not any update to pfSense itself.
I don't know your experience level with an IPS/IDS, but they are not "install and forget about it" systems. They require constant vigilance by the admin to both keep them relevant with rules updates and weed out potential false positives. And anytime something that "was working" and suddenly "is not working" when an IPS/IDS is installed, you should immediately consider the IPS/IDS the culprit and examine the alert logs to see what it is doing. Rules need periodic review to ensure they are adequate for the current threat environment.
I am the creator of the Suricata package for pfSense and I maintained it for many years. Last year I retired from maintenance of the Suricata and Snort packages on pfSense. I have not run nor tested Suricata on the 2.9.0 CE release (nor even on pfSense Plus). So far as I know it continues to function well, but I have no verification other than the several thousand pfSense users. I assume this forum would be filled with posts if Suricata was not working for those users on 2.9.0 CE.
For what it's worth, Netgate has apparently chosen to natively support the new Snort3 system in pfSense Plus as part of the Nexus package. That setup, as of now, is not available in CE. So, I don't know what the plans are for IDS/IPS in CE going forward.
-
B bmeeks referenced this topic
-
You may have hit this issue at upgrade when it re-installs Suricata: https://redmine.pfsense.org/issues/16808
-
It's possibly that bug, but the default EVENTS rules distributed with Suricata are all set to ALERT for the action and not DROP. Since the OP states they are using Inline IPS Mode, only rules specifically set to DROP would be causing blocking.
No rules from any ruleset source are distributed with DROP as the action. They are all distributed with ALERT as the default action. It takes specific action by the admin to change rule actions to DROP. That is done by either using the SID MGMT tab feature, manually changing the action on the RULES or ALERTS tabs, or selecting an IPS Policy (but this is only available for Snort rules when using those in Suricata).
-
Not to mention #16808 is hardly a bug.
-
@tinfoilmatt said in webgui freezes after upgrading to 2.9:
Not to mention #16808 is hardly a bug.
Yeah, the issue described in that bug report is a side effect of an attempt to be sure any new EVENTS rules distributed with periodic Suricata source code updates get added to pfSense. In hindsight I probably could have put some additional intelligence in that section of the post-installation script that added only rules that were missing from BOTH the "enabled" and "manually disabled" rules. The various EVENTS rules are distributed by default in the Suricata source code and get reinstalled with each update or installation of the binary package.
But in almost all cases those EVENTS rules should never be enabled for blocking (or dropping). They are designed to be "information only" type rules and don't necessarily indicate malicious traffic.
-
i've been using suricata for years. i know it's not set it and forget it, but if i ever had an issue in the past, it was always blocking outside webpages, never the pfsense webgui.
i uninstalled it, shut everything down for a few minutes, restarted the firewall and let it all idle for a while without any issue. then i ran the suricata installer.. all seemed fine for several minutes, and then it froze again.
-
@jc1976 said in webgui freezes after upgrading to 2.9:
i've been using suricata for years. i know it's not set it and forget it, but if i ever had an issue in the past, it was always blocking outside webpages, never the pfsense webgui.
i uninstalled it, shut everything down for a few minutes, restarted the firewall and let it all idle for a while without any issue. then i ran the suricata installer.. all seemed fine for several minutes, and then it froze again.
I've told you the first step yet you have not done that yet. You need to examine the alert log accessible via the ALERTS tab and see which rule is actually causing the dropped traffic. Post back a screenshot of the ALERTS tab.
When you say "Internet connectivity is unaffected", do you mean that from the same machine you can browse the web even when access attempts to the pfSense web GUI fail?
Reinstalling and rebooting are not the path to the solution. You need to see what is actually happening. Is it a blocking (false positive) rule or not?
-
I updated to the latest stable version of suricata whenever it came out and had no issues with it. this probably only occurred after updating pfsnese to 2.9.
-
@jc1976 said in webgui freezes after upgrading to 2.9:
I updated to the latest stable version of suricata whenever it came out and had no issues with it. this probably only occurred after updating pfsnese to 2.9.
Don't get distracted by the past and any updates. First determine if a rule is triggering and causing the problem. If zero rules are triggering dropped traffic from your PC, then we can move on to perhaps another possible cause. But until you can absolutely rule out a triggering DROP rule as the source of the lockout, then any other steps are wasted effort.
-
how will i be able to tell if a rule is causing the problem? if the webgui locks up and i lose access to it, what should i do and look for at that point?
-
also, isn't there a way to add an ip address or url that is allowed to bypass all scanning?
i tried and iptv service a little while back and suricata was blocking it because the feeds were all unencrypted. i remember being able to put a url in somewhere, basically whitelist the provider so that suricata would ignore anything coming from that address..
-
@jc1976 said in webgui freezes after upgrading to 2.9:
how will i be able to tell if a rule is causing the problem? if the webgui locks up and i lose access to it, what should i do and look for at that point?
Login to the firewall with an SSH session. Stop or kill the Suricata processes. This command executed from a shell prompt should stop all running Suricata processes:
/usr/local/etc/rc.d/suricata.sh stopIf that does not work, then use the
kill -9method you mentioned in an earlier post. That should let you log into the GUI at that point according to your previous post.Once in the GUI, then go to the Suricata ALERTS tab and then examine all the entries shown on that tab. Those will be the rules that triggered. Unless you wiped the log, it should have a good bit of history in it going back several days or weeks. The default sort is "most recent alerts first". Find any alerts containing the IP address of your PC. See if the alert time shown for the logged event matches the time interval when you were locked out of the GUI. Any match, and you have found the culprit.
You can post a screen capture of the ALERTS tab along with your PC's IP address (I assume it's a private RFC1918 address). From the capture I may be able to identify the faulting rule if that's the cause. Just go to SERVICES ... SURICATA in the pfSense menu and then click the ALERTS tab to see all the triggered rule alerts. You can choose an interface from the drop-down selector on that tab.
-
@jc1976 said in webgui freezes after upgrading to 2.9:
also, isn't there a way to add an ip address or url that is allowed to bypass all scanning?
i tried and iptv service a little while back and suricata was blocking it because the feeds were all unencrypted. i remember being able to put a url in somewhere, basically whitelist the provider so that suricata would ignore anything coming from that address..
When using Inline IPS Mode, you can create a custom PASS rule with a specific IP address. PASS rules are evaluated first and any match bypasses all other rule comparisons. Not sure what you mean by URL, though. PASS rules only work with IP addresses.
-
@bmeeks i would run top from my ssh session and suircata would be at the top of the list, monopolizing 1 core at 97%..
dunno if that means anything to you..
-
@jc1976 said in webgui freezes after upgrading to 2.9:
@bmeeks i would run top from my ssh session and suircata would be at the top of the list, monopolizing 1 core at 97%..
dunno if that means anything to you..
Yeah, that is unusual. But that should also impact Internet connectivity at the same time. You said in your first post that "I don't lose internet connectivity". That would not mesh with Suricata locking up the firewall and becoming unresponsive.
Perhaps I am misunderstanding your description of the issue. If Suricata is crashing or locking up, I would expect that to potentially freeze up all traffic flowing over the interface including "internet connectivity". The fact you said "I don't lose internet connectivity" made me think other devices could continue to work just fine through the firewall. Is that not the case?
For example, while you are locked out of the pfSense GUI, can you still browse to other sites out on the Internet without any problems from the same PC?
-
the firewall is still working, just the webgui is freezing. pfblocker is still going,, i can run a speedtest and i still get what i'm allotted from my cable provider..
just the webgui.. i can hit the home button and it takes me to my default homepage.. if i try to go back to the pfsense webgui, nothing happens and eventually it just gives me a timeout. but the firewall itself is still going..
-
@jc1976 said in webgui freezes after upgrading to 2.9:
the firewall is still working, just the webgui is freezing. pfblocker is still going,, i can run a speedtest and i still get what i'm allotted from my cable provider..
just the webgui.. i can hit the home button and it takes me to my default homepage.. if i try to go back to the pfsense webgui, nothing happens and eventually it just gives me a timeout. but the firewall itself is still going..
Is there anything in the pfSense system log during this interval?
I'm struggling to tie the loss of the GUI to Suricata. Maybe a wild guess would be something in the new FreeBSD kernel code associated with 2.9.0 CE that is at odds with your particular NIC card. That's only a wild supposition, though.
Try changing Suricata to non-blocking mode by going to the Suricata INTERFACE SETTINGS tab for your LAN and uncheck the BLOCK OFFENDERS checkbox to disable blocking. Save the change and then start Suricata and see what happens. At that point Suricata will be in alerts-only mode and won't block. That will also stop it from trying to initialize the netmap driver. There is an outside chance (part of my wild guess above) that trying to bring up the netmap driver is causing a problem. But again, I really would expect that sort of issue to kill traffic through the LAN interface.
Other than your client PC getting blocked or having its traffic dropped by Suricata, the only other thing that could be happening is the
nginxweb server of the GUI getting locked up or killed by Suricata running. But I have never had that issue reported before.Are all clients locked out of the pfSense GUI when this event occurs? For example, can you login to the GUI from another PC or perhaps your phone (on WiFi that is the same network as your LAN)?
-
well whaddya know....??
cleared out all my alert logs, etc.. and tried logging in, which i couldn't.. killed suricata and logged in, and yup!, it was a rule.. disabled it and all is working as it should.
Thanks bmeeks!!
Privacy Policy · Cookie Policy