Suricata 5.0.2_2 update breaks routing

pwnell

I just updated my Suricata package on 2.4.5 to 5.0.2_2 and before it was working, after the update no traffic flows through the firewall any longer. Suricata is in inline IPS mode on LAN interface. Disabling suricata causes traffic to flow again, enabling it causes NO traffic to traverse the LAN interface.

What changed?

SteveITS

Don't have an answer for you but does it work in Legacy mode instead of inline?

pwnell

Did not test yet, this is a production system and it worked for more than a year fine in inline IPS mode before this last update.

bmeeks

@pwnell said in Suricata 5.0.2_2 update breaks routing:

Did not test yet, this is a production system and it worked for more than a year fine in inline IPS mode before this last update.

More information from you would be useful in diagnosing this.

What possibly relevant messages are logged in the pfSense system log?
Have you checked the suricata.log file for the interface on the LOGS VIEW tab to see if anything is showing in there?
Did you possibly update the Suricata package before you updated to pfSense-2.4.5? Maybe not, but it's not 100% clear from your post. Updating Suricata first (before pfSense) will cause many issues.

pwnell

@bmeeks Sure I will provide what I can:

I see Suricata start, and no abnormal or error messages in pfSense log.
Nothing weird in suricata.log - some errors due to malformed rules but that has been there since the beginning and never interfered.
I updated to pfSense 2.4.5 when it was released (weeks ago), then only this morning did I update Suricata.

I restarted Suricata and apart for a glitch in network connectivity for about 1 minute, it has recovered and so far it is working. I will be monitoring to see if this changes. My other pfSense boxes on 2.4.5 and the new Suricata works fine.

bmeeks

@pwnell said in Suricata 5.0.2_2 update breaks routing:

@bmeeks Sure I will provide what I can:

I see Suricata start, and no abnormal or error messages in pfSense log.

Nothing weird in suricata.log - some errors due to malformed rules but that has been there since the beginning and never interfered.

I updated to pfSense 2.4.5 when it was released (weeks ago), then only this morning did I update Suricata.

I restarted Suricata and apart for a glitch in network connectivity for about 1 minute, it has recovered and so far it is working. I will be monitoring to see if this changes. My other pfSense boxes on 2.4.5 and the new Suricata works fine.

Ah, okay. You are likely seeing some flavor of the pfctl bug that is discussed at length here: https://forum.netgate.com/topic/149595/2-4-5-a-20200110-1421-and-earlier-high-cpu-usage-from-pfctl/65. This bug crops anytime the pf filter is reloaded. Suricata starting can trigger that. So can DNSBL updates and other list maintenance activities from pfBlockerNG if you have either of those installed. The IPv6 bogons table also causes the issue if that blocking mode is enabled on an interface.

So Suricata is more likely a sort of victim and not the true root cause. The good news is the pfSense team has identified the cause. Now they are working on the best fix. I would expect a pfSense update in the near future to take care of this.

One thing you might want to do is go to the GLOBAL SETTINGS tab and enable the option for "Live Rule Swaps" (it's worded close to that way, don't have a Suricata GUI in front of me at the moment). With that option enabled, Suricata is not physically restarted at the end of each rules update. Otherwise, each time Suricata updates your rules it will physically restart the binary and that will trigger the network interruption again (just like the initial start-up).

pwnell

Thanks for the update. Live rule swap is already on. I do have pfBlocker and Suricata so that might very well be it.