After upgrade to pf+ 23.09 Surricata says it's starting but..

Euman

After upgrade to pf+ 23.09 Surricata says it's starting but doesn't start on any interface.

I upgraded, restarted pf+, tried getting surricata to run w/o success.
removed the pkg, reinstalled the pkg, reboot the netgate 6100 and nothing worked.
I deinstalled the pkg again and installed snort, snort would not start either.

fairly stressed about upgrading my router anymore.

please advise,

kind regard

NogBadTheBad

@Euman Same here, in the logs it mentions removing /var/run/suricataINTERFACE.pid.

Removed the above files and it still fails to start.

Euman

Hello Andy,

/var/run/suricataXXX.pid

there are NO (0) files or directories that have the name surricata in /var/run

Bob.Dig

@Euman For me it wasn't running after the upgrade but I could start it manually without problem.
After the next reboot it was starting automatically, so no problem here.

NogBadTheBad

From my logs:-

Nov 6 18:48:32 suricata 2303 [102133] <Error> -- pid file '/var/run/suricata_igb02860.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_igb02860.pid. Aborting!
Nov 6 18:48:31 suricata 2303 [102133] <Notice> -- This is Suricata version 7.0.0 RELEASE running in SYSTEM mode
Nov 6 18:48:31 SuricataStartup 2231 Suricata START for LAN(2860_igb0)...
Nov 6 18:48:30 suricata 97432 [102133] <Error> -- pid file '/var/run/suricata_pppoe05774.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_pppoe05774.pid. Aborting!
Nov 6 18:48:29 suricata 97432 [102133] <Notice> -- This is Suricata version 7.0.0 RELEASE running in SYSTEM mode
Nov 6 18:48:29 SuricataStartup 97204 Suricata START for WAN(5774_pppoe0)...

Ah its writing the pid files in /var/log/suricata:-

[23.09-RELEASE][admin@pfsense]/var/log/suricata: ls
suricata_igb02860 suricata_pppoe05774 suricata_rules_update.log
[23.09-RELEASE][admin@pfsense]/var/log/suricata

NogBadTheBad

Its doing a core dump:-

[23.09-RELEASE][admin@pfsense]/root: ls -al
total 59036
drwxr-xr-x 5 root wheel 512 Nov 6 19:04 .
drwxr-xr-x 24 root wheel 1024 Nov 6 17:05 ..
drwx------ 2 root wheel 512 Feb 17 2021 .cache
-rw-r--r-- 2 root wheel 1011 Nov 1 21:58 .cshrc
-rw-r--r-- 1 root wheel 0 Nov 6 17:04 .hushlogin
-rw-r--r-- 1 root wheel 68 Nov 1 21:58 .k5login
-rw------- 1 root wheel 82 Oct 13 18:41 .lesshst
-rw-r--r-- 1 root wheel 316 Nov 1 21:58 .login
-rw------- 1 root wheel 1848 Dec 27 2020 .lsof_pfsense
-rw-r--r-- 2 root wheel 1199 Nov 6 17:04 .profile
-rw------- 1 root wheel 1024 Aug 15 08:08 .rnd
-rw------- 1 root wheel 190 Nov 6 19:02 .sh_history
-rw-r--r-- 1 root wheel 2143 Nov 6 17:04 .shrc
drwx------ 2 root wheel 512 May 4 2023 .ssh
-rw-r--r-- 1 root wheel 3410 Nov 6 17:04 .tcshrc
-rw-r--r-- 1 root wheel 0 Mar 10 2023 packetcapture.start
drwxr-xr-x 2 root wheel 512 Sep 13 2021 scripts
-rw------- 1 root wheel 86564864 Nov 6 19:04 suricata.core
[23.09-RELEASE][admin@pfsense]/root:

Euman

@Bob-Dig I tried starting manually, was a NO-GO

Bob.Dig

@Euman Running here in legacy mode on a x86 VM without problems.

NRgia

@Euman I don't have any problem with Suricata, running on a white box with Intel Atom C3558 same as 6100

bmeeks

Are you guys having Suricata 7 startup issues runing with Legacy Blocking Mode or Inline IPS Mode enabled?

I see at least one of you appears to have Suricata enabled on a PPPoE interface. Inline IPS Mode will most definitely not work on that interface. Also looks like for one of you there is more than one Suricata instance enabled on the box (PPPoE on WAN, I assume, and another instance on an internal igb0 interface).

A core dump is obviously a serious issue, and when the Suricata binary daemon crashes like that it will leave behind the stale PID file it complains about on the next startup attempt. Those files will be in /var/run/ on the firewall. Delete any Suricata PID files you find there if having startup problems.

But if you continue getting core dumps, that is something much more serious than a dangling PID file. This new package version contains the latest upstream binary from the 7.x Suricata branch. It's entirely possible there is something with your hardware or current configuration that Suricata 7 does not play well with. There are users reporting no problems updating to and running the new packge version, so this failure to start does not appear to be a widespread issue. Also, this version has been running and available in the 23.09 snapshot development branch for quite some time with no reported issues.

NogBadTheBad

@bmeeks

Legacy mode

Netgate SG-4860 hardware with igb interfaces

I just tried running it on the LAN interface and it still core dumped.

bmeeks

@NogBadTheBad said in After upgrade to pf+ 23.09 Surricata says it's starting but..:

@bmeeks

Legacy mode

Netgate SG-4860 hardware with igb interfaces

I just tried running it on the LAN interface and it still core dumped.

Don't know how I can help you. I do not have an SG-4860 to test on. I also have nothing with a PPPoE interface configured.

Unfortunately for you there is no way to go back to Suricata 6.x unless you rollback your firewall to 23.05.1. That's not a long-term solution as that pfSense version will eventually go EOL with no support.

But to be honest, I would abandon using an IDS/IPS unless you also have MITM encryption interception enabled and working. The IDS/IPS is blind to the vast majority of traffic traversing a firewall these days. I no longer run any IDS/IPS package on my personal system for that very reason. I have only a couple of testing virtual machines for maintaining/testing the Suricata and Snort packages.

But if you want to run Suricata, then you will need different hardware it seems; or else rollback to the older pfSense Plus and hope something happens upstream in Suricata to produce a future fix. But if the core dump problem is not widespread, then it likely won't get identifed and/or fixed upstream.

NogBadTheBad

@bmeeks it’s a standard intel processor if I’m having issues I’m sure other people will too, perhaps I need to go back to snort.

Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
4 CPUs : 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (inactive)
IPsec-MB Crypto: Yes (inactive)
QAT Crypto: Yes (active)

Is it not possible to have the two versions available?

Euman

@bmeeks I run a mixed mode setup.. Wan is IPS a couple Lan's are IDS legacy

I do hope someone with the equipment and knowledge can figure this out.. a few of us really are stuck owning netgate hardware.

bmeeks

@Euman said in After upgrade to pf+ 23.09 Surricata says it's starting but..:

I do hope someone with the equipment and knowledge can figure this out.. a few of us really are stuck owning netgate hardware.

You have the equipment and likely as much basic knowledge of what's wrong as anyone else at the moment.

Start by simplifying your setup. Turn off all blocking, save the changes, then attempt to start Suricata.
If that fails, save a config.xml backup and then uncheck the "save settings" option in Suricata (on the GLOBAL SETTINGS tab) and remove the package under SYSTEM > PACKAGE MANAGER.
Start over with a clean slate: no blocking and a default configuration. See if Suricata starts and runs then.
Next, add your rules archives back (or some of them), update the rules, enable a few, and see if you can still start and run Suricata.
Somewhere in the above chain you might stumble upon the "where it breaks" point. If not, then restore you previous config.xml (just the PACKAGES portion) to bring back the original Suricata settings. See what happens then when starting.

If it works fine with a clean slate, then you will know it's something in your configuration. If it fails to start with a nearly blank default setup, then it's some hardware issue (unlikely, but not impossible).

Euman

@bmeeks This issue is resolved after completely removing any existing configuration, removing the surricata pkg, rebooting the router, reinstalling surricata and setting up a new configuration.

unsure why we'd have to iron-fist the pkg and configuration however..

Kind Regard

bmeeks

@Euman said in After upgrade to pf+ 23.09 Surricata says it's starting but..:

unsure why we'd have to iron-fist the pkg and configuration however..

Because among the thousands of users of the package there are countless variations in the configurations. Also countless variations in the hardware the package is being run on. That means there are lots of places for something to get sideways. There is no physical way to test all those differences. I can only test on what hardware and configurations I have personally. This is a volunteer created and maintained package. No relation to Netgate at all.

In my many years of maintaining the Snort and Suricata packages I've seen users do some mighty weird things in their configurations - usually by operating under invalid assumptions and making configuration decisions from those faulty assumptions. That might have been your issue, or it may simply have been a random cosmic ray from space altered some RAM cell value .

Take the fact pfSense runs perfectly fine for the vast majority of users, and they easily apply each incremental upgrade without a single hitch. I've never once experienced a single problem updating my pfSense firewalls all the way back to the 1.x RC series. But a handful of others post here with problem after problem with almost each and every pfSense update. Who knows why they have issues???

Euman

@bmeeks I appreciate everyones help

non-netgate hardware pfsense users aren't getting:
-> Default optimized configurations for Netgate hardware appliances

Could it be the single reason in this case?.. pf+ is a different beast!

Didn't intend a flame war.

bmeeks

@Euman said in After upgrade to pf+ 23.09 Surricata says it's starting but..:

@bmeeks I appreciate everyones help

non-netgate hardware pfsense users aren't getting:
-> Default optimized configurations for Netgate hardware appliances

Could it be the single reason in this case?.. pf+ is a different beast!

Didn't intend a flame war.

No flaming implied .

I will admit it has become increasingly difficult to create and test both the Snort and Suricata packages now as all I have for test platforms are CE environments. I have an SG-5100 for my personal network, but it's production and I install no packages on it. And no, I no longer run any IDS/IPS on my personal firewall and have not for more than three years due to the reasons I stated up above in a previous post in this thread.

NRgia

@Euman said in After upgrade to pf+ 23.09 Surricata says it's starting but..:

@bmeeks I appreciate everyones help

non-netgate hardware pfsense users aren't getting:
-> Default optimized configurations for Netgate hardware appliances

Could it be the single reason in this case?.. pf+ is a different beast!

Didn't intend a flame war.

How can Netgate or any other company can deliver "optimized configurations" for infinite combinations of hardware setups? I run a pfSense+ on a whitebox myself, and I have to optimize it myself, or I ask the devs here.