Suricata wont Start after updating pfSense to 2.4.5-RELEASE

bmeeks

I suspected a failure on a dependent library. That means your upgrade went south OR you updated Suricata BEFORE you did pfSense. If you did that, do not ever do that again. Always update pfSense FIRST when both a pfSense upgrade AND package updates are both showing.

The pkg system on your firewall is confused. That library dependency is fulfilled by the openresty library on pfSense.

You can try two steps to fix this. The first one is less invasive, but not as thorough of a fix. The second is better than a complete reinstall of pfSense-2.4.5, but still not 100% guaranteed to work.

Least Invasive Step

From a shell prompt type:

pkg install luajit-openresty

More Thorough Step

Follow the instructions found here: https://docs.netgate.com/pfsense/en/latest/install/upgrade-troubleshooting.html#forced-pkg-reinstall.

skylinetech

bmeeks, thank you so much. I had to do the "more thorough step". But it works!!

Thank you again!!

bmeeks

@skylinetech said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

bmeeks, thank you so much. I had to do the "more thorough step". But it works!!

Thank you again!!

You're welcome!

Stewart

@bmeeks said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

pkg install luajit-openresty

I hate to jump into someone else's thread but this is exactly what I faced. My unit was upgraded by going into the CLI and running option 13 to Update from console. For me, that's always been the most risk-free way of upgrading. The solution was, as linked to above, running:

pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade
Then:
pkg-static upgrade -f
Then:
reboot

skylinetech

bmeeks, again, thank you for your help. However, I've noticed things are not working correctly. I have stopped suricata on all interfaces yet it keeps blocking IP addresses.

bmeeks

@skylinetech said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

bmeeks, again, thank you for your help. However, I've noticed things are not working correctly. I have stopped suricata on all interfaces yet it keeps blocking IP addresses.

In Legacy Mode, Suricata blocks by putting IP addresses in a special table in the firewall. Those addresses, once put there, remain until either periodically cleared by the "Remove Blocked Hosts" cron task (if enabled on the GLOBAL SETTINGS tab), they are manually removed by the user, or the firewall is rebooted (rebooting clears out the table as it is a RAM construct).

Simply stopping Suricata will not remove those previous blocks. You have to do that manually by going to the BLOCKED HOSTS tab and clearing any blocks.

The exception to this is Inline IPS Mode. When you top Suricata in that mode, the netmap pipe will be torn down and traffic will flow without drops.

skylinetech

@bmeeks , I'm sorry, I should have been more detailed. I'm in legacy mode and I have manually cleared the blocked hosts.
...

Interfaces all stopped.
Blocks manually cleared.
Wait a few seconds/minutes.
Checking either the alerts and/or blocks...new ip's are blocked. !?!?

bmeeks

@skylinetech said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

@bmeeks , I'm sorry, I should have been more detailed. I'm in legacy mode and I have manually cleared the blocked hosts.
...

Interfaces all stopped.
Blocks manually cleared.
Wait a few seconds/minutes.
Checking either the alerts and/or blocks...new ip's are blocked. !?!?

Then you have a zombie Suricata process still running. Do this to find it and kill it:

Stop Suricata on all interfaces using the GUI icon on the INTERFACES tab.
Open a shell prompt on the firewall and execute the following command sequence:

ps -ax | grep suricata

You should see no running Suricata instances. If you do, get the process id <PID> of any running Suricata process, then use this command to kill that process:

kill -9 <pid>

Run the following command again to verify no more Suricata processes exist:

ps -ax | suricata

This won't remove any existing blocks, though. Like I said in my earlier post, you will need to go to the BLOCKED HOSTS tab and manually remove any existing blocks (this will clear the snort2c pf table).

Note: what I mean by "zombie process" is that it is a running instance of the Suricata binary that the GUI code has lost track of and thus can no longer control or see, but that instance will continue running using its configuration from startup (and it can continue to add IP addresses to the blocking table when its rules fire).

genuine

Hi,

I had the same problem with suricata afther the upgrade now i'm facing another problem i have tons of logs errors ips is in inline mode

<Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write failed: libnet_write_raw_ipv4(): -1 bytes written (Permission denied)

<Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write_raw_ipv4 failed: libnet_write_raw_ipv4(): -1 bytes written (Invalid argument)

Hope someone can help me out here

Kind Regards

Genine collin

bmeeks

@genuine said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

Hi,

I had the same problem with suricata afther the upgrade now i'm facing another problem i have tons of logs errors ips is in inline mode

<Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write failed: libnet_write_raw_ipv4(): -1 bytes written (Permission denied)

<Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write_raw_ipv4 failed: libnet_write_raw_ipv4(): -1 bytes written (Invalid argument)

Hope someone can help me out here

Kind Regards

Genine collin

Is this with only DROP rules, or do you have some rules set to REJECT? And how is your pfSense firewall configured? Do you by chance have it in Bridge Mode?

You state you had the "same problem with Suricata after the upgrade". What problem precisely. Did you attempt to update Suricata BEFORE you updated pfSense to 2.4.5? If so, you probably have a hodge-podge of library versions on your box.

genuine

sorry for the short explication
well after upgrading pfsense 2.4.5 he did also upgraded suricata with was not starting anymore.
so I did a clean uninstall and removed also the settings and did a reinstall
it was not starting I was looking in the log and there was a packet missing I think it was libluajit I'm not sure
so I installed the lib and suricata was starting up configured as inline mode
the error appear with drops and rejects
for the firewall it is configured as normal nothing exotic also not in bridge mode everything was working before the upgrade
without problems.
if i have a hodge-podge of library versions how can i check and fixed this

bmeeks

@genuine said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

sorry for the short explication
well after upgrading pfsense 2.4.5 he did also upgraded suricata with was not starting anymore.
so I did a clean uninstall and removed also the settings and did a reinstall
it was not starting I was looking in the log and there was a packet missing I think it was libluajit I'm not sure
so I installed the lib and suricata was starting up configured as inline mode
the error appear with drops and rejects
for the firewall it is configured as normal nothing exotic also not in bridge mode everything was working before the upgrade
without problems.
if i have a hodge-podge of library versions how can i check and fixed this

You very likely have a mixture of FreeBSD 11.2 and FreeBSD 11.3 libraries as a result of how you updated. That missing libjuit package is one example. I suspect your libdnet package might also be the wrong version and hence you are getting your current Suricata error. From your symptoms, I'm going to guess you were on pfSense 2.4.4 and saw an update for Suricata posted. But that Suricata update was for the 2.4.5 version of pfSense and has new shared library versions/dependencies that can only be satisfied when pfSense-2.4.5 is already installed. You installed the new Suricata onto a pfSense-2.4.4 system and it would not start (that missing libjuit package is a classic symptom of this upgrade path). So then you updated to pfSense-2.4.5, but that still will not properly update all of the dependent libraries that third-party packages might use. So now you are experiencing weird errors because of the library problems.

I would recommend you do this. You should reinstall pfSense itself from a clean install and then put your packages back. That will guarantee that you get the correct versions of all the supporting libraries.

If you don't want to perform a complete reinstall of pfSense, then try this series of commands to refresh the pkg database.

pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade

The commands above came from this link in the pfSense documentation: https://docs.netgate.com/pfsense/en/latest/install/upgrade-troubleshooting.html.

And next time you see a pfSense version upgrade notice on the Dashboard, DO NOT update any packages until AFTER you have upgraded pfSense to the new version!