Suricata wont Start after updating pfSense to 2.4.5-RELEASE


  • Sorry if this has been asked/answered; I did do a lot of searching beforehand.

    And thank you VERY much in advanced for your help!!

    Everything has been working great for over 1 year, but I updated pfSense to 2.4.5-RELEASE a few days ago. I noticed yesterday that suricata 5.0.2_2 was not running on any interface.

    I have tried the following:

    1. re-installed suricata
    2. uninstalled suricata, rebooted, installed suricata
    3. disabled ETOpen Emerging Threat rules (as it was failing to update... 404 error)
    4. Increased Fragmentation Memory Cap, Flow Memory Cap, & Stream Memory Cap to 265MB (268435456) per a solution I found in the forums for likely causes of suricata not starting.

    I have NOT uninstalled (deleting settings) as I'm really hoping to avoid having to re-tune.

    Most of my logs are blank. If I need to turn any on and post the contents, please be specific as to which ones and if you could please tell me where to do this; if its not too much trouble.

    Running:
    pfSense 2.45-RELEASE
    FreeBSD 11.2-STABLE
    CPU: Intel i5-5200 @ 2.2Ghz
    4 CPU's: 1 package(s) x 2 core(s) x 2 hardware threads
    AES-NI CPU Crypto: Yes (active)

    Suricata 5.0.2_2

    Thanks again!!


  • These two values you included in your post are not possible together --

    pfSense 2.45-RELEASE
    FreeBSD 11.2-STABLE

    I am going to assume the bottom value is a typo. pfSense-2.4.5 is built upon FreeBSD 11.3-STABLE, not 11.2. I'm thinking maybe the 11.2 is a typo on your part. If not, then your box has a partial upgrade and is quite confused.

    Attempt to start Suricata on an interface using the GUI icons on the INTERFACES tab in Suricata. You should see the "starting" icon spin a few seconds. If it changes to a red X, then Suricata failed to start.

    Immediately, without doing anything else, click on the LOGS VIEW tab, choose the interface in the drop-down that you just tried to start Suricata on, and choose the suricata.log file to view. Paste the contents of that log file back here. If there is something in your configuration that Suricata does not like, it will print an error message in that log.

    If that log is blank, then Suricata is never really ever getting out of the gate. That could be due to a missing or wrong version dependent library. To see if that is the case, check the pfSense system log to see if anything is printed there that might provide a clue. If that log is empty or has no other clue in it, then proceed to the next step below.

    Get to a shell prompt on the firewall either by accessing the console directly if possible, or by using a SSH session. Execute this command at the shell prompt:

    /usr/local/bin/suricata -V
    

    That should result in Suricata printing some version information and exiting. It you see any error messages during this process, post those back here.

    From the steps above, you should find a clue about why Suricata is not starting.


  • Thank you so much for the reply.

    Manually starting an interface fails as usual and the corresponding suricata.log is blank.

    The only pertinent pfSense system log shows:
    Apr 10 10:07:56 php [Suricata] Suricata START for WAN(igb0)...

    When I run suricata -V from the /usr/local/bin I get the following (missing dependency):
    Shared object "libluajit-5.1.so.2" not found, required by "suricata"


  • I suspected a failure on a dependent library. That means your upgrade went south OR you updated Suricata BEFORE you did pfSense. If you did that, do not ever do that again. Always update pfSense FIRST when both a pfSense upgrade AND package updates are both showing.

    The pkg system on your firewall is confused. That library dependency is fulfilled by the openresty library on pfSense.

    You can try two steps to fix this. The first one is less invasive, but not as thorough of a fix. The second is better than a complete reinstall of pfSense-2.4.5, but still not 100% guaranteed to work.

    Least Invasive Step

    From a shell prompt type:

    pkg install luajit-openresty
    

    More Thorough Step

    Follow the instructions found here: https://docs.netgate.com/pfsense/en/latest/install/upgrade-troubleshooting.html#forced-pkg-reinstall.


  • bmeeks, thank you so much. I had to do the "more thorough step". But it works!!

    Thank you again!!


  • @skylinetech said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

    bmeeks, thank you so much. I had to do the "more thorough step". But it works!!

    Thank you again!!

    You're welcome!


  • @bmeeks said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

    pkg install luajit-openresty

    I hate to jump into someone else's thread but this is exactly what I faced. My unit was upgraded by going into the CLI and running option 13 to Update from console. For me, that's always been the most risk-free way of upgrading. The solution was, as linked to above, running:

    pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade
    Then:
    pkg-static upgrade -f
    Then:
    reboot


  • bmeeks, again, thank you for your help. However, I've noticed things are not working correctly. I have stopped suricata on all interfaces yet it keeps blocking IP addresses.


  • @skylinetech said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

    bmeeks, again, thank you for your help. However, I've noticed things are not working correctly. I have stopped suricata on all interfaces yet it keeps blocking IP addresses.

    In Legacy Mode, Suricata blocks by putting IP addresses in a special table in the firewall. Those addresses, once put there, remain until either periodically cleared by the "Remove Blocked Hosts" cron task (if enabled on the GLOBAL SETTINGS tab), they are manually removed by the user, or the firewall is rebooted (rebooting clears out the table as it is a RAM construct).

    Simply stopping Suricata will not remove those previous blocks. You have to do that manually by going to the BLOCKED HOSTS tab and clearing any blocks.

    The exception to this is Inline IPS Mode. When you top Suricata in that mode, the netmap pipe will be torn down and traffic will flow without drops.


  • @bmeeks , I'm sorry, I should have been more detailed. I'm in legacy mode and I have manually cleared the blocked hosts.
    ...

    Interfaces all stopped.
    Blocks manually cleared.
    Wait a few seconds/minutes.
    Checking either the alerts and/or blocks...new ip's are blocked. !?!?


  • @skylinetech said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

    @bmeeks , I'm sorry, I should have been more detailed. I'm in legacy mode and I have manually cleared the blocked hosts.
    ...

    Interfaces all stopped.
    Blocks manually cleared.
    Wait a few seconds/minutes.
    Checking either the alerts and/or blocks...new ip's are blocked. !?!?

    Then you have a zombie Suricata process still running. Do this to find it and kill it:

    1. Stop Suricata on all interfaces using the GUI icon on the INTERFACES tab.

    2. Open a shell prompt on the firewall and execute the following command sequence:

    ps -ax | grep suricata
    

    You should see no running Suricata instances. If you do, get the process id <PID> of any running Suricata process, then use this command to kill that process:

    kill -9 <pid>
    
    1. Run the following command again to verify no more Suricata processes exist:
    ps -ax | suricata
    

    This won't remove any existing blocks, though. Like I said in my earlier post, you will need to go to the BLOCKED HOSTS tab and manually remove any existing blocks (this will clear the snort2c pf table).

    Note: what I mean by "zombie process" is that it is a running instance of the Suricata binary that the GUI code has lost track of and thus can no longer control or see, but that instance will continue running using its configuration from startup (and it can continue to add IP addresses to the blocking table when its rules fire).


  • Hi,

    I had the same problem with suricata afther the upgrade now i'm facing another problem i have tons of logs errors ips is in inline mode

    <Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write failed: libnet_write_raw_ipv4(): -1 bytes written (Permission denied)

    <Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write_raw_ipv4 failed: libnet_write_raw_ipv4(): -1 bytes written (Invalid argument)

    Hope someone can help me out here

    Kind Regards

    Genine collin


  • @genuine said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

    Hi,

    I had the same problem with suricata afther the upgrade now i'm facing another problem i have tons of logs errors ips is in inline mode

    <Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write failed: libnet_write_raw_ipv4(): -1 bytes written (Permission denied)

    <Error> -- [ERRCODE: SC_ERR_LIBNET_WRITE_FAILED(147)] - libnet_write_raw_ipv4 failed: libnet_write_raw_ipv4(): -1 bytes written (Invalid argument)

    Hope someone can help me out here

    Kind Regards

    Genine collin

    Is this with only DROP rules, or do you have some rules set to REJECT? And how is your pfSense firewall configured? Do you by chance have it in Bridge Mode?

    You state you had the "same problem with Suricata after the upgrade". What problem precisely. Did you attempt to update Suricata BEFORE you updated pfSense to 2.4.5? If so, you probably have a hodge-podge of library versions on your box.


  • sorry for the short explication
    well after upgrading pfsense 2.4.5 he did also upgraded suricata with was not starting anymore.
    so I did a clean uninstall and removed also the settings and did a reinstall
    it was not starting I was looking in the log and there was a packet missing I think it was libluajit I'm not sure
    so I installed the lib and suricata was starting up configured as inline mode
    the error appear with drops and rejects
    for the firewall it is configured as normal nothing exotic also not in bridge mode everything was working before the upgrade
    without problems.
    if i have a hodge-podge of library versions how can i check and fixed this


  • @genuine said in Suricata wont Start after updating pfSense to 2.4.5-RELEASE:

    sorry for the short explication
    well after upgrading pfsense 2.4.5 he did also upgraded suricata with was not starting anymore.
    so I did a clean uninstall and removed also the settings and did a reinstall
    it was not starting I was looking in the log and there was a packet missing I think it was libluajit I'm not sure
    so I installed the lib and suricata was starting up configured as inline mode
    the error appear with drops and rejects
    for the firewall it is configured as normal nothing exotic also not in bridge mode everything was working before the upgrade
    without problems.
    if i have a hodge-podge of library versions how can i check and fixed this

    You very likely have a mixture of FreeBSD 11.2 and FreeBSD 11.3 libraries as a result of how you updated. That missing libjuit package is one example. I suspect your libdnet package might also be the wrong version and hence you are getting your current Suricata error. From your symptoms, I'm going to guess you were on pfSense 2.4.4 and saw an update for Suricata posted. But that Suricata update was for the 2.4.5 version of pfSense and has new shared library versions/dependencies that can only be satisfied when pfSense-2.4.5 is already installed. You installed the new Suricata onto a pfSense-2.4.4 system and it would not start (that missing libjuit package is a classic symptom of this upgrade path). So then you updated to pfSense-2.4.5, but that still will not properly update all of the dependent libraries that third-party packages might use. So now you are experiencing weird errors because of the library problems.

    I would recommend you do this. You should reinstall pfSense itself from a clean install and then put your packages back. That will guarantee that you get the correct versions of all the supporting libraries.

    If you don't want to perform a complete reinstall of pfSense, then try this series of commands to refresh the pkg database.

    pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade
    

    The commands above came from this link in the pfSense documentation: https://docs.netgate.com/pfsense/en/latest/install/upgrade-troubleshooting.html.

    And next time you see a pfSense version upgrade notice on the Dashboard, DO NOT update any packages until AFTER you have upgraded pfSense to the new version!