Still seeing suricata stop an interface due to .pid error



  • 7/6/2018 -- 12:55:39 - <Notice> -- This is Suricata version 4.0.4 RELEASE
    7/6/2018 -- 12:55:39 - <Info> -- CPUs/cores online: 2
    7/6/2018 -- 12:55:39 - <Info> -- HTTP memcap: 67108864
    7/6/2018 -- 12:55:39 - <Notice> -- using flow hash instead of active packets
    7/6/2018 -- 12:55:39 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_mvneta210310.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_mvneta210310.pid. Aborting!

    Then when I stop the service, the file's gone. Restart. Crashes with same error.



  • This post is deleted!


  • I've deleted my previous update. It's still not working.

    I'm basically using the same parameters and lists in snort. But trying to config suricata, I can get alerts working but as soon as I enable blocking the interface service dies.



  • Is it this bug in 2.4.3_1 by chance?
    https://redmine.pfsense.org/issues/8518
    (rule syntax error on incomplete rule (missing IPs): "There were error(s) loading the rules: /tmp/rules.debug:371: syntax error - The line in question reads [371]: pass out route-to ( vmx0 xx.xx.xx.xx ) from to !/ tracker 1000027964 keep state allow-opts label "let out anything from firewall host itself"; @ ...")



  • I started having this same problem since yesterday:

    7/9/2018 -- 22:42:06 - <Notice> -- This is Suricata version 4.0.5 RELEASE
    7/9/2018 -- 22:42:06 - <Info> -- CPUs/cores online: 4
    7/9/2018 -- 22:42:06 - <Info> -- HTTP memcap: 67108864
    7/9/2018 -- 22:42:06 - <Notice> -- using flow hash instead of active packets
    7/9/2018 -- 22:42:06 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_pppoe030843.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_pppoe030843.pid. Aborting!
    

    Running pfSense 2.4.3-RELEASE-p1.



  • @occamsrazor said in Still seeing suricata stop an interface due to .pid error:

    I started having this same problem since yesterday:

    7/9/2018 -- 22:42:06 - <Notice> -- This is Suricata version 4.0.5 RELEASE
    7/9/2018 -- 22:42:06 - <Info> -- CPUs/cores online: 4
    7/9/2018 -- 22:42:06 - <Info> -- HTTP memcap: 67108864
    7/9/2018 -- 22:42:06 - <Notice> -- using flow hash instead of active packets
    7/9/2018 -- 22:42:06 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_pppoe030843.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_pppoe030843.pid. Aborting!
    

    Running pfSense 2.4.3-RELEASE-p1.

    You can try manually deleting that PID file and try the start again. If it fails to start again, look carefully through the suricata.log file and the system log to see what errors are being posted.



  • @bmeeks

    I actually found some of your advice in an older thread:
    https://forum.netgate.com/topic/120647/suricata-crash-on-latest-2-4-0-rc/6
    ..and doubled the Stream Mem Cap setting to 134217728. PS - I have 8GB RAM.
    Perhaps too soon to really tell, but so far it seems to have fixed the problem.



  • My Stream Mem setting it's at 4294967296
    suricata still tend to have pid file problem from time to time.
    Even with a fresh install.



  • Look through the suricata.log file for the affected interface. You can open and view that file's contents on the LOGS VIEW tab. Look for any lines that start with ERRCODE: SC_ERR. Ignore the line about the PID file. That is a symptom from another root problem.



  • ouch, the log file it's plauged with those error....

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"ET TROJAN Cobalt Group SSL Certificate Detected"; flow:established,from_server; content:"|55 04 03|"; content:"|0f|dns-verifon.com"; distance:1; within:16; metadata: former_category TROJAN; reference:md5,26406f5cc72e13c798485f80ad3cbbdb; classtype:trojan-activity; sid:2025438; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_26, malware_family Cobalt_Group, performance_impact Low, updated_at 2018_03_26;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10100

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Win32/Spy.Banker.AAQD Checkin"; flow:to_server,established; content:"POST"; http_method; nocase; content:"valor="; depth:6; http_client_body; content:"verde"; http_client_body; content:"branco"; http_client_body; content:"vermelho"; fast_pattern:only; http_client_body; http_header_names; content:!"Referer"; metadata: former_category TROJAN; reference:md5,759db11b07f3a370338f2e0a28eb1def; reference:url,www.virusradar.com/en/Win32_Spy.Banker.AAQD/description; classtype:trojan-activity; sid:2018516; rev:5; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2014_04_24, performance_impact Low, updated_at 2018_03_26;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10101

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Possible Sharik/Smoke Loader Microsoft Connectivity check M2"; flow:established,to_server; content:"GET"; http_method; content:"/vstudio"; http_uri; fast_pattern; urilen:8; content:"msdn.microsoft.com"; http_host; http_header_names; content:"|0d 0a|Cache-Control|0d 0a|Connection|0d 0a|Pragma|0d 0a|User-Agent|0d 0a|Host|0d 0a 0d 0a|"; metadata: former_category TROJAN; reference:md5,e297f2ed2d162ad925ac140915a21405; classtype:trojan-activity; sid:2025439; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_27, malware_family Smoke_Loader, updated_at 2018_03_27;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10102

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Possible Sharik/Smoke Loader Microsoft Connectivity check M3"; flow:established,to_server; content:"GET"; http_method; content:"/visualstudio/"; http_uri; fast_pattern; urilen:14; content:"www.microsoft.com"; http_host; http_header_names; content:"|0d 0a|Cache-Control|0d 0a|Connection|0d 0a|Pragma|0d 0a|User-Agent|0d 0a|Host|0d 0a 0d 0a|"; metadata: former_category TROJAN; reference:md5,e297f2ed2d162ad925ac140915a21405; classtype:trojan-activity; sid:2025440; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_27, malware_family Smoke_Loader, updated_at 2018_03_27;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10103

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Sharik/Smoke CnC Beacon 10"; flow:established,to_server; content:"POST"; http_method; pcre:"//\d+/$/U"; pcre:"/(?:MSIE|rv\x3a)/V"; pcre:"/^[\x20-\x7e\r\n]{0,20}[^\x20-\x7e\r\n]/Ps"; http_content_len; content:"63"; fast_pattern; http_content_type; content:"application/x-www-form-urlencoded"; http_connection; content:"keep-alive"; nocase; http_header_names; content:"|0d 0a|Content-Type|0d 0a|Host|0d 0a|User-Agent"; content:!"Referer"; content:!"Accept"; metadata: former_category TROJAN; reference:md5,e297f2ed2d162ad925ac140915a21405; classtype:trojan-activity; sid:2025441; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_27, malware_family Smoke_Loader, performance_impact Moderate, updated_at 2018_03_27;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10104

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"



  • @val said in Still seeing suricata stop an interface due to .pid error:

    ouch, the log file it's plauged with those error....

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"ET TROJAN Cobalt Group SSL Certificate Detected"; flow:established,from_server; content:"|55 04 03|"; content:"|0f|dns-verifon.com"; distance:1; within:16; metadata: former_category TROJAN; reference:md5,26406f5cc72e13c798485f80ad3cbbdb; classtype:trojan-activity; sid:2025438; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_26, malware_family Cobalt_Group, performance_impact Low, updated_at 2018_03_26;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10100

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Win32/Spy.Banker.AAQD Checkin"; flow:to_server,established; content:"POST"; http_method; nocase; content:"valor="; depth:6; http_client_body; content:"verde"; http_client_body; content:"branco"; http_client_body; content:"vermelho"; fast_pattern:only; http_client_body; http_header_names; content:!"Referer"; metadata: former_category TROJAN; reference:md5,759db11b07f3a370338f2e0a28eb1def; reference:url,www.virusradar.com/en/Win32_Spy.Banker.AAQD/description; classtype:trojan-activity; sid:2018516; rev:5; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2014_04_24, performance_impact Low, updated_at 2018_03_26;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10101

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Possible Sharik/Smoke Loader Microsoft Connectivity check M2"; flow:established,to_server; content:"GET"; http_method; content:"/vstudio"; http_uri; fast_pattern; urilen:8; content:"msdn.microsoft.com"; http_host; http_header_names; content:"|0d 0a|Cache-Control|0d 0a|Connection|0d 0a|Pragma|0d 0a|User-Agent|0d 0a|Host|0d 0a 0d 0a|"; metadata: former_category TROJAN; reference:md5,e297f2ed2d162ad925ac140915a21405; classtype:trojan-activity; sid:2025439; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_27, malware_family Smoke_Loader, updated_at 2018_03_27;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10102

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Possible Sharik/Smoke Loader Microsoft Connectivity check M3"; flow:established,to_server; content:"GET"; http_method; content:"/visualstudio/"; http_uri; fast_pattern; urilen:14; content:"www.microsoft.com"; http_host; http_header_names; content:"|0d 0a|Cache-Control|0d 0a|Connection|0d 0a|Pragma|0d 0a|User-Agent|0d 0a|Host|0d 0a 0d 0a|"; metadata: former_category TROJAN; reference:md5,e297f2ed2d162ad925ac140915a21405; classtype:trojan-activity; sid:2025440; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_27, malware_family Smoke_Loader, updated_at 2018_03_27;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10103

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Sharik/Smoke CnC Beacon 10"; flow:established,to_server; content:"POST"; http_method; pcre:"//\d+/$/U"; pcre:"/(?:MSIE|rv\x3a)/V"; pcre:"/^[\x20-\x7e\r\n]{0,20}[^\x20-\x7e\r\n]/Ps"; http_content_len; content:"63"; fast_pattern; http_content_type; content:"application/x-www-form-urlencoded"; http_connection; content:"keep-alive"; nocase; http_header_names; content:"|0d 0a|Content-Type|0d 0a|Host|0d 0a|User-Agent"; content:!"Referer"; content:!"Accept"; metadata: former_category TROJAN; reference:md5,e297f2ed2d162ad925ac140915a21405; classtype:trojan-activity; sid:2025441; rev:2; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, signature_severity Major, created_at 2018_03_27, malware_family Smoke_Loader, performance_impact Moderate, updated_at 2018_03_27;)" from file /usr/local/etc/suricata/suricata_15439_pppoe0/rules/suricata.rules at line 10104

    • 13/11/2018 -- 23:02:14 - <Error> -- [ERRCODE: SC_ERR_REFERENCE_UNKNOWN(150)] - unknown reference key "md5". Supported keys are defined in reference.config file. Please have a look at the conf param "reference-config-file"

    Which rule sets are you using? Are they the Snort Subscriber Rules or are they from Emerging Threats. At first glance those errors appear to be coming from Snort Subscriber Rules (formerly known as "Snort VRT" rules). See my other posts on this topic. Not all Snort rule keywords are understood and processed by Suricata. Snort is a competing IDS/IPS package, and the Snort Rules are specifically written for Snort by the Snort team. Suricata digests most of the older Snort rule keywords, but not all of them. Hence you will always see rule load failures in Suricata if you use Snort rules. How many failed rules depends on which specific rules in the Snort group you enable.

    Another potential issue is trying to use the Snort rules designed only for Snort3. Suricata does NOT like those rules! Do not use the Snort3 rules archive for Suricata.



  • @bmeeks said in Still seeing suricata stop an interface due to .pid error:

    Look through the suricata.log file for the affected interface. You can open and view that file's contents on the LOGS VIEW tab. Look for any lines that start with ERRCODE: SC_ERR. Ignore the line about the PID file. That is a symptom from another root problem.

    I actually ran into this upgrading a client's HA/CARP dual routers from pfSense 2.4.2 to 2.4.4. After upgrading router 2, on Friday on router1 I entered CARP maintenance mode, uninstalled Suricata (and pfBlockerNG), upgraded, and reinstalled (thereby upgrading Suricata from 4.0.1 to 4.0.5). Yesterday I realized Suricata was not running on either router, and wouldn't start due to the .pid file. It looks to me like it did start after installation but apparently crashed at some point after? I deleted the .pid file (via Diagnostics/Command Prompt) and it started fine on both.

    Log from router1 (an SG-3100):

    9/11/2018 -- 15:26:46 - <Notice> -- This is Suricata version 4.0.5 RELEASE
    9/11/2018 -- 15:26:46 - <Info> -- CPUs/cores online: 2
    9/11/2018 -- 15:26:46 - <Info> -- HTTP memcap: 67108864
    9/11/2018 -- 15:26:46 - <Notice> -- using flow hash instead of active packets
    9/11/2018 -- 15:27:11 - <Info> -- 1 rule files processed. 12073 rules successfully loaded, 0 rules failed
    9/11/2018 -- 15:27:11 - <Info> -- Threshold config parsed: 0 rule(s) found
    9/11/2018 -- 15:27:11 - <Info> -- 12073 signatures processed. 873 are IP-only rules, 4437 are inspecting packet payload, 8258 inspect application layer, 102 are decoder event only
    9/11/2018 -- 15:27:18 - <Info> -- alert-pf -> Creating automatic firewall interface IP address Pass List.
    (...adding firewall interfaces...)
    9/11/2018 -- 15:27:18 - <Info> -- alert-pf output device (regular) initialized: block.log
    9/11/2018 -- 15:27:18 - <Info> -- alert-pf -> Pass List /usr/local/etc/suricata/suricata_25752_mvneta1/passlist parsed: 22 IP addresses loaded.
    9/11/2018 -- 15:27:18 - <Info> -- alert-pf -> Created firewall interface IP change monitor thread for auto-whitelisting of firewall interface IP addresses.
    9/11/2018 -- 15:27:18 - <Info> -- alert-pf output initialized, pf-table=snort2c block-ip=both kill-state=on block-drops-only=off
    9/11/2018 -- 15:27:18 - <Info> -- alert-pf -> Firewall interface IP address change notification monitoring thread started.
    9/11/2018 -- 15:27:18 - <Info> -- fast output device (regular) initialized: alerts.log
    9/11/2018 -- 15:27:18 - <Info> -- http-log output device (regular) initialized: http.log
    9/11/2018 -- 15:27:18 - <Info> -- Using 1 live device(s).
    9/11/2018 -- 15:27:18 - <Info> -- using interface mvneta1
    9/11/2018 -- 15:27:18 - <Info> -- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
    9/11/2018 -- 15:27:18 - <Info> -- Set snaplen to 1518 for 'mvneta1'
    9/11/2018 -- 15:27:18 - <Info> -- RunModeIdsPcapAutoFp initialised
    9/11/2018 -- 15:27:18 - <Notice> -- all 3 packet processing threads, 2 management threads initialized, engine started.
    12/11/2018 -- 10:32:02 - <Notice> -- This is Suricata version 4.0.5 RELEASE
    12/11/2018 -- 10:32:02 - <Info> -- CPUs/cores online: 2
    12/11/2018 -- 10:32:02 - <Info> -- HTTP memcap: 67108864
    12/11/2018 -- 10:32:02 - <Notice> -- using flow hash instead of active packets
    12/11/2018 -- 10:32:02 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_mvneta125752.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_mvneta125752.pid. Aborting!

    Note the 3 day skip between the last entry and when I tried to start it yesterday. The last alert was 9/11/2018 at 14:51:18 which was just before I upgraded router1 so I am not sure it stayed running very long afterwards.



  • The “stale PID file” error is a red herring consequence of the Suricata process stopping abruptly. When it crashes, it fails to remove the PID file. So the root cause is the Suricata process crashing.

    Look in the pfSense system log to see if there are any Suricata related errors shown — maybe a Signal 10 error since you mentioned this is an SG-3100 appliance.



  • Right, I got that the .pid is just a leftover. You're right there is a "signal 10 (core dumped)" though it doesn't log the word "error" with it. 1m 4s after I logged out.



  • @teamits said in Still seeing suricata stop an interface due to .pid error:

    Right, I got that the .pid is just a leftover. You're right there is a "signal 10 (core dumped)" though it doesn't log the word "error" with it. 1m 4s after I logged out.

    I know where that’s coming from. Will need to work with the pfSense crew to fix it, though. Might take a couple of days to get fixed. This problem is specific to armv7 hardware like the SG-3100. The usual fix is to disable compiler optimizations when compiling a package for armv7 chips like the one used in the SG-3100.



  • Hi @bmeeks

    I found the core dump from pfSense log but it's signal 11 and I am not on a armv7 hardware.

    Nov 18 11:24:59 kernel pid 56681 (suricata), uid 0: exited on signal 11 (core dumped)
    Nov 18 11:24:59 kernel pppoe0: promiscuous mode disabled

    I've googled and found this back in 2.4.0 RC
    https://redmine.pfsense.org/issues/7891

    Also I have done coupe times of fresh installs and still happening.

    Let me know if I could provide more information to help to resolve this problem.



  • @val said in Still seeing suricata stop an interface due to .pid error:

    Hi @bmeeks

    I found the core dump from pfSense log but it's signal 11 and I am not on a armv7 hardware.

    Nov 18 11:24:59 kernel pid 56681 (suricata), uid 0: exited on signal 11 (core dumped)
    Nov 18 11:24:59 kernel pppoe0: promiscuous mode disabled

    I've googled and found this back in 2.4.0 RC
    https://redmine.pfsense.org/issues/7891

    Also I have done coupe times of fresh installs and still happening.

    Let me know if I could provide more information to help to resolve this problem.

    I will need some correlated logs to give a hint where the issue might be. So I will need the few lines before and after the Signal 11 error from the pfSense log (with timestamps) and also the contents of the suricata.log file for the interface. I need the log data and timestamps so that I can correlate the Signal 11 crash with what Suricata might have been doing at that instant. That's where the suricata.log file comes in. Secondly, do you know if this crash occurs during startup, or does Suricata run for a while and then stop? Suricata is generally fully started in about 15 to 30 seconds on most hardware.

    I can tell you that Suricata and PPPoE connections on FreeBSD have not liked each other in the past. There are also apparently some current issues in FreeBSD 11.x with PPPoE connections. There are a few posts in other sub-forums here from users having issues with PPPoE connections (and those issues are not due to Suricata).



  • @bmeeks

    The timestamps are like 20 30 mins apart so I didn't post them, I will post them now, and Suricata did runs for a while and then stop. and I have to manually remove the pid file for it can be starts again.
    Also yes my ISP required PPPoE.

    Log from pfSense log:-

    Nov 18 10:34:08	php-fpm	5977	/index.php: Session timed out for user 'admin' from: 192.168.2.8 (Local Database)
    Nov 18 10:34:12	php-fpm	6331	/index.php: Successful login for user 'admin' from: 192.168.2.8 (Local Database)
    Nov 18 10:42:46	check_reload_status		Syncing firewall
    Nov 18 10:42:46	php-fpm	55264	/suricata/suricata_suppress_edit.php: Beginning configuration backup to .https://acb.netgate.com/save
    Nov 18 10:42:49	php-fpm	55264	/suricata/suricata_suppress_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
    Nov 18 11:24:59	kernel		pid 56681 (suricata), uid 0: exited on signal 11 (core dumped)
    Nov 18 11:24:59	kernel		pppoe0: promiscuous mode disabled
    Nov 18 12:30:00	php-cgi		rc.update_urltables: /etc/rc.update_urltables: Starting up.
    Nov 18 12:30:00	php-cgi		rc.update_urltables: /etc/rc.update_urltables: Sleeping for 31 seconds.
    Nov 18 12:30:31	php-cgi		rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates
    Nov 18 12:30:31	php-cgi		rc.update_urltables: /etc/rc.update_urltables: pfB_GameCompanyList_v4 does not need updating.
    

    Log from Suricata matching my pppoe interface and timestamps:-

    11/18/2018-11:21:28.452082 us.launcher.battle.net[**]/service/wow/alert/en-us[**]Battle.net/1.12.5.10733[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]289 bytes[**]202.124.109.10:64409 -> 24.105.29.21:80
    11/18/2018-11:21:31.934692 ping.pinyin.sogou.com[**]/pingback_bubble.gif?h=2014BC9687A87830D4738DC313449F9B&v=9.0.0.2502&r=0000_sogou_pinyin_77c&passport=&ppversion=3.1.0.2064&bb_type=Query_news[**]SogouIMEMiniSetup_imepopup[**]<no referer>[**]GET[**]HTTP/1.1[**]404[**]0 bytes[**]202.124.109.10:11572 -> 120.92.1.21:80
    11/18/2018-11:21:33.198607 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:21:40.343920 clientconfig.akamai.steamstatic.com[**]/appinfo/323190/sha/6c2ddd3442c281035e27a4a956caf61fa3815be0.txt.gz[**]Valve/Steam HTTP Client 1.0[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]2798 bytes[**]202.124.109.10:28884 -> 104.96.169.40:80
    11/18/2018-11:21:48.197728 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:21:53.421853 heartbeat.dm.origin.com[**]/pulse?authon&user=9554E389C33698512EDEC7A6508FF617&url_heartbeat=1,0,212,212,0&db_conn=1,0,0,0,0[**]Mozilla/5.0 EA Download Manager Origin/10.5.30.15625[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]0 bytes[**]202.124.109.10:37490 -> 52.201.16.228:80
    11/18/2018-11:21:53.743444 us.patch.battle.net[**]/catalogs/cdns?nocache=15424933128533668[**]Battle.net/1.12.5.10733[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]1843 bytes[**]202.124.109.10:2973 -> 37.244.26.41:1119
    11/18/2018-11:21:53.744112 us.patch.battle.net[**]/catalogs/versions?nocache=15424933128533668[**]Battle.net/1.12.5.10733[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]553 bytes[**]202.124.109.10:20113 -> 37.244.26.41:1119
    11/18/2018-11:22:03.199546 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:22:18.198418 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:22:27.301041 us.launcher.battle.net[**]/service/app/maintenance/en-us[**]Battle.net/1.12.5.10733[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]1 bytes[**]202.124.109.10:44870 -> 24.105.29.21:80
    11/18/2018-11:22:27.316961 us.launcher.battle.net[**]/service/app/alert/en-us[**]Battle.net/1.12.5.10733[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]1 bytes[**]202.124.109.10:44333 -> 24.105.29.21:80
    11/18/2018-11:22:33.198484 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:22:48.198356 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:22:53.421475 heartbeat.dm.origin.com[**]/pulse?authon&user=9554E389C33698512EDEC7A6508FF617&url_heartbeat=1,0,211,211,0&db_conn=1,0,0,0,0[**]Mozilla/5.0 EA Download Manager Origin/10.5.30.15625[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]0 bytes[**]202.124.109.10:37490 -> 52.201.16.228:80
    11/18/2018-11:23:03.199116 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:23:18.197696 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:23:33.198524 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:23:48.197903 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:23:53.423492 heartbeat.dm.origin.com[**]/pulse?authon&user=9554E389C33698512EDEC7A6508FF617&url_heartbeat=1,0,212,212,0&db_conn=1,0,0,0,0[**]Mozilla/5.0 EA Download Manager Origin/10.5.30.15625[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]0 bytes[**]202.124.109.10:37490 -> 52.201.16.228:80
    11/18/2018-11:24:03.198443 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:24:18.198552 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:24:33.199220 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:24:43.329167 www.interwise.com.tw[**]/fs/[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]105327 bytes[**]202.124.109.10:14625 -> 211.72.207.204:80
    11/18/2018-11:24:43.610280 www.interwise.com.tw[**]/fs/vendor/font-awesome/css/font-awesome.min.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]30998 bytes[**]202.124.109.10:11191 -> 211.72.207.204:80
    11/18/2018-11:24:44.245200 www.interwise.com.tw[**]/fs/vendor/owl.carousel/assets/owl.carousel.min.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]2935 bytes[**]202.124.109.10:24358 -> 211.72.207.204:80
    11/18/2018-11:24:44.290835 www.interwise.com.tw[**]/fs/vendor/magnific-popup/magnific-popup.min.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]5204 bytes[**]202.124.109.10:25975 -> 211.72.207.204:80
    11/18/2018-11:24:44.052305 www.interwise.com.tw[**]/fs/vendor/bootstrap/css/bootstrap.min.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]127343 bytes[**]202.124.109.10:46209 -> 211.72.207.204:80
    11/18/2018-11:24:44.249147 www.interwise.com.tw[**]/fs/vendor/owl.carousel/assets/owl.theme.default.min.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]935 bytes[**]202.124.109.10:21746 -> 211.72.207.204:80
    11/18/2018-11:24:44.448604 www.interwise.com.tw[**]/fs/vendor/simple-line-icons/css/simple-line-icons.min.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]10943 bytes[**]202.124.109.10:40085 -> 211.72.207.204:80
    11/18/2018-11:24:44.844444 www.interwise.com.tw[**]/fs/css/theme-blog.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]6099 bytes[**]202.124.109.10:65373 -> 211.72.207.204:80
    11/18/2018-11:24:44.948229 www.interwise.com.tw[**]/fs/vendor/animate/animate.min.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]52789 bytes[**]202.124.109.10:35514 -> 211.72.207.204:80
    11/18/2018-11:24:45.112465 www.interwise.com.tw[**]/fs/css/theme-shop.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]13271 bytes[**]202.124.109.10:52479 -> 211.72.207.204:80
    11/18/2018-11:24:45.568022 www.interwise.com.tw[**]/fs/vendor/rs-plugin/css/settings.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]29609 bytes[**]202.124.109.10:43818 -> 211.72.207.204:80
    11/18/2018-11:24:45.790644 www.interwise.com.tw[**]/fs/css/theme.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]110236 bytes[**]202.124.109.10:15582 -> 211.72.207.204:80
    11/18/2018-11:24:46.145201 www.interwise.com.tw[**]/fs/css/custom.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]43 bytes[**]202.124.109.10:26807 -> 211.72.207.204:80
    11/18/2018-11:24:46.669329 www.interwise.com.tw[**]/fs/vendor/modernizr/modernizr.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]10333 bytes[**]202.124.109.10:21894 -> 211.72.207.204:80
    11/18/2018-11:24:46.578166 www.interwise.com.tw[**]/fs/css/theme-elements.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]164999 bytes[**]202.124.109.10:4462 -> 211.72.207.204:80
    11/18/2018-11:24:46.588670 www.interwise.com.tw[**]/fs/vendor/rs-plugin/css/navigation.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]57274 bytes[**]202.124.109.10:30876 -> 211.72.207.204:80
    11/18/2018-11:24:46.772840 www.interwise.com.tw[**]/fs/vendor/rs-plugin/css/layers.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]132922 bytes[**]202.124.109.10:16740 -> 211.72.207.204:80
    11/18/2018-11:24:47.060061 www.interwise.com.tw[**]/fs/css/skins/skin-church.css[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]137749 bytes[**]202.124.109.10:61653 -> 211.72.207.204:80
    11/18/2018-11:24:47.641189 www.interwise.com.tw[**]/fs/vendor/jquery.easing/jquery.easing.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]5144 bytes[**]202.124.109.10:40212 -> 211.72.207.204:80
    11/18/2018-11:24:47.645259 www.interwise.com.tw[**]/fs/vendor/jquery-cookie/jquery-cookie.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]1414 bytes[**]202.124.109.10:42891 -> 211.72.207.204:80
    11/18/2018-11:24:47.636911 www.interwise.com.tw[**]/fs/vendor/jquery.appear/jquery.appear.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]1584 bytes[**]202.124.109.10:38081 -> 211.72.207.204:80
    11/18/2018-11:24:48.199170 www.gstatic.com[**]/generate_204[**]Go-http-client/1.1[**]<no referer>[**]GET[**]HTTP/1.1[**]204[**]0 bytes[**]2406:e007:1e4e:5c00:0000:0000:0000:1d05:44214 -> 2404:6800:4006:0808:0000:0000:0000:2003:80
    11/18/2018-11:24:48.258652 www.interwise.com.tw[**]/fs/vendor/popper/umd/popper.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]19236 bytes[**]202.124.109.10:46707 -> 211.72.207.204:80
    11/18/2018-11:24:48.483124 www.interwise.com.tw[**]/fs/img/interwise_logo.png[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]12052 bytes[**]202.124.109.10:12250 -> 211.72.207.204:80
    11/18/2018-11:24:48.485143 www.interwise.com.tw[**]/fs/img/H2.png[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]11437 bytes[**]202.124.109.10:47173 -> 211.72.207.204:80
    11/18/2018-11:24:48.539082 www.interwise.com.tw[**]/fs/vendor/jquery/jquery.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]86659 bytes[**]202.124.109.10:40746 -> 211.72.207.204:80
    11/18/2018-11:24:48.747570 www.interwise.com.tw[**]/fs/vendor/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/vendor/font-awesome/css/font-awesome.min.css[**]GET[**]HTTP/1.1[**]200[**]77160 bytes[**]202.124.109.10:6694 -> 211.72.207.204:80
    11/18/2018-11:24:49.091739 www.interwise.com.tw[**]/fs/vendor/jquery.easy-pie-chart/jquery.easy-pie-chart.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]3970 bytes[**]202.124.109.10:51559 -> 211.72.207.204:80
    11/18/2018-11:24:49.306355 www.interwise.com.tw[**]/fs/vendor/common/common.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]14509 bytes[**]202.124.109.10:51938 -> 211.72.207.204:80
    11/18/2018-11:24:49.329520 www.interwise.com.tw[**]/fs/vendor/jquery.gmap/jquery.gmap.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]3874 bytes[**]202.124.109.10:9437 -> 211.72.207.204:80
    11/18/2018-11:24:49.654591 www.interwise.com.tw[**]/fs/vendor/jquery.validation/jquery.validation.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]23495 bytes[**]202.124.109.10:38696 -> 211.72.207.204:80
    11/18/2018-11:24:49.643125 www.interwise.com.tw[**]/fs/vendor/jquery.lazyload/jquery.lazyload.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]3601 bytes[**]202.124.109.10:23052 -> 211.72.207.204:80
    11/18/2018-11:24:49.658938 www.interwise.com.tw[**]/fs/vendor/bootstrap/js/bootstrap.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]50564 bytes[**]202.124.109.10:60081 -> 211.72.207.204:80
    11/18/2018-11:24:50.230249 www.interwise.com.tw[**]/fs/vendor/vide/vide.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]4480 bytes[**]202.124.109.10:13393 -> 211.72.207.204:80
    11/18/2018-11:24:50.750558 www.interwise.com.tw[**]/fs/vendor/magnific-popup/jquery.magnific-popup.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]20216 bytes[**]202.124.109.10:55108 -> 211.72.207.204:80
    11/18/2018-11:24:50.701299 www.interwise.com.tw[**]/fs/vendor/isotope/jquery.isotope.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]35313 bytes[**]202.124.109.10:37170 -> 211.72.207.204:80
    11/18/2018-11:24:50.798556 www.interwise.com.tw[**]/fs/vendor/owl.carousel/owl.carousel.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]42766 bytes[**]202.124.109.10:44305 -> 211.72.207.204:80
    11/18/2018-11:24:51.302041 www.interwise.com.tw[**]/fs/js/custom.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]0 bytes[**]202.124.109.10:60651 -> 211.72.207.204:80
    11/18/2018-11:24:51.327699 www.interwise.com.tw[**]/fs/js/theme.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]63176 bytes[**]202.124.109.10:43178 -> 211.72.207.204:80
    11/18/2018-11:24:51.656842 www.interwise.com.tw[**]/fs/js/theme.init.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]8995 bytes[**]202.124.109.10:20940 -> 211.72.207.204:80
    11/18/2018-11:24:51.903590 www.interwise.com.tw[**]/fs/js/examples/examples.gallery.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]2259 bytes[**]202.124.109.10:32939 -> 211.72.207.204:80
    11/18/2018-11:24:51.989379 www.interwise.com.tw[**]/fs/img/bg-full.jpg?ts=20181114001[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]661247 bytes[**]202.124.109.10:59813 -> 211.72.207.204:80
    11/18/2018-11:24:52.172264 www.interwise.com.tw[**]/fs/js/examples/examples.portfolio.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]13664 bytes[**]202.124.109.10:22182 -> 211.72.207.204:80
    11/18/2018-11:24:52.263620 www.interwise.com.tw[**]/fs/vendor/rs-plugin/js/jquery.themepunch.tools.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]107382 bytes[**]202.124.109.10:16879 -> 211.72.207.204:80
    11/18/2018-11:24:52.492355 www.interwise.com.tw[**]/fs/img/15.png[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]4308 bytes[**]202.124.109.10:26453 -> 211.72.207.204:80
    11/18/2018-11:24:53.295498 www.interwise.com.tw[**]/fs/vendor/rs-plugin/js/jquery.themepunch.revolution.min.js[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]227827 bytes[**]202.124.109.10:53349 -> 211.72.207.204:80
    11/18/2018-11:24:53.416145 heartbeat.dm.origin.com[**]/pulse?authon&user=9554E389C33698512EDEC7A6508FF617&url_heartbeat=1,0,213,213,0&db_conn=1,0,0,0,0[**]Mozilla/5.0 EA Download Manager Origin/10.5.30.15625[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]0 bytes[**]202.124.109.10:37490 -> 52.201.16.228:80
    11/18/2018-11:24:53.402891 www.interwise.com.tw[**]/fs/img/products/cover01.jpg?[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]86845 bytes[**]202.124.109.10:44927 -> 211.72.207.204:80
    11/18/2018-11:24:53.664732 www.interwise.com.tw[**]/fs/img/products/cover02.jpg?[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]87431 bytes[**]202.124.109.10:63144 -> 211.72.207.204:80
    11/18/2018-11:24:53.846365 www.interwise.com.tw[**]/fs/img/products/cover03.jpg?[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]102568 bytes[**]202.124.109.10:4551 -> 211.72.207.204:80
    11/18/2018-11:24:54.182473 www.interwise.com.tw[**]/fs/vendor/rs-plugin/fonts/revicons/revicons.woff?5510888[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/vendor/rs-plugin/css/settings.css[**]GET[**]HTTP/1.1[**]200[**]7536 bytes[**]202.124.109.10:18230 -> 211.72.207.204:80
    11/18/2018-11:24:54.245086 www.interwise.com.tw[**]/fs/img/products/cover05.jpg?[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]86932 bytes[**]202.124.109.10:59645 -> 211.72.207.204:80
    11/18/2018-11:24:54.281927 www.interwise.com.tw[**]/fs/img/products/cover04.jpg?[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]103155 bytes[**]202.124.109.10:64158 -> 211.72.207.204:80
    11/18/2018-11:24:54.952878 www.interwise.com.tw[**]/fs/img/storyword.png[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]27791 bytes[**]202.124.109.10:42012 -> 211.72.207.204:80
    11/18/2018-11:24:55.431790 www.interwise.com.tw[**]/fs/img/products/cover06.jpg?[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]87524 bytes[**]202.124.109.10:50166 -> 211.72.207.204:80
    11/18/2018-11:24:55.689660 www.interwise.com.tw[**]/fs/img/products/cover09.jpg?[**]Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36[**]http://www.interwise.com.tw/fs/[**]GET[**]HTTP/1.1[**]200[**]85979 bytes[**]202.124.109.10:10364 -> 211.72.207.204:80
    11/18/2018-12:36:35.780694 us.patch.battle.net[**]/bna/versions?nocache=3494586659[**]agent/2.15.4.6478[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]996 bytes[**]202.124.109.10:18503 -> 37.244.26.41:1119
    11/18/2018-12:36:36.431875 us.patch.battle.net[**]/bna/cdns?nocache=4048471578[**]agent/2.15.4.6478[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]2101 bytes[**]202.124.109.10:32536 -> 37.244.26.41:1119
    11/18/2018-12:36:38.310906 us.launcher.battle.net[**]/service/app/alert/en-us[**]Battle.net/1.12.5.10733[**]<no referer>[**]GET[**]HTTP/1.1[**]200[**]1 bytes[**]202.124.109.10:54578 -> 24.105.29.21:80
    


  • That's not the Suricata log I need to see. I need to see the contents of the suricata.log file in the interface sub-directory under /var/log/suricata. It appears you posted the contents of a logging file showing HTTP traffic instead.



  • Please advise if I'm hijacking and I'll start another thread.

    I've been having this same problem since I first started using Suricata over a year ago. I've just now come back to trying to solve this.

    On both LAN and WAN interfaces the following command shows only the stale PID errors:

    cat suricata.log | grep "ERRCODE: SC_ERR" |
    grep -v SC_ERR_INVALID_SIGNATURE |
    grep -v SC_ERR_RULE_KEYWORD_UNKNOWN |
    grep -v SC_ERR_REFERENCE_UNKNOWN |
    grep -v SC_ERR_PCRE_PARSE |
    grep -v SC_ERR_UNDEFINED_VAR |
    grep -v SC_ERR_EVENT_ENGINE |
    grep -v SC_ERR_CONFLICTING_RULE_KEYWORDS

    In other words: above are the only visible errors, and all seem to relate to rule updates.

    I am using Snort subscriber rules snortrules-snapshot-2990.tar.gz (so not 3.0). I've disabled those temporarily as of my reading of this thread.

    My stream caps have been extremely high for some time now, and I don't see any memory failures in either syslog or suricata.log.

    I do run on both WAN and LAN, and I vaguely recall you saying in a previous thread that there might be some race condition when it comes to restarting after a rules load? Could that be an issue?

    I have disabled "Live Rule Swap on Update" to no avail.

    Let me know if anything else might help get to the bottom of this and I'll get it for you ASAP.

    Thanks!



  • @boobletins said in Still seeing suricata stop an interface due to .pid error:

    grep -v SC_ERR_UNDEFINED_VAR |
    grep -v SC_ERR_EVENT_ENGINE |

    All your listed errors except these two are due to running Snort rules with Suricata. I would like to have the entire line of error text from the log for these two errors:

    grep -v SC_ERR_UNDEFINED_VAR |
    grep -v SC_ERR_EVENT_ENGINE |
    

    While I don't think it's going to magically fix your problem, the lastest Snort rules snapshot is snortrules-snapshot-29120.tar.gz. That's the file I would suggest running.

    I don't think this is a widespread issue on pfSense. So far you and the OP are the only reports I have out of quite a few Suricata users on pfSense. If you are also seeing the "stale PID error", that indicates Suricata is crashing (likely either a Signal 11 or Signal 10) and thus not cleaning up after itself. During an orderly shutdown, Suricata itself deletes that PID file it is complaining about. Upon startup, it expects that file to be gone as the previousj orderly shutdown should have resulted in it being deleted. If Suricata crashes and is terminated by the OS, then the file is not deleted and thus the error is thrown at the next startup attempt.

    Search your pfSense system log to see if there are any Signal 11 or Signal 10 errors reported for Suricata. Also let me know what kind of WAN interface you have. Is it PPPoE or a standard DHCP or static IP mapping?



  • Thanks for the reply!

    The undefined var errors are uniformly:

    SC_ERR_UNDEFINED_VAR(101)] - Variable "SIP_PORTS" is not defined in configuration file

    The event engine error does not appear on the LAN interface and is uniformly:
    [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2200075, gid 1: unknown rule

    Both interfaces are provisioned using DHCP.

    I assume the errors above could safely be ignored?

    system.log is maybe more interesting--it contains the following signal 11 notifications:

    Nov 18 05:47:56 XXXX kernel: pid 72925 (suricata), uid 0: exited on signal 11 (core dumped)
    Nov 18 05:50:21 XXXX kernel: pid 31920 (suricata), uid 0: exited on signal 11 (core dumped)

    Here's the nearby log with some redaction:

    Nov 18 05:47:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:47:20 XXXX suricata[72925]: [Drop] [1:2402000:5002] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 198.108.67.36:35236 -> X.X.X.X:9955
    Nov 18 05:47:32 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:47:38 XXXX rc.gateway_alarm[21524]: >>> Gateway alarm: WAN_DHCP6 (Addr:IPV6-1%em0 Alarm:1 RTT:132.089ms RTTsd:8.936ms Loss:22%)
    Nov 18 05:47:38 XXXX check_reload_status: updating dyndns WAN_DHCP6
    Nov 18 05:47:38 XXXX check_reload_status: Restarting ipsec tunnels
    Nov 18 05:47:38 XXXX check_reload_status: Restarting OpenVPN tunnels/interfaces
    Nov 18 05:47:38 XXXX check_reload_status: Reloading filter
    Nov 18 05:47:38 XXXX rc.gateway_alarm[23488]: >>> Gateway alarm: WAN_DHCP (Addr:X.X.X.1 Alarm:1 RTT:132.204ms RTTsd:9.258ms Loss:22%)
    Nov 18 05:47:38 XXXX check_reload_status: updating dyndns WAN_DHCP
    Nov 18 05:47:38 XXXX check_reload_status: Restarting ipsec tunnels
    Nov 18 05:47:38 XXXX check_reload_status: Restarting OpenVPN tunnels/interfaces
    Nov 18 05:47:38 XXXX check_reload_status: Reloading filter
    Nov 18 05:47:39 XXXX php-fpm[33153]: /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
    Nov 18 05:47:39 XXXX php-fpm[33153]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
    Nov 18 05:47:39 XXXX php-fpm[37754]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:47:39 XXXX php-fpm[88455]: /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
    Nov 18 05:47:39 XXXX php-fpm[88455]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
    Nov 18 05:47:39 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:47:39 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:47:39 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:47:39 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:47:39 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:47:40 XXXX php-fpm[37754]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:47:40 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:47:40 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:47:40 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:47:40 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:47:40 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:47:50 XXXX kernel: 670.657772 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 99 m 0xfffff80128ee6a00
    Nov 18 05:47:50 XXXX kernel: 670.697930 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff802164f6900
    Nov 18 05:47:50 XXXX kernel: 670.712976 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 108 m 0xfffff802b8d49d00
    Nov 18 05:47:50 XXXX kernel: 670.733007 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 85 m 0xfffff802164f6900
    Nov 18 05:47:50 XXXX kernel: 670.738608 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802164f6900
    Nov 18 05:47:50 XXXX kernel: 670.738924 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8d49d00
    Nov 18 05:47:50 XXXX kernel: 670.738961 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff80128ee6a00
    Nov 18 05:47:50 XXXX kernel: 670.739752 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff802b8d49d00
    Nov 18 05:47:50 XXXX kernel: 670.740041 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8d49d00
    Nov 18 05:47:50 XXXX kernel: 670.741516 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 86 m 0xfffff802164f6900
    Nov 18 05:47:51 XXXX kernel: 671.032906 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff801a05e6d00
    Nov 18 05:47:51 XXXX kernel: 671.038964 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff8000d278b00
    Nov 18 05:47:51 XXXX kernel: 671.043875 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 95 m 0xfffff801a05e6700
    Nov 18 05:47:51 XXXX kernel: 671.095031 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 108 m 0xfffff8000d881900
    Nov 18 05:47:51 XXXX kernel: 671.125010 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 85 m 0xfffff801a05e6d00
    Nov 18 05:47:51 XXXX kernel: 671.125037 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 86 m 0xfffff801a05e6d00
    Nov 18 05:47:51 XXXX kernel: 671.125073 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 111 m 0xfffff801a05e6d00
    Nov 18 05:47:51 XXXX kernel: 671.140838 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff801a05e6700
    Nov 18 05:47:51 XXXX kernel: 671.155689 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 87 m 0xfffff8000d278b00
    Nov 18 05:47:51 XXXX kernel: 671.158230 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff8000d881900
    Nov 18 05:47:52 XXXX kernel: 672.036022 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 96 m 0xfffff802b8c36200
    Nov 18 05:47:52 XXXX kernel: 672.043806 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff801a05e6700
    Nov 18 05:47:52 XXXX kernel: 672.045644 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff802b8c36200
    Nov 18 05:47:52 XXXX kernel: 672.075390 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d978700
    Nov 18 05:47:52 XXXX kernel: 672.075414 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff8000d8e5200
    Nov 18 05:47:52 XXXX kernel: 672.075432 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 82 m 0xfffff80129007300
    Nov 18 05:47:52 XXXX kernel: 672.075451 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff801a0473700
    Nov 18 05:47:52 XXXX kernel: 672.075895 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d8e5200
    Nov 18 05:47:52 XXXX kernel: 672.080115 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 119 m 0xfffff8000d978700
    Nov 18 05:47:52 XXXX kernel: 672.093778 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff8000d978700
    Nov 18 05:47:53 XXXX kernel: 673.012356 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 111 m 0xfffff802b8b2b200
    Nov 18 05:47:53 XXXX kernel: 673.034875 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 87 m 0xfffff801a098b300
    Nov 18 05:47:53 XXXX kernel: 673.039092 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 87 m 0xfffff801a098b300
    Nov 18 05:47:53 XXXX kernel: 673.050711 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff8027f059600
    Nov 18 05:47:53 XXXX kernel: 673.065905 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8984200
    Nov 18 05:47:53 XXXX kernel: 673.065925 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d86f700
    Nov 18 05:47:53 XXXX kernel: 673.065941 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8984200
    Nov 18 05:47:53 XXXX kernel: 673.065958 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff801a098b300
    Nov 18 05:47:53 XXXX kernel: 673.070275 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 105 m 0xfffff8027f059600
    Nov 18 05:47:53 XXXX kernel: 673.080830 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff8027f059600
    Nov 18 05:47:54 XXXX kernel: 674.013828 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff802b8af4b00
    Nov 18 05:47:54 XXXX kernel: 674.015016 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 102 m 0xfffff80128921800
    Nov 18 05:47:54 XXXX kernel: 674.032843 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 118 m 0xfffff802b8af4b00
    Nov 18 05:47:54 XXXX kernel: 674.056702 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff801a098b300
    Nov 18 05:47:54 XXXX kernel: 674.062713 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 74 m 0xfffff8000d863c00
    Nov 18 05:47:54 XXXX kernel: 674.068904 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff80128dbf800
    Nov 18 05:47:54 XXXX kernel: 674.107820 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 88 m 0xfffff802b8af4b00
    Nov 18 05:47:54 XXXX kernel: 674.112807 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 95 m 0xfffff802b8af4b00
    Nov 18 05:47:54 XXXX kernel: 674.128842 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 118 m 0xfffff802b8af4b00
    Nov 18 05:47:54 XXXX kernel: 674.130808 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 115 m 0xfffff802b8af4b00
    Nov 18 05:47:55 XXXX kernel: 675.056754 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff802334fac00
    Nov 18 05:47:55 XXXX kernel: 675.058729 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff802334fac00
    Nov 18 05:47:55 XXXX kernel: 675.059914 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802334fac00
    Nov 18 05:47:55 XXXX kernel: 675.059955 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802334fac00
    Nov 18 05:47:55 XXXX kernel: 675.061751 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff80128e56100
    Nov 18 05:47:55 XXXX kernel: 675.067740 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff80128e56100
    Nov 18 05:47:55 XXXX kernel: 675.074865 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 82 m 0xfffff801cb0f9300
    Nov 18 05:47:55 XXXX kernel: 675.082237 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff80262178d00
    Nov 18 05:47:55 XXXX kernel: 675.082756 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 86 m 0xfffff80262178d00
    Nov 18 05:47:55 XXXX kernel: 675.099097 [2925] netmap_transmit           em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d84ce00
    Nov 18 05:47:56 XXXX kernel: pid 72925 (suricata), uid 0: exited on signal 11 (core dumped)
    Nov 18 05:47:56 XXXX kernel: em0: link state changed to DOWN
    Nov 18 05:47:56 XXXX check_reload_status: Linkup starting em0
    Nov 18 05:47:57 XXXX php-fpm[60994]: /rc.linkup: DEVD Ethernet detached event for wan
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:59 XXXX kernel: em0: link state changed to UP
    Nov 18 05:47:59 XXXX check_reload_status: Linkup starting em0
    Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:48:00 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:48:00 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:48:00 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:48:00 XXXX kernel: cannot forward src fe80:1::3af7:3dff:fe40:9e6b, dst 2600:6c40:500:8ac:230:18ff:fece:19d0, nxt 58, rcvif igb0, outif em0
    Nov 18 05:48:01 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:48:01 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:48:01 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0
    Nov 18 05:48:02 XXXX php-fpm[60994]: /rc.linkup: Shutting down Router Advertisment daemon cleanly
    Nov 18 05:48:02 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: DEVD Ethernet attached event for wan
    Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: HOTPLUG: Configuring interface wan
    Nov 18 05:48:02 XXXX check_reload_status: rc.newwanip starting em0
    Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: calling interface_dhcpv6_configure.
    Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: Accept router advertisements on interface em0
    Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: Starting rtsold process
    Nov 18 05:48:03 XXXX php-fpm[27412]: /rc.newwanip: rc.newwanip: Info: starting on em0.
    Nov 18 05:48:03 XXXX php-fpm[27412]: /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: WAN[wan]) (real interface: em0).
    Nov 18 05:48:03 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:03 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:03 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:03 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:04 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} 192.168.1.90:34325 -> 172.217.4.227:80
    Nov 18 05:48:04 XXXX php-fpm[33153]: /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
    Nov 18 05:48:04 XXXX php-fpm[33153]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
    Nov 18 05:48:04 XXXX check_reload_status: Restarting ipsec tunnels
    Nov 18 05:48:04 XXXX rtsold: Received RA specifying route IPV6-1 for interface wan(em0)
    Nov 18 05:48:04 XXXX rtsold: Starting dhcp6 client for interface wan(em0)
    Nov 18 05:48:04 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:04 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:04 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:04 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:05 XXXX php-fpm[27412]: /rc.newwanip: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0
    Nov 18 05:48:06 XXXX php-fpm[13145]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0.
    Nov 18 05:48:06 XXXX php-fpm[13145]: /rc.newwanipv6: rc.newwanipv6: on (IP address: IPV6.2) (interface: wan) (real interface: em0).
    Nov 18 05:48:07 XXXX php-fpm[13145]: /rc.newwanipv6: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1542541687] unbound[798:0] error: bind: address already in use [1542541687] unbound[798:0] fatal error: could not open ports'
    Nov 18 05:48:07 XXXX php-fpm[33153]: /rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1542541687] unbound[16614:0] error: bind: address already in use [1542541687] unbound[16614:0] fatal error: could not open ports'
    Nov 18 05:48:09 XXXX php-fpm[37754]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'couldn't get address for 'update.dyndns.com': failure syntax error'
    Nov 18 05:48:09 XXXX php-fpm[37754]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X)
    Nov 18 05:48:09 XXXX php-fpm[97376]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'couldn't get address for 'update.dyndns.com': failure syntax error'
    Nov 18 05:48:09 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X)
    Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)'
    Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X)
    Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:10 XXXX php-fpm[13145]: /rc.newwanipv6: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0
    Nov 18 05:48:11 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:11 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
    Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Creating rrd update script
    Nov 18 05:48:12 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:12 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:12 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:12 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:12 XXXX check_reload_status: updating dyndns wan
    Nov 18 05:48:12 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:14 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:14 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:14 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:14 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:14 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:14 XXXX php-fpm[27412]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - X.X.X.X ->  X.X.X.X - Restarting packages.
    Nov 18 05:48:14 XXXX check_reload_status: Starting packages
    Nov 18 05:48:15 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:15 XXXX php-fpm[88455]: /rc.start_packages: Restarting/Starting all packages.
    Nov 18 05:48:15 XXXX php-fpm[88455]: [pfBlockerNG] Starting cron process.
    Nov 18 05:48:15 XXXX SuricataStartup[3518]: Suricata START for WAN(34205_em0)...
    Nov 18 05:48:15 XXXX check_reload_status: Syncing firewall
    Nov 18 05:48:15 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)'
    Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X)
    Nov 18 05:48:17 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:17 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:17 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:17 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] - squid_resync function call pr:1 bp: rpc:no
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Adding cronjobs ...
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Antivirus features disabled.
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Removing freshclam cronjob.
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Stopping any running proxy monitors
    Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Reloading for configuration sync...
    Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Starting a proxy monitor script
    Nov 18 05:48:24 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:09 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X)
    Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)'
    Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X)
    Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:10 XXXX php-fpm[13145]: /rc.newwanipv6: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0
    Nov 18 05:48:11 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:11 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
    Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Creating rrd update script
    Nov 18 05:48:12 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:12 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:12 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:12 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:12 XXXX check_reload_status: updating dyndns wan
    Nov 18 05:48:12 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:14 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:14 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:14 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:14 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:14 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:14 XXXX php-fpm[27412]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - X.X.X.X ->  X.X.X.X - Restarting packages.
    Nov 18 05:48:14 XXXX check_reload_status: Starting packages
    Nov 18 05:48:15 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
    Nov 18 05:48:15 XXXX php-fpm[88455]: /rc.start_packages: Restarting/Starting all packages.
    Nov 18 05:48:15 XXXX php-fpm[88455]: [pfBlockerNG] Starting cron process.
    Nov 18 05:48:15 XXXX SuricataStartup[3518]: Suricata START for WAN(34205_em0)...
    Nov 18 05:48:15 XXXX check_reload_status: Syncing firewall
    Nov 18 05:48:15 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)'
    Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X)
    Nov 18 05:48:17 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:17 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:17 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:17 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] - squid_resync function call pr:1 bp: rpc:no
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Adding cronjobs ...
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Antivirus features disabled.
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Removing freshclam cronjob.
    Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Stopping any running proxy monitors
    Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Reloading for configuration sync...
    Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Starting a proxy monitor script
    Nov 18 05:48:24 XXXX check_reload_status: Reloading filter
    Nov 18 05:48:24 XXXX php-cgi: haproxy: reload old pid:64450
    Nov 18 05:48:24 XXXX php-cgi: haproxy: started new pid:45715
    Nov 18 05:48:26 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:48:26 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:48:26 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:48:26 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:48:32 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:48:47 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:49:02 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:49:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:49:32 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:49:47 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:50:02 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80
    Nov 18 05:50:21 XXXX check_reload_status: Linkup starting igb0
    Nov 18 05:50:21 XXXX kernel: pid 31920 (suricata), uid 0: exited on signal 11 (core dumped)
    Nov 18 05:50:21 XXXX kernel: igb0: link state changed to DOWN
    Nov 18 05:50:22 XXXX php-fpm[37754]: /rc.linkup: DEVD Ethernet detached event for lan
    Nov 18 05:50:22 XXXX check_reload_status: Reloading filter
    Nov 18 05:50:23 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:50:23 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:50:23 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:50:23 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:50:25 XXXX kernel: igb0: link state changed to UP
    Nov 18 05:50:25 XXXX check_reload_status: Linkup starting igb0
    Nov 18 05:50:26 XXXX php-fpm[27412]: /rc.linkup: DEVD Ethernet attached event for lan
    Nov 18 05:50:26 XXXX php-fpm[27412]: /rc.linkup: HOTPLUG: Configuring interface lan
    Nov 18 05:50:26 XXXX check_reload_status: Restarting ipsec tunnels
    Nov 18 05:50:26 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0.
    Nov 18 05:50:26 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: No IPv6 address found for interface WAN [wan].
    Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0.
    Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: No IPv6 address found for interface WAN [wan].
    Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0.
    Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: on (IP address: IPV6.2) (interface: wan) (real interface: em0).
    Nov 18 05:50:29 XXXX php-fpm[27412]: /rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1542541829] unbound[2124:0] error: bind: address already in use [1542541829] unbound[2124:0] fatal error: could not open ports'
    Nov 18 05:50:30 XXXX php-fpm[60994]: /rc.newwanipv6: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0
    Nov 18 05:50:30 XXXX check_reload_status: Reloading filter
    Nov 18 05:50:32 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:50:32 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:50:32 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:50:32 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 05:50:32 XXXX check_reload_status: updating dyndns lan
    Nov 18 05:50:32 XXXX check_reload_status: Reloading filter
    Nov 18 05:50:33 XXXX xinetd[88961]: Starting reconfiguration
    Nov 18 05:50:33 XXXX xinetd[88961]: Swapping defaults
    Nov 18 05:50:33 XXXX xinetd[88961]: readjusting service 19000-tcp
    Nov 18 05:50:33 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 18 06:05:04 XXXX php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules are up to date...
    Nov 18 06:05:16 XXXX php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules are up to date...
    Nov 18 06:05:16 XXXX php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
    Nov 18 06:05:16 XXXX check_reload_status: Syncing firewall
    Nov 18 06:15:07 XXXX php: [pfBlockerNG] Starting cron process.
    

    It looks like both interfaces are going down for new IPs and Suricata is crashing shortly thereafter as the netmap queue fills up? Maybe? heh

    Or the interface is failing with high packet loss, resetting, and Suricata crashes?



  • @boobletins

    You are running Snort and Suricata on the same box? That is certainly not recommended. They will potentially step on each other, especially when using Legacy Mode. I do see you appear to be using Inline IPS mode for Suricata, but running both Snort and Suricata on the same box is not a good decision. Choose only one of the IDS/IPS systems to use. Do not use both on the same box.

    Your second issue is something appears to be causing your interface to flap. Each time that happens the internal pfSense system will restart all the packages. That's how Suricata (and Snort) can collide with multiple copies of themselves, especially if one startup is still happening and then pfSense comes along and issues another closely spaced "restart all packages" command in response to an interface down/up cycle. I'm not positive that's what is happening in your case, but it appears from the log data your interface cycled at least twice in a relatively short period.



  • I started with Snort and switched to Suricata ~1 year ago. Snort has no configured interfaces, but it looks like it was still updating rules (I didn't notice they were still running until you pointed it out). I'll remove the pkg entirely if that might be causing an issue.

    I agree it looks like the interfaces are cycling. It looks like both LAN igb0 and WAN em0 cycled around the same time. It looks to me like rc.gateway_alarm detects 22% packet loss and cycles the interfaces assuming something went wrong.

    I'll try to investigate what's going on with the interfaces, but in the meantime, would it be an acceptable solution to modify rc.start_packages (or similar) to simply rm *.pid in the appropriate suricata directories? Or would that be dangerous in ways I don't understand?



  • @boobletins said in Still seeing suricata stop an interface due to .pid error:

    I started with Snort and switched to Suricata ~1 year ago. Snort has no configured interfaces, but it looks like it was still updating rules (I didn't notice they were still running until you pointed it out). I'll remove the pkg entirely if that might be causing an issue.

    I agree it looks like the interfaces are cycling. It looks like both LAN igb0 and WAN em0 cycled around the same time. It looks to me like rc.gateway_alarm detects 22% packet loss and cycles the interfaces assuming something went wrong.

    I'll try to investigate what's going on with the interfaces, but in the meantime, would it be an acceptable solution to modify rc.start_packages (or similar) to simply rm *.pid in the appropriate suricata directories? Or would that be dangerous in ways I don't understand?

    No, no danger at all in deleting the file. But that is just going to be masking the problem. The file being there is a symptom caused by something crashing Suricata.

    I would try changing whatever host you are "pinging" for the gateway monitoring. Maybe that host is tardy responding to ICMP requests or even drops them when it gets busy.



  • @bmeeks

    PM you the log file....it's way to big to post here.

    Thanks bmeeks.



  • @val said in Still seeing suricata stop an interface due to .pid error:

    @bmeeks

    PM you the log file....it's way to big to post here.

    Thanks bmeeks.

    I looked through you log file. What version of the Snort Rules Snapshot file are you using? You should be using only rules packages for Snort 2.9.x if you are running Snort rules with Suricata. Your file name should be snortrules-snapshot-29120.tar.gz. Do not use the Snort3 rules (that means do not use any Snort rules file with 3 in the name). You should not be seeing those "unknown reference" error messages. The only time I've noticed those is when the user has downloaded the rules meant for use only with the new Snort3 beta package from the Snort team.



  • @bmeeks

    The version I am using and file name it's:-
    snortrules-snapshot-29111.tar.gz

    Thanks for that info I will change it to see if process still kill it self.



  • Just want to add I have been having the same issue with the Suricata .pid file becoming stale, and the engine failing to restart because of this after a crash. I am using SG-3100. I also run pfBlocker. I notice when I am tweaking setting within here, most of the time, that's when Suricata crashes. Having to manually rm the .pid file to start Suricata.



  • @bhjitsense said in Still seeing suricata stop an interface due to .pid error:

    Just want to add I have been having the same issue with the Suricata .pid file becoming stale, and the engine failing to restart because of this after a crash. I am using SG-3100. I also run pfBlocker. I notice when I am tweaking setting within here, most of the time, that's when Suricata crashes. Having to manually rm the .pid file to start Suricata.

    The SG-3100 crash is due to a compiler optimization problem for armv6 and armv7 CPUs (like those used in the SG-1000 and SG-3100 appliances). I've been in contact with the pfSense team about this, but so far there is no resolution posted. The only fix for now is to NOT run Suricata on SG-3100 hardware. If you do, it will continue to randomly crash with the Signal 10 Bus Error. The Signal 10 crash leaves the PID file in place, so the next time you attempt to start Suricata it will see the file remaining from the previously crashed instance and complain. The stale PID file is a symptom and not a cause in this case.

    You can research Google for what Signal 10 Bus Errors are and what causes them. It is due to the clang/llvm compiler generating machine opcodes that do not support unaligned memory access on arm processors.



  • @bmeeks said in Still seeing suricata stop an interface due to .pid error:

    The only fix for now is to NOT run Suricata on SG-3100 hardware.

    Hmm, interesting. I checked the router I had posted about a week ago, and Suricata is still running. Is the crash random or only when making changes as @bhjitsense suggested might be the case? (I have made no changes in the past week)



  • @teamits said in Still seeing suricata stop an interface due to .pid error:

    @bmeeks said in Still seeing suricata stop an interface due to .pid error:

    The only fix for now is to NOT run Suricata on SG-3100 hardware.

    Hmm, interesting. I checked the router I had posted about a week ago, and Suricata is still running. Is the crash random or only when making changes as @bhjitsense suggested might be the case? (I have made no changes in the past week)

    It will appear to be somewhat random, although in fact each and every time a particular piece of code is hit (the part with the opcodes I mentioned) the error will be thrown. The randomness comes in from the fact that the problem code might be part of an if-then statement in code where one piece of code is executed if a tested condition is true while a different section of code is executed if the condition tests as false. I have no idea where specifically in the Suricata code the problem lies. It very well could exist in several places.

    This is not a problem on Intel hardware because all Intel CPUs will automatically fix-up and execute data loads or stores to unaligned addresses. Intel CPUs do this by default with the processor hardware. Since developers mostly target Intel hardware, they have all gotten quite complacent and sloppy with data access via structures and pointer casting in C programming. Intel hardware will cover for their sloppiness, but other hardware (like the armv6 and armv7 CPU) is not as forgiving. Of course part of the fault here also lies with the clang/llvm compiler used to produce the binary code for the arm hardware.

    I also want to make clear this is an issue within the acutal Suricata binary code and has nothing at all to do with the PHP GUI code of pfSense. Remember all that the GUI packages for Suricata and Snort do on pfSense is provide a wrapper to let users easily create the text configuration files the underlying binaries need to run.



  • Thanks for the explanation. I meant to ask, is this issue with the latest release version of the Suricata package/binaries or prior versions also?



  • @teamits said in Still seeing suricata stop an interface due to .pid error:

    Thanks for the explanation. I meant to ask, is this issue with the latest release version of the Suricata package/binaries or prior versions also?

    It can very well exist in any previous version, but appears to have reared its head in the latest binary update. The explanation of what unaligned access is and how it happens in C programming code is a bit long-winded and requires a good bit of hardware understanding (things like memory bus widths, CPU register loads/stores and how memory access works at a hardware level) in order to fully grasp the concept. You can Google "unaligned memory access" and start your research if interested.

    It could be that a simple code change that happened in the latest binary (maybe fixing some other bug) caused this issue on arm hardware. Key to understanding the impact of unaligned access issues is also understanding what role compilers play in producing the acutal binary instruction codes from the higher-level C programming code. On FreeBSD, the compiler used for this is clang/llvm. Linux traditionally uses gcc. Of course Windows uses Microsoft-supplied compilers. Each compiler will produce slightly different binary code from the exact same C high-level code. On FreeBSD, that compiler produces a poorly chosen set of opcodes for certain memory access operations. It chooses to use a pair of opcodes that the arm CPU cannot fix-up on the fly for unaligned access. There are other opcodes that can perform the same operation and the arm CPU can auto fix-up those memory accesses to prevent the unaligned access. Research LDM/STM and LDR/STR instruction opcodes on armv6 and armv7 microprocessors to see what I mean.

    EDIT: up above, when I say "poorly chosen set of opcodes" I mean it chooses speed over compatibility. The opcodes it chooses to use (LDM/STM) do execute ever so slightly faster than their counterparts (LDR/STR), but the latter codes support unaligned memory access without crashing on the Signal 10 Bus Error. The former LDM/STM instructions will crash if the original Suricata binary source code programmer inadvertently asked the CPU to load or store a piece of data that causes an unaligned memory access.

    There are also frequent battles of words between compiler developers and other C programmers. The compiler gurus say the C programmers should not be so sloppy and make sure to avoid unaligned access issues, but the C programmers retaliate with the old "well it works on Intel without issue" or "it works fine on gcc", etc. So that can mean a lot of posturing on each side with nothing useful really happening. There is reluctance for say the Suricata team to dive into this because it is only affecting Suricata running on arm hardware where Suricata was compiled by clang/llvm. That's a pretty small footprint of users compared to Linux land. Thus no incentive for the Suricata developers to spend hours trying to find where the issue is.



  • @bmeeks

    !@#%$(*!@#^%

    I had a whole post written up describing how to solve this on my igb0 card and the forum thought it was spam and poof. Extremely frustrating.

    IPV6 TX Checksums are not disabled via the GUI when they appear to be. Command line ifconfig can fix this issue (until a reboot). I can reliably reproduce this on my igb interface, haven't tested the em yet.

    Hope this helps someone? If it lets me post..........

    Edit: I give up, PMing you Bill... this is absurd.



  • @boobletins said in Still seeing suricata stop an interface due to .pid error:

    @bmeeks

    !@#%$(*!@#^%

    I had a whole post written up describing how to solve this on my igb0 card and the forum thought it was spam and poof. Extremely frustrating.

    IPV6 TX Checksums are not disabled via the GUI when they appear to be. Command line ifconfig can fix this issue (until a reboot). I can reliably reproduce this on my igb interface, haven't tested the em yet.

    Hope this helps someone? If it lets me post..........

    Edit: I give up, PMing you Bill... this is absurd.

    Replied to your PM.



  • @boobletins said in Still seeing suricata stop an interface due to .pid error:

    @bmeeks

    !@#%$(*!@#^%

    I had a whole post written up describing how to solve this on my igb0 card and the forum thought it was spam and poof. Extremely frustrating.

    IPV6 TX Checksums are not disabled via the GUI when they appear to be. Command line ifconfig can fix this issue (until a reboot). I can reliably reproduce this on my igb interface, haven't tested the em yet.

    Hope this helps someone? If it lets me post..........

    Edit: I give up, PMing you Bill... this is absurd.

    I'm not 100% positive this is the cause of the Signal 10 error. That error definitely indicates an unaligned memory access. Now it is possible that the checksum failures may be leading to that "particular section" of bad opcodes I mentioned in my earlier post getting executed and thus throwing the Signal 10 error.