Suricata - bans LAN device -new behavior on new pf install

bmeeks

@MaxBishop:
I'm copying and pasting in some instructions from a different thread. Follow the steps below to generate a special debug log.

There is a hidden passlist debug switch that will print a log file of Pass List logic actions. Here is how to enable it:

Go to the DIAGNOSTICS > EDIT FILE menu and browse to and edit the following file: /usr/local/pkg/suricata/suricata_yaml_template.inc.
Find this section in the file (it's near the top):

# alert-pf custom blocking plugin for pfSense only
- alert-pf:
    enabled: {$suri_blockoffenders}
    kill-state: {$suri_killstates}
    block-drops-only: {$suri_blockdrops}
    pass-list: {$suri_passlist}
    block-ip: {$suri_blockip}
    pf-table: {$suri_pf_table}
    passlist-debugging: no   # Do not enable debugging on production systems!

Change the passlist-debugging: no line to read passlist-debugging: yes and save the edit. Do not change anything else!

Return to the Suricata INTERFACES tab and edit the interface where you have Pass List entries being blocked. On the INTERFACE EDIT page, click the Save button to regenerate the suricata.yaml file for the interface. No need to change anything on the edit page, just click Save and that will regenerate the suricata.yaml conf file.

Go to the INTERFACES tab in Suricata and restart the interface you edited so that it will see the new suricata.yaml file.

You will then find a new passlist debug text log in the Suricata logging directory for the interface. That will be /var/log/suricata/suricata_xxxx_yyyy where xxxx and yyyy are the physical interface name and a random UUID.

On a busy network this log file will start to grow quite large over time, so keep an eye on it and rotate it if necessary. Also, Suricata will be a little less performant - hence the warning about not enabling the debug option on production systems. But running for a short period of time to test is not going to cause a large problem.

When you notice a blocked IP that should have been covered by a Pass List entry and not blocked, look through the passlist debug log to see what is recorded there for the IP in question. Post the results back here as well.

WARNING: it is likely the log file will eventually grow too big to view in the pfSense GUI. You will need a method to transfer a copy off to a PC for viewing. WinSCP is an excellent free Windows application for doing this.

To return to normal non-debug mode for the Pass List, simply repeat the steps except change the passlist-debugging: yes to passlist-debugging: no.

SteveITS

@MaxBishop See if checksum offloading is disabled:

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

I can confirm my issue is now resolved. Disabling the hardware checksum offloading did the trick, as unlikely or inexplicable as a solution as it may be. All my interfaces have now has alerts that only blocked the external IP. The IP listed in the default pass list was not blocked.

MaxBishop

@bmeeks

Followed your instructions. I then forced a block by changing a simple rule from an alert to a block. I chose the MS Metadata UA because it's a frequent alert.

From an ssh terminal:
cat /var/log/suricata/suricata_em061146/block.log

Starting a Windows LAN device (192.168.1.12) instantly created the following log entry (and booted the machine off the LAN).

01/08/2024-18:15:28.187537 [Block Src] [] [1:2027390:4] ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent [] [Classification: Misc activity] [Priority: 3] {TCP} 192.168.1.12:49687
01/08/2024-18:15:28.187537 [Block Dst] [] [1:2027390:4] ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent [] [Classification: Misc activity] [Priority: 3] {TCP} 23.220.138.208:80

(checksum offloading is disabled)

bmeeks

@MaxBishop:
I don't need to see the block log. We know it's being blocked by what you said previously. I didn't say I did not believe you . I said I wanted to see the output of the logs related to passlist operation on the box.

What I want to see is the full output from the passlist_debug.log file which should exist now if you followed the steps outlined previously. That file will contain all the steps the custom blocking module took as it processed that packet. It can be found in the location I mentioned earlier:

/var/log/suricata/suricata_xxxxyyyy/

where xxxx is the physical interface name and yyyy will be a UUID.

Please also post the contents of the suricata.log file for the interface. That will show all the steps Suricata was taking during startup, and it will show other actions related to the pass list operation. You can view this log on the LOGS VIEW tab in the GUI.

bmeeks

Two more questions for clarity:

Is this a fresh install of pfSense 2.7.2, or is it an older version that you had an image for?

Is the Suricata package version 7.0.2_3?

MaxBishop

@bmeeks

Stupid me, I posted the wrong file.

cat passlist_debug.log 
01/08/2024-18:07:43.441161  Pass List debugging enabled. Processing file: /usr/local/etc/suricata/suricata_61146_em0/passlist.
01/08/2024-18:07:43.441206  Added IPv4 address 8.8.8.8/32 from Pass List.
01/08/2024-18:07:43.441218  IPv4 address 10.10.10.1/32 from Pass List exactly matches an existing entry, so not adding it again.
01/08/2024-18:07:43.441220  IPv4 address 127.0.0.1/32 from Pass List exactly matches an existing entry, so not adding it again.
01/08/2024-18:07:43.441221  Added IPv4 netblock 192.168.1.0/24 to IPv4 Radix Tree created from Pass List entry 192.168.1.0/24.
01/08/2024-18:07:43.441224  Added IPv4 address 192.168.2.1/32 from Pass List.
01/08/2024-18:07:43.441229  Added IPv4 address 216.146.35.35/32 from Pass List.
01/08/2024-18:07:43.441236  Completed processing Pass List /usr/local/etc/suricata/suricata_61146_em0/passlist. Total entries parsed: 6, Unique IP addresses/netblocks/aliases added to Radix Trees: 4, IP addresses/netblocks ignored because they were covered by existing Radix Tree entries: 2.

01/08/2024-18:15:28.187537  Thread: W#11  SRC IP: 192.168.1.12 did not match any Pass List entry, so adding to block list.
01/08/2024-18:15:28.348983  Thread: W#11  Successfully added IP: 192.168.1.12 to pf table snort2c for blocking.
01/08/2024-18:15:28.525927  Thread: W#11  Successfully killed any open states for IP: 192.168.1.12, so any stateful traffic is blocked.
01/08/2024-18:15:28.187537  Thread: W#11  DST IP: 23.220.138.208 did not match any Pass List entry, so adding to block list.
01/08/2024-18:15:28.525954  Thread: W#11  Successfully added IP: 23.220.138.208 to pf table snort2c for blocking.
01/08/2024-18:15:28.702605  Thread: W#11  Successfully killed any open states for IP: 23.220.138.208, so any stateful traffic is blocked.

Here's HOME_NET:
8.8.8.8/32
10.10.10.1/32
127.0.0.1/32
192.168.1.0/24
192.168.2.1/32
192.168.2.20/32
216.146.35.35/32
::1/128
fe80::21b:21ff:fe63:fbb9/128
fe80::21b:21ff:feee:a5bf/128

This is a fresh install from pfSense-CE-memstick-2.7.2-RELEASE-amd64 with the Suricata 7.0.2_3 package version.

bmeeks

@MaxBishop said in Suricata - bans LAN device -new behavior on new pf install:

@bmeeks

Stupid me, I posted the wrong file.

cat passlist_debug.log 
01/08/2024-18:07:43.441161  Pass List debugging enabled. Processing file: /usr/local/etc/suricata/suricata_61146_em0/passlist.
01/08/2024-18:07:43.441206  Added IPv4 address 8.8.8.8/32 from Pass List.
01/08/2024-18:07:43.441218  IPv4 address 10.10.10.1/32 from Pass List exactly matches an existing entry, so not adding it again.
01/08/2024-18:07:43.441220  IPv4 address 127.0.0.1/32 from Pass List exactly matches an existing entry, so not adding it again.
01/08/2024-18:07:43.441221  Added IPv4 netblock 192.168.1.0/24 to IPv4 Radix Tree created from Pass List entry 192.168.1.0/24.
01/08/2024-18:07:43.441224  Added IPv4 address 192.168.2.1/32 from Pass List.
01/08/2024-18:07:43.441229  Added IPv4 address 216.146.35.35/32 from Pass List.
01/08/2024-18:07:43.441236  Completed processing Pass List /usr/local/etc/suricata/suricata_61146_em0/passlist. Total entries parsed: 6, Unique IP addresses/netblocks/aliases added to Radix Trees: 4, IP addresses/netblocks ignored because they were covered by existing Radix Tree entries: 2.

01/08/2024-18:15:28.187537  Thread: W#11  SRC IP: 192.168.1.12 did not match any Pass List entry, so adding to block list.
01/08/2024-18:15:28.348983  Thread: W#11  Successfully added IP: 192.168.1.12 to pf table snort2c for blocking.
01/08/2024-18:15:28.525927  Thread: W#11  Successfully killed any open states for IP: 192.168.1.12, so any stateful traffic is blocked.
01/08/2024-18:15:28.187537  Thread: W#11  DST IP: 23.220.138.208 did not match any Pass List entry, so adding to block list.
01/08/2024-18:15:28.525954  Thread: W#11  Successfully added IP: 23.220.138.208 to pf table snort2c for blocking.
01/08/2024-18:15:28.702605  Thread: W#11  Successfully killed any open states for IP: 23.220.138.208, so any stateful traffic is blocked.

Here's HOME_NET:
8.8.8.8/32
10.10.10.1/32
127.0.0.1/32
192.168.1.0/24
192.168.2.1/32
192.168.2.20/32
216.146.35.35/32
::1/128
fe80::21b:21ff:fe63:fbb9/128
fe80::21b:21ff:feee:a5bf/128

This is a fresh install from pfSense-CE-memstick-2.7.2-RELEASE-amd64 with the Suricata 7.0.2_3 package version.

Thank you for this. Can you also post the content of the suricata.log file? You can view it under the LOGS VIEW tab and simply copy and paste the content into a post here. You can obfuscate your public WAN IP if desired for privacy.

It's obvious from the above the code thought the IP address was not in the Radix Tree used by the pass list logic. It should actually be there because the initial startup lines show the netblock being added.

What type of hardware are you running? It appears you have a large multicore CPU as I see the block was logged by thread W#11. That would indicate maybe 12 or more packet processing threads. Wondering if this is somehow thread related ???

One other question: is the new hardware different in terms of the number of CPU cores?

I will dig into this issue some more, but the suricata.log file will be helpful if you can post that.

MaxBishop

@bmeeks

Hi,

First, Thank you for your effort.

Yup, it's a 12-core, AMD Ryzen processor.

The computers I use at home are ones I have cycled out of production at work. Typically this happens when a lab instrument is replaced. The original machine here was probably a similar build. The machines I build are almost always 12 or 16-core Ryzen-based machines. I have another PfSense unit at a home office at a different location with essentially the same processor and memory. I might try importing its config on the troublesome machine.

I should also add that here I have a separate perimeter router between the PfSense machine and the internet. This perimeter router is on subnet 192.168.2.0/24 and provides a static address for the PfSense WAN interface.

Here's the suricata log:

[102949 - Suricata-Main] 2024-01-09 08:55:35 Notice: suricata: This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
[102949 - Suricata-Main] 2024-01-09 08:55:35 Info: cpu: CPUs/cores online: 12
[102949 - Suricata-Main] 2024-01-09 08:55:35 Info: suricata: Setting engine mode to IDS mode by default
[102949 - Suricata-Main] 2024-01-09 08:55:35 Info: app-layer-htp-mem: HTTP memcap: 67108864
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Creating automatic firewall interface IP address Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface igb0 IPv6 address fe80:0000:0000:0000:021b:21ff:feee:a5bf to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface igb0 IPv4 address 192.168.2.20 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface em0 IPv6 address fe80:0000:0000:0000:021b:21ff:fe63:fbb9 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface em0 IPv4 address 192.168.1.1 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface em0 IPv6 address fe80:0000:0000:0000:0000:0000:0001:0001 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface em0 IPv4 address 10.10.10.1 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: logopenfile: alert-pf output device (regular) initialized: block.log
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_61146_em0/passlist.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Pass List /usr/local/etc/suricata/suricata_61146_em0/passlist processed: Total entries parsed: 6, IP addresses/netblocks/aliases added to No Block list: 4, IP addresses/netblocks ignored because they were covered by existing entries: 2.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c  block-ip=both  kill-state=yes  block-drops-only=yes  passlist-debugging=no
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: alert-pf: Created Interface IP Address change monitoring thread for auto-whitelisting of firewall interface IP addresses.
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: logopenfile: fast output device (regular) initialized: alerts.log
[101180 - Suricata-Main] 2024-01-09 08:55:36 Info: logopenfile: http-log output device (regular) initialized: http.log
[103465 - Suricata-IM#01] 2024-01-09 08:55:36 Info: alert-pf: Firewall Interface IP Address Change Monitor Thread IM#01 has successfully started.
[101180 - Suricata-Main] 2024-01-09 08:55:37 Error: detect-tls-ja3-hash: ja3 support is not enabled
[101180 - Suricata-Main] 2024-01-09 08:55:37 Error: detect: error parsing signature "alert tls $HOME_NET any -> $EXTERNAL_NET any (msg:"ET JA3 Hash - Suspected Cobalt Strike Malleable C2 M1 (set)"; flow:established,to_server; ja3.hash; content:"eb88d0b3e1961a0562f006e5ce2a0b87"; ja3.string; content:"771,49192-49191-49172-49171"; flowbits:set,ET.cobaltstrike.ja3; flowbits:noalert; classtype:command-and-control; sid:2028831; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2019_10_15, deployment Perimeter, former_category JA3, malware_family Cobalt_Strike, confidence Low, signature_severity Major, updated_at 2019_10_15, mitre_tactic_id TA0011, mitre_tactic_name Command_And_Control, mitre_technique_id T1001, mitre_technique_name Data_Obfuscation;)" from file /usr/local/etc/suricata/suricata_61146_em0/rules/suricata.rules at line 5296
[101180 - Suricata-Main] 2024-01-09 08:55:37 Error: detect-tls-ja3s-hash: ja3(s) support is not enabled
[101180 - Suricata-Main] 2024-01-09 08:55:37 Error: detect: error parsing signature "alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"ET JA3 HASH - Possible RustyBuer Server Response"; flowbits:isset,ET.rustybuer; ja3s.hash; content:"f6dfdd25d1522e4e1c7cd09bd37ce619"; reference:md5,ea98a9d6ca6f5b2a0820303a1d327593; classtype:bad-unknown; sid:2032960; rev:1; metadata:attack_target Client_Endpoint, created_at 2021_05_13, deployment Perimeter, former_category JA3, malware_family RustyBuer, performance_impact Low, confidence Low, signature_severity Major, updated_at 2021_05_13;)" from file /usr/local/etc/suricata/suricata_61146_em0/rules/suricata.rules at line 5392
[101180 - Suricata-Main] 2024-01-09 08:55:43 Info: detect: 2 rule files processed. 35559 rules successfully loaded, 107 rules failed
[101180 - Suricata-Main] 2024-01-09 08:55:43 Warning: threshold-config: can't suppress sid 2012758, gid 1: unknown rule
[101180 - Suricata-Main] 2024-01-09 08:55:43 Warning: threshold-config: can't suppress sid 2042687, gid 1: unknown rule
[101180 - Suricata-Main] 2024-01-09 08:55:43 Warning: threshold-config: can't suppress sid 2042360, gid 1: unknown rule
[101180 - Suricata-Main] 2024-01-09 08:55:43 Info: threshold-config: Threshold config parsed: 12 rule(s) found
[101180 - Suricata-Main] 2024-01-09 08:55:43 Info: detect: 35562 signatures processed. 1362 are IP-only rules, 4040 are inspecting packet payload, 29714 inspect application layer, 108 are decoder event only
[101180 - Suricata-Main] 2024-01-09 08:55:43 Warning: detect-flowbits: flowbit 'file.xls&file.ole' is checked but not set. Checked in 30990 and 0 other sigs
[101180 - Suricata-Main] 2024-01-09 08:55:56 Info: runmodes: Using 1 live device(s).
[103466 - RX#01-em0] 2024-01-09 08:55:57 Info: pcap: em0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[103466 - RX#01-em0] 2024-01-09 08:55:57 Info: pcap: em0: snaplen set to 1518
[101180 - Suricata-Main] 2024-01-09 08:55:57 Notice: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1   Engine started.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Deleted address 192.168.1.1 from automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Deleted address 192.168.1.1 from automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Deleted address fe80:0000:0000:0000:0000:0000:0001:0001 from automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Deleted address fe80:0000:0000:0000:0000:0000:0001:0001 from automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Deleted address 10.10.10.1 from automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Deleted address 10.10.10.1 from automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Added address 192.168.1.1 to automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Added address 192.168.1.1 to automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Added address fe80:0000:0000:0000:0000:0000:0001:0001 to automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Added address fe80:0000:0000:0000:0000:0000:0001:0001 to automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Added address 10.10.10.1 to automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:01 Info: alert-pf: Added address 10.10.10.1 to automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Deleted address 192.168.1.1 from automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Deleted address fe80:0000:0000:0000:0000:0000:0001:0001 from automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Deleted address fe80:0000:0000:0000:0000:0000:0001:0001 from automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Deleted address 10.10.10.1 from automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Deleted address 10.10.10.1 from automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Added address 192.168.1.1 to automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Added address 192.168.1.1 to automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Added address fe80:0000:0000:0000:0000:0000:0001:0001 to automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Added address fe80:0000:0000:0000:0000:0000:0001:0001 to automatic firewall interface IP Pass List.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface em0.
[102109 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Added address 10.10.10.1 to automatic firewall interface IP Pass List.
[103465 - Suricata-IM#01] 2024-01-09 08:56:06 Info: alert-pf: Added address 10.10.10.1 to automatic firewall interface IP Pass List.
[103466 - RX#01-em0] 2024-01-09 08:56:42 Info: checksum: Less than 1/10th of packets have an invalid checksum, assuming checksum offloading is NOT used (29/1000)
[101180 - Suricata-Main] 2024-01-09 09:00:20 Notice: detect: rule reload starting
[101180 - Suricata-Main] 2024-01-09 09:00:20 Info: conf-yaml-loader: Configuration node 'filetype' redefined.
[101180 - Suricata-Main] 2024-01-09 09:00:21 Error: detect-tls-ja3-hash: ja3 support is not enabled
[101180 - Suricata-Main] 2024-01-09 09:00:21 Error: detect: error parsing signature "alert tls $HOME_NET any -> $EXTERNAL_NET any (msg:"ET JA3 Hash - Suspected Cobalt Strike Malleable C2 M1 (set)"; flow:established,to_server; ja3.hash; content:"eb88d0b3e1961a0562f006e5ce2a0b87"; ja3.string; content:"771,49192-49191-49172-49171"; flowbits:set,ET.cobaltstrike.ja3; flowbits:noalert; classtype:command-and-control; sid:2028831; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2019_10_15, deployment Perimeter, former_category JA3, malware_family Cobalt_Strike, confidence Low, signature_severity Major, updated_at 2019_10_15, mitre_tactic_id TA0011, mitre_tactic_name Command_And_Control, mitre_technique_id T1001, mitre_technique_name Data_Obfuscation;)" from file /usr/local/etc/suricata/suricata_61146_em0/rules/suricata.rules at line 5296
[101180 - Suricata-Main] 2024-01-09 09:00:21 Error: detect-tls-ja3s-hash: ja3(s) support is not enabled
[101180 - Suricata-Main] 2024-01-09 09:00:21 Error: detect: error parsing signature "alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"ET JA3 HASH - Possible RustyBuer Server Response"; flowbits:isset,ET.rustybuer; ja3s.hash; content:"f6dfdd25d1522e4e1c7cd09bd37ce619"; reference:md5,ea98a9d6ca6f5b2a0820303a1d327593; classtype:bad-unknown; sid:2032960; rev:1; metadata:attack_target Client_Endpoint, created_at 2021_05_13, deployment Perimeter, former_category JA3, malware_family RustyBuer, performance_impact Low, confidence Low, signature_severity Major, updated_at 2021_05_13;)" from file /usr/local/etc/suricata/suricata_61146_em0/rules/suricata.rules at line 5392
[101180 - Suricata-Main] 2024-01-09 09:00:27 Info: detect: 2 rule files processed. 35559 rules successfully loaded, 107 rules failed
[101180 - Suricata-Main] 2024-01-09 09:00:27 Warning: threshold-config: can't suppress sid 2012758, gid 1: unknown rule
[101180 - Suricata-Main] 2024-01-09 09:00:27 Warning: threshold-config: can't suppress sid 2042687, gid 1: unknown rule
[101180 - Suricata-Main] 2024-01-09 09:00:27 Warning: threshold-config: can't suppress sid 2042360, gid 1: unknown rule
[101180 - Suricata-Main] 2024-01-09 09:00:27 Info: threshold-config: Threshold config parsed: 12 rule(s) found
[101180 - Suricata-Main] 2024-01-09 09:00:27 Info: detect: 35562 signatures processed. 1362 are IP-only rules, 4040 are inspecting packet payload, 29714 inspect application layer, 108 are decoder event only
[101180 - Suricata-Main] 2024-01-09 09:00:27 Warning: detect-flowbits: flowbit 'file.xls&file.ole' is checked but not set. Checked in 30990 and 0 other sigs
[101180 - Suricata-Main] 2024-01-09 09:00:40 Notice: detect: rule reload complete

code_text

bmeeks

@MaxBishop:
Thank you for the additional suricata.log file. I'm currently looking into this issue some more. It has been a long-term random problem. Some users have been hit with it while others are not. I have only seen it happen exactly once in my test environment (and I could never reproduce it again).

Running some diagnostic tests using TSAN (the LLVM threads sanitizer). I'm thinking this issue is likely thread related as it is random in nature (affecting some and not others). A hard-fault in the logic would impact everyone the same way. A threading problem might be data dependent or hardware dependent (more CPU cores equals more threads equals a higher incidence of the problem).

MaxBishop

@bmeeks

A few more tests.

I went to my home office and downloaded the configuration then uploaded the configuration to the new machine at home. No change.

Next, I configured Suricata not to save settings on package removal then removed Suricata. After a fresh Suricata install the issue persists.

Finally, I swapped out the network cards. Sill has issue.

I am leaning towards hardware too. Later this week I will update the BIOS again and configure it to the defaults.

bmeeks

@MaxBishop said in Suricata - bans LAN device -new behavior on new pf install:

I am leaning towards hardware too. Later this week I will update the BIOS again and configure it to the defaults.

I'm not saying hardware is the direct cause of the problem. I was just wondering if the CPU core count was much higher. More cores equals more threads active.

But after scrutinizing the code for a very long time today, I've not found any obvious issue. I'm using read/write mutex locks to control access to the Radix Trees that contain the Pass List IP information. That should prevent any sort of multithreaded race conditions.

I have made a change in the order of how a couple of steps are called when setting up the Radix Trees during Suricata initialization. If you are willing, I would like to provide a test Suricata binary component for you to install and see if it works any better.

MaxBishop

@bmeeks

OK, how do I get it.

bmeeks

@MaxBishop said in Suricata - bans LAN device -new behavior on new pf install:

@bmeeks

OK, how do I get it.

Here is the link: https://drive.google.com/file/d/1L-rCf8rF-_C93TFISOx4iWPRgW95sFww/view?usp=sharing

This will pull from my Google Drive folder.

Here are the instructions for installing (and then later removing) the test binary.

To begin, download the suricata-7.0.2_7.pkg file from the link above and transfer it to your firewall placing it in the /root directory. IMPORTANT: make sure you transfer the file in binary (unaltered) form! So, if using WinSCP for the transfer from a Windows PC, choose "Binary" for the transfer type.
Stop all running Suricata instances by executing this command from a shell prompt on the firewall:

/usr/local/etc/rc.d/suricata.sh stop

Install the updated version of the Suricata binary using the command below at a shell prompt on the firewall:

pkg-static install -f /root/suricata-7.0.2_7.pkg

That command forcibly updates the binary portion of Suricata with a new package leaving the GUI portion unaltered.

Return to the pfSense GUI and restart Suricata on the interfaces using the icons on the INTERFACES tab.

Report back if there is any change in behavior. I sort of don't really expect a change, but maybe we get lucky. This has proven to be an extraordinarily difficult nut to crack in the past (evidenced by the fact I still have not found a true root cause and thus effective solution). Not being able to reproduce it on my end is what makes finding the bug so hard. I have consulted with the upstream Suricata developers, and they told me the Radix Tree code is thread-safe.

Be sure you leave the passlist-debugging: yes option set in suricata.yaml to give me the maximum level of debugging log messages to work with.

To revert, you will need to first remove the Suricata package, verify the updated binary was also removed, then install the package again from the pfSense menu under SYSTEM > PACKAGE MANAGER.

Remove the package using the SYSTEM > PACKAGE MANAGER menu option.
Next, run this command from a shell prompt:

pkg-static delete suricata-7.0.2_7

That insures the updated test binary is truly removed. If you receive a "not found" or "not installed" error, that simply means the updated binary was removed when the package was removed.

Return to the SYSTEM > PACKAGE MANAGER menu and install Suricata again from the official pfSense repo. This will pull down the current RELEASE package version.