Exempt IPSEC Traffic Suricata

bmeeks

My first suspicion is that you have a duplicate Suricata process running on an interface. One symptom of a duplicate process is the apparent ignoring of GUI changes by Suricata. The duplicate process does not get notified of the GUI change and continues with the old setting.

Do this to clear all Suricata processes and restart them.

Go to the INTERFACES tab in Suricata and stop each running instance by clicking the appropriate icon. Wait for all instances to show STOPPED.
Obtain a CLI shell prompt on the firewall either directly on the console or via an SSH remote session. From the console menu choose option 8 to drop to a shell session.
Execute the following command to list all running Suricata processes:

ps -ax | grep suricata

You should see none, but if you have any still running each will need to be stopped. Locate the Process ID (PID) of each listed running Suricata process and kill it using this command:

kill -9 <pid>

Verify all running Suricata processes are terminated by repeating step 3 above.
Now you can return to the GUI and restart your Suricata instances there, or you can execute this command while still in a shell session to do the same thing:

/usr/local/etc/rc.d/suricata.sh start

When assigning a Pass List to an interface, you must click Save down at the bottom of the INTERFACE SETTINGS page. I assume you are doing that ??? Then you must restart Suricata on the interface where you made the Pass List change.

Do not confuse a Pass List with a Suppress List. They are similar but have different roles. A Pass List contains IP addresses that should not be blocked, but you will still receive alerts from IP addresses on a Pass List. A Suppress List suppresses an alerting rule entirely. It is equivalent to disabling the matching rule, so no alerts will be recorded and no blocks imposed. Suppress Lists allow some additional filtering criteria such as only suppressing an alert for specific source or destination IP addresses, for example.

Do NOT modify the values for the HOME_NET and EXTERNAL_NET options unless you are very skilled in operating an IDS/IPS. The default values for those parameters are almost always correct for users. Only in advanced situations should they be modified. Incorrect choices there can result in many rules not properly triggering because their $HOME_NET and/or $EXTERNAL_NET conditionals will not be satisfied.

Overcon

@bmeeks

Hi Meeks.

I tried your steps, and I didn't see multiple listings. I got:

ps -ax | grep suricata
46534  -  Ss      508:41.41 /usr/local/bin/suricata -i igc0 -D -c /usr/local/etc/suricata/suricata_44891_igc0/suricata.
59966  0  S+        0:00.00 grep suricata

I have done SAVE on the pass list. I checked the logs, and I see it applying the pass lists. I have checked what logs I can find, and I am not seeing anything that I can tell is the rule it is using to block this traffic.

It is rather frustrating; I have to say. All I know is to search logs for my IPs (Internal, External, Remote) and then try to find the associated rule, but I am to the point I don't see any of my IPs. Here is a section of what it is loading (suricata.log).

14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> Creating automatic firewall interface IP address Pass List.
14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> adding firewall interface igc0 IPv4 address 68.xxx.158.163 to automatic interface IP Pass List.
14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> adding firewall interface igc1 IPv4 address 172.16.1.1 to automatic interface IP Pass List.
14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.
14/10/2022 -- 17:28:03 - <Info> -- alert-pf output device (regular) initialized: block.log
14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_5278_igc1/passlist.
14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> Added IPv4 address 23.175.xxx.26 from assigned Pass List.
14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> Added IPv4 address 23.175.xxx.27 from assigned Pass List.
14/10/2022 -- 17:28:03 - <Info> -- alert-pf -> Added IPv4 address 24.236.xxx.186 from assigned Pass List.
etc...

bmeeks

Your reply is bit confusing to me when you say you are searching the logs and not finding your blocked IP. Not sure what you are doing or maybe trying to say there.

To find out what IP addresses are currently being blocked by Suricata when using Legacy Blocking Mode, simply look on the BLOCKS tab. That tab shows all the IP addresses that are currently being blocked by Suricata. If the host you think is being blocked is not listed on that tab, then Suricata is not blocking it. The PHP code on that tab populates the display by querying the pf firewall's snort2c table and showing the IPs existing in that table. All IP addresses present in that table are being blocked by the firewall.

If you want to see which rule caused the block, that will be shown not only on the BLOCKS tab, but also on the ALERTS tab. On the ALERTS tab you can filter the data using combinations of the columns. You can also sort the data by clicking on a column, so click the SOURCE or DESTINATION IP address column and then look for your blocked host's IP address and find what rule blocked it.

In Legacy Blocking Mode, everything Suricata does is shown on the ALERTS and BLOCKS tab, so if your host is not showing up there, Suricata blocking is not your problem.

P.S. - the only way that an alert would not be shown on the ALERTS tab is if there are a ton of alerts getting logged and the alerts log is being rapidly rotated. The code on the ALERTS tab will not go searching in any rotated logs. It only examines the currently active log file when displaying alerts.

Overcon

@bmeeks Sorry Meeks, it's hard to verbally explain it when you're unfamiliar with the terminology. My issue is I am not seeing a familiar IP that is being blocked (aside from ones I already knew about). I have a website, and on the site, it is streaming a camera to a remote IP using RTSP. The website stays up, and it works fine, but the camera feed goes down. I have searched the blocks for the IP of the server that is reaching out to the camera (the remote server), but I am not seeing it in the blocks. All I know when I clear the blocks, the camera comes back online. But for the life of me, I can't find what rule is clocking that traffic or how to figure it out.

I have added all the IPs involved in the pass list, but still no go.

Another issue, and I think they might be related (or maybe just related because of how my network pretty much depends on the IPSEC tunnel being up) is that Suricata is also blocking some specific traffic coming across the IPSEC tunnel, but once I clear the blocks, it comes back up.

How can I find what rule(s) are causing this, if the IPs are in the pass list, the IPSEC tunnel should be exempted by default and I can't find a block associated with any of the IPs?

bmeeks

Do you know the IP address of the remote server that is querying the camera? Is that IP static, or might it be changing on the remote server's end (meaning, is it possibly a dynamic WAN IP for that remote device)?

There is no physical way for Suricata to block an IP in Legacy Blocking mode without that blocked IP showing up on the BLOCKS tab and in the snort2c table when you view its contents under DIAGNOSTICS > TABLES in the pfSense menu. So, if clearing blocks makes the camera start working, then you must be looking for the wrong IP address in the BLOCKS list. Either that, or another IP in the traffic chain such as a gateway is getting blocked and that hinders the camera-to-server communications. Trace the full path traffic must take to get from the camera to the remote server. See if any of those IP addresses are showing up in the BLOCKS table.

Once you find the applicable IP that is being blocked, then you can search the ALERTS tab to find which rule is triggering the block. You can then disable that rule. You can also create a Suppress List entry to suppress alerts (and thus blocks) by IP address. So, again, once you find the alerting rule, you can suppress it for a specific IP address.

Overcon

@bmeeks said in Exempt IPSEC Traffic Suricata:

Do you know the IP address of the remote server that is querying the camera? Is that IP static, or might it be changing on the remote server's end (meaning, is it possibly a dynamic WAN IP for that remote device)?

There is no physical way for Suricata to block an IP in Legacy Blocking mode without that blocked IP showing up on the BLOCKS tab and in the snort2c table when you view its contents under DIAGNOSTICS > TABLES in the pfSense menu. So, if clearing blocks makes the camera start working, then you must be looking for the wrong IP address in the BLOCKS list. Either that, or another IP in the traffic chain such as a gateway is getting blocked and that hinders the camera-to-server communications. Trace the full path traffic must take to get from the camera to the remote server. See if any of those IP addresses are showing up in the BLOCKS table.

Once you find the applicable IP that is being blocked, then you can search the ALERTS tab to find which rule is triggering the block. You can then disable that rule. You can also create a Suppress List entry to suppress alerts (and thus blocks) by IP address. So, again, once you find the alerting rule, you can suppress it for a specific IP address.

Yes, the remote IP is a static IP. I have that IP in the pass list. The company normally requests data from servers all over the world, but a part of their service is that you can request a static IP so you can allow the RTSP traffic through your firewall from their server instead of creating an Alias and entering all their IPs from around the world or something like that.

That is a great idea, I will do a trace and check those IPs, I didn't even think of that. Let me do that and I will get back.

bmeeks

@overcon said in Exempt IPSEC Traffic Suricata:

That is a great idea, I will do a trace and check those IPs, I didn't even think of that. Let me do that and I will get back.

Not every IP address in a trace matters. Only the gateway you talk to, or that your IPSEC VPN is connected to. Intervening IPs along the hop route do not matter as Suricata will never see them.

Overcon

@bmeeks said in Exempt IPSEC Traffic Suricata:

@overcon said in Exempt IPSEC Traffic Suricata:

That is a great idea, I will do a trace and check those IPs, I didn't even think of that. Let me do that and I will get back.

Not every IP address in a trace matters. Only the gateway you talk to, or that your IPSEC VPN is connected to. Intervening IPs along the hop route do not matter as Suricata will never see them.

I didn't find any of the IP's in the block or alert pages (that isn't already excluded). I checked for my gateway and the last pingable device in the traceroute, which I don't know where it is 208.115.251.46. Anyway, I have searched for my public IPs, my gateway IP, my IPSEC tunnel IPs, the cameras IP, the server that polls the cameras IP, and the last replying IP when I did a TR to the IP of the remote server.

bmeeks

@overcon said in Exempt IPSEC Traffic Suricata:

I didn't find any of the IP's in the block or alert pages (that isn't already excluded).

That last part in parentheses is confusing me. What do you mean by "that isn't already excluded"?

The task here is pretty simple. Assume the camera is IP 1.2.3.4 and the remote server querying the camera has the IP 4.3.2.1.

If you don't see 1.2.3.4 or 4.3.2.1 in the list of alerts on the ALERTS tab, and
you don't see either of those two IP addresses in the list on the BLOCKS tab, and
finally, you don't see either of those IP addresses in the list of addresses in the snort2c table when looking under DIAGNOSTICS > TABLES in the pfSense menu; then Suricata is not blocking the device. Something else is going on.

Perhaps, if you have this traffic encapsulated in an IPSEC tunnel, one of the two tunnel endpoint IPs is getting blocked ?? Have you checked for those?

It might be easier for you to just disable Suricata in this situation.

Overcon

@bmeeks said in Exempt IPSEC Traffic Suricata:

@overcon said in Exempt IPSEC Traffic Suricata:

I didn't find any of the IP's in the block or alert pages (that isn't already excluded).

That last part in parentheses is confusing me. What do you mean by "that isn't already excluded"?

The task here is pretty simple. Assume the camera is IP 1.2.3.4 and the remote server querying the camera has the IP 4.3.2.1.

If you don't see 1.2.3.4 or 4.3.2.1 in the list of alerts on the ALERTS tab, and

you don't see either of those two IP addresses in the list on the BLOCKS tab, and

finally, you don't see either of those IP addresses in the list of addresses in the snort2c table when looking under DIAGNOSTICS > TABLES in the pfSense menu; then Suricata is not blocking the device. Something else is going on.

Perhaps, if you have this traffic encapsulated in an IPSEC tunnel, one of the two tunnel endpoint IPs is getting blocked ?? Have you checked for those?

It might be easier for you to just disable Suricata in this situation.

I mean I have already excluded rules that were triggering on those IPs, in alerts.

Like these. I have already excluded these rules; which are triggering on the remote IP and the local IP. I do not see either IP for the ends of the IPSEC tunnel listed, and I have them in the PASS list. I also have my public IPs in the passlist, and while I do see them in the alerts list, I don't see them in relation to the IPCAM or the remote servers IP.

I am going to check the snort2c right now. Sorry, I was going to check it last night, but I did an update on my mail server (to 22.04), and it killed it, so I was trying to see if I could fix that.

There are seven alerts that reference the IPs, all those rules are excluded (disabled).
alt text

Overcon

@bmeeks I checked the snort2c, and I don't see any of the IPs I am dealing with listed.

Should I suppress all the alerts to see if maybe they are triggering even though I have them disabled? I haven't done that yet. The thing I don't get is when the camera gets blocked, I have searched the blocks and cannot find any IP related to the camera.

bmeeks

@overcon said in Exempt IPSEC Traffic Suricata:

@bmeeks I checked the snort2c, and I don't see any of the IPs I am dealing with listed.

Should I suppress all the alerts to see if maybe they are triggering even though I have them disabled? I haven't done that yet. The thing I don't get is when the camera gets blocked, I have searched the blocks and cannot find any IP related to the camera.

If you do not see any IP related to the camera in the blocks listed on the BLOCKS tab nor in the IP's shown in snort2c table when viewing it under DIAGNOSTICS > TABLES, then Suricata is not your problem and looking through rules is wasted time. Suricata cannot block without placing the IP in the snort2c table. That table is literally how the custom plugin that does Legacy Mode blocking works. It puts the IP in that table, and then the pfSense firewall engine performs the actual block of traffic. So no related IP in that table means Suricata is not the root cause of your issue.

I know how all that works because I wrote the code for the custom plugin that does the Legacy Mode blocking.

Overcon

@bmeeks said in Exempt IPSEC Traffic Suricata:

So no related IP in that table means Suricata is not the root cause of your issue.

I believe you, I just can't see how it can't be a Suricata issue if clearing all the blocks allows the camera to resume working.

bmeeks

@overcon said in Exempt IPSEC Traffic Suricata:

@bmeeks said in Exempt IPSEC Traffic Suricata:

So no related IP in that table means Suricata is not the root cause of your issue.

I believe you, I just can't see how it can't be a Suricata issue if clearing all the blocks allows the camera to resume working.

Have you gone to DIAGNOSTICS > TABLES in the pfSense menu, selected the snort2c table in the drop-down, and then examined all the IP addresses displayed?

Those IP addresses are the only thing Suricata is blocking. If clearing blocks (if by that you mean going to the BLOCKS tab in Suricata and clicking the Clear button) restores the camera to operating status, then you are not looking for the correct IP when trying to find why the camera is blocked. There is no other answer possible. Only IP addresses in the snort2c table are blocked (by Suricata), so if you do not see the correct IP in the table, then Suricata is not blocking it. Conversely, if clearing blocks (which empties the snort2c table of IP addresses) restores the camera's operation, then the only logical conclusion is you are searching for the wrong IP when looking in the table and on the BLOCKS tab.

Overcon

@bmeeks said in Exempt IPSEC Traffic Suricata:

@overcon said in Exempt IPSEC Traffic Suricata:

@bmeeks said in Exempt IPSEC Traffic Suricata:

So no related IP in that table means Suricata is not the root cause of your issue.

I believe you, I just can't see how it can't be a Suricata issue if clearing all the blocks allows the camera to resume working.

Have you gone to DIAGNOSTICS > TABLES in the pfSense menu, selected the snort2c table in the drop-down, and then examined all the IP addresses displayed?

Those IP addresses are the only thing Suricata is blocking. If clearing blocks (if by that you mean going to the BLOCKS tab in Suricata and clicking the Clear button) restores the camera to operating status, then you are not looking for the correct IP when trying to find why the camera is blocked. There is no other answer possible. Only IP addresses in the snort2c table are blocked (by Suricata), so if you do not see the correct IP in the table, then Suricata is not blocking it. Conversely, if clearing blocks (which empties the snort2c table of IP addresses) restores the camera's operation, then the only logical conclusion is you are searching for the wrong IP when looking in the table and on the BLOCKS tab.

I only searched for the IPs I know that are involved. I can't confirm with 100% that they are not sending traffic from another IP, as I don't know how their systems are configured. I just know the IP they gave me and that they told me traffic would only come from it (so I could lock it down in my firewall rules).

They also sent me a list of all their IPs, which I plugged into an ALIAS. I will change the rule to use that and see if that changes anything. The reason I haven't done that is that I assumed the firewall would drop any traffic that wasn't from the specified IP in the rule and Suricata wouldn't be involved since the firewall would drop any traffic before Suricata would see it, but I'll give it a go and see.

The next time it gets blocked, I will search the snort2c table and see if any of those IPs are in there.

Overcon

@bmeeks I think I found it! So I waited until it blocked it again. Then I searched the snort2c list and found the remote IP of the server. So I confirmed it was getting blocked. Then I searched the alerts, but I increased the list count to 5000 instead of 500 and turned off auto-refresh. Then I found this new alert:

#SURICATA HTTP URI terminated by non-compliant character
suppress gen_id 1, sig_id 2221029

So I suppressed that one, removed the IP in the snort2c table, and the camera came back up!

So now, I just have to see if it holds! I will set the firewall back up to allow that IP and see if it makes a difference since I don't think changing it to the alias with all their IPs made a difference.

Thanks for the patient help on this!

Here is the site with the camera:

Lovington Weather