Suricata Rules Update Drops Internet Connection (briefly)

uplink

Thanks for the quick reply !

So, I'm not using inline mode (Block Offenders unchecked) because I am in the process of tuning the rules before I start blocking traffic. I can try that "Live Rule Swap on Update" feature you mentioned, it sounds promising. Unless, that only works when you are in inline mode?

SteveITS

@uplink The live swap keeps both old and new rules in RAM (so needs 2x RAM) instead of stop/start Suricata. I haven't used Inline but I vaguely recall posts about the Suricata stop/start. Legacy mode copies the packets so I wouldn't expect a connection drop like that.

bmeeks

Inline IPS Mode uses the netmap kernel device, and that device will drop and restart the physical interface when it is closed and opened. This happens each time Suricata stops and restarts on a netmap-enabled interface.

Turning on "live swap" under GLOBAL SETTINGS tells Suricata to swap out in-memory rules during an update without restarting. As mentioned, that will briefly increase RAM consumption as two copies of the rules will be in memory for a short time until the complete swap is made. But the benefit is Suricata does not restart and thus the netmap-enabled interfaces used with Inline IPS Mode do not cycle down and back up.

Using Legacy Blocking Mode or non-blocking IDS-only mode should never result in the interface cycling (or at least it did not used to do that). Those two modes use libpcap to capture data for the Suricata engine. Unless something changed in a recent upstream update in either Suricata or FreeBSD , libpcap does not bring the interface down and back up. It would be new behavior if that is happening now. But thinking back a little, I do recall some changes to PCAP logic being made in Suricata by the upstream developers. Maybe that introduced a new behavior ??

uplink

@bmeeks

"Inline IPS Mode uses the netmap kernel device, and that device will drop and restart the physical interface when it is closed and opened."

As I mentioned before I am not using either inline or legacy mode the entire "Block Offenders" option is unchecked.

Also this might help. Below is a log output at 7am from /var/log/system.log. I think I can see it taking down the WAN and my LAN at some point in the process. This explains why I detected 2 drops in internet from my desktop. So, it seems that this is happening regardless of being in inline mode or legacy mode. Is this a fair observation?

Feb  1 07:00:00 router php[49876]: [pfBlockerNG] Starting cron process.
Feb  1 07:00:31 router php[49069]: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Feb  1 07:00:34 router php[49069]: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Feb  1 07:00:39 router php[49069]: [Suricata] Snort GPLv2 Community Rules are up to date...
Feb  1 07:00:39 router php[49069]: [Suricata] Hide Deprecated Rules is enabled.  Removing obsoleted rules categories.
Feb  1 07:00:39 router php[49069]: [Suricata] Removed 0 obsoleted rules category files.
Feb  1 07:00:39 router php[49069]: [Suricata] Updating rules configuration for: WAN ...
Feb  1 07:00:40 router php[49069]: [Suricata] Enabling any flowbit-required rules for: WAN...
Feb  1 07:00:40 router php[49069]: [Suricata] Building new sid-msg.map file for WAN...
Feb  1 07:00:40 router php[49069]: [Suricata] Suricata STOP for WAN(igc0)...
Feb  1 07:00:55 router php[49069]: [Suricata] Suricata START for WAN(igc0)...
Feb  1 07:00:55 router php[49069]: [Suricata] Suricata has restarted with your new set of rules for WAN...
Feb  1 07:00:55 router php[49069]: [Suricata] Updating rules configuration for: LAN ...
Feb  1 07:00:55 router tail_pfb[66267]: [pfBlockerNG] Firewall Filter Service stopped
Feb  1 07:00:55 router php_pfb[66445]: [pfBlockerNG] filterlog daemon stopped
Feb  1 07:00:55 router tail_pfb[68823]: [pfBlockerNG] Firewall Filter Service stopped
Feb  1 07:00:55 router php_pfb[68963]: [pfBlockerNG] filterlog daemon stopped
Feb  1 07:00:55 router tail_pfb[70785]: [pfBlockerNG] Firewall Filter Service started
Feb  1 07:00:55 router php[49069]: [Suricata] Enabling any flowbit-required rules for: LAN...
Feb  1 07:00:55 router php[71123]: [pfBlockerNG] filterlog daemon started
Feb  1 07:00:55 router php[49069]: [Suricata] Building new sid-msg.map file for LAN...
Feb  1 07:00:56 router php[49069]: [Suricata] Suricata STOP for LAN(igc1)...
Feb  1 07:00:56 router check_reload_status[412]: Linkup starting igc1
Feb  1 07:00:56 router kernel: igc1: link state changed to DOWN
Feb  1 07:00:56 router kernel: igc1.20: link state changed to DOWN
Feb  1 07:00:56 router kernel: igc1.40: link state changed to DOWN
Feb  1 07:00:56 router kernel: igc1.30: link state changed to DOWN
Feb  1 07:00:56 router check_reload_status[412]: Linkup starting igc1.20
Feb  1 07:00:56 router check_reload_status[412]: Linkup starting igc1.40
Feb  1 07:00:56 router check_reload_status[412]: Linkup starting igc1.30
Feb  1 07:00:57 router php-fpm[72321]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (192.168.10.1 )
Feb  1 07:00:57 router php-fpm[78706]: /rc.linkup: Hotplug event detected for IOT(opt1) static IP (192.168.20.1 )
Feb  1 07:00:57 router php-fpm[374]: /rc.linkup: Hotplug event detected for GUEST(opt3) static IP (192.168.40.1 )
Feb  1 07:00:57 router php-fpm[9938]: /rc.linkup: Hotplug event detected for WORK(opt2) static IP (192.168.30.1 )
Feb  1 07:00:57 router check_reload_status[412]: Reloading filter
Feb  1 07:00:57 router check_reload_status[412]: Reloading filter
Feb  1 07:00:58 router php[49876]: [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Feb  1 07:00:58 router php[49876]: 
Feb  1 07:01:00 router check_reload_status[412]: Linkup starting igc1
Feb  1 07:01:00 router kernel: igc1: link state changed to UP
Feb  1 07:01:00 router kernel: igc1.20: link state changed to UP
Feb  1 07:01:00 router kernel: igc1.40: link state changed to UP
Feb  1 07:01:00 router kernel: igc1.30: link state changed to UP
Feb  1 07:01:00 router check_reload_status[412]: Linkup starting igc1.20
Feb  1 07:01:00 router check_reload_status[412]: Linkup starting igc1.40
Feb  1 07:01:00 router check_reload_status[412]: Linkup starting igc1.30

bmeeks

I am not sure Suricata is the culprit here. pfBlockerNG is triggering one of its cron tasks at the exact same time. You can see that in the logs. It is cycling things in the Firewall Filter Service. That may be the cause of your problem, not Suricata.

Here is the pfBlockerNG cron task start entry:

Feb  1 07:00:00 router php[49876]: [pfBlockerNG] Starting cron process.

uplink

@bmeeks

Yeah, got catch! I checked PfBlocker and the cron job starts at the top of the hour (see screenshot below). I probably should change that to on the half hour 00:30 so they don't collide. I'll try that and report back :)

PFTR

I have a similar issue that causes unbound and several services to restart each Suricata update:

<13>1 2023-03-07T00:07:09.434063+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
<13>1 2023-03-07T00:07:10.257326+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Emerging Threats Open rules file update downloaded successfully.
<13>1 2023-03-07T00:07:11.716728+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Snort VRT rules are up to date...
<13>1 2023-03-07T00:07:11.922606+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Snort GPLv2 Community Rules are up to date...
<13>1 2023-03-07T00:07:13.104309+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Updating rules configuration for: COMCAST ...
<13>1 2023-03-07T00:07:14.677505+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Enabling any flowbit-required rules for: COMCAST...
<13>1 2023-03-07T00:07:14.781372+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Building new sid-msg.map file for COMCAST...
<13>1 2023-03-07T00:07:15.050909+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Suricata STOP for COMCAST(igc0)...
<141>1 2023-03-07T00:07:15.057326+00:00 pf-firewall.local suricata 19752 - - [132670] <Notice> -- Signal Received.  Stopping engine.
<141>1 2023-03-07T00:07:16.790513+00:00 pf-firewall.local suricata 19752 - - [132670] <Notice> -- Stats for 'igc0':  pkts: 403025839, drop: 116912 (0.03%), invalid chksum: 0
<13>1 2023-03-07T00:07:30.064085+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Suricata START for COMCAST(igc0)...
<13>1 2023-03-07T00:07:30.066399+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Suricata has restarted with your new set of rules for COMCAST...
<13>1 2023-03-07T00:07:30.066777+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Updating rules configuration for: CRADLEPOINT ...
<141>1 2023-03-07T00:07:30.085878+00:00 pf-firewall.local suricata 97216 - - [132670] <Notice> -- This is Suricata version 6.0.10 RELEASE running in SYSTEM mode
<13>1 2023-03-07T00:07:31.542601+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Enabling any flowbit-required rules for: CRADLEPOINT...
<13>1 2023-03-07T00:07:31.744669+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Building new sid-msg.map file for CRADLEPOINT...
<13>1 2023-03-07T00:07:32.029850+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Suricata STOP for Cradlepoint(lagg0.4089)...
<141>1 2023-03-07T00:07:32.036821+00:00 pf-firewall.local suricata 26230 - - [132677] <Notice> -- Signal Received.  Stopping engine.
<141>1 2023-03-07T00:07:33.585507+00:00 pf-firewall.local suricata 26230 - - [132677] <Notice> -- Stats for 'lagg0.4089':  pkts: 14779281, drop: 0 (0.00%), invalid chksum: 0
<13>1 2023-03-07T00:07:47.046791+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Suricata START for Cradlepoint(lagg0.4089)...
<13>1 2023-03-07T00:07:47.049262+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] Suricata has restarted with your new set of rules for CRADLEPOINT...
<13>1 2023-03-07T00:07:47.050522+00:00 pf-firewall.local php-cgi 10300 - - [Suricata] The Rules update has finished.
<141>1 2023-03-07T00:07:47.065461+00:00 pf-firewall.local suricata 88999 - - [131480] <Notice> -- This is Suricata version 6.0.10 RELEASE running in SYSTEM mode
<13>1 2023-03-07T00:08:26.366538+00:00 pf-firewall.local check_reload_status 387 - - Linkup starting igc0
<6>1 2023-03-07T00:08:26.366671+00:00 pf-firewall.local kernel - - - igc0: link state changed to DOWN
<141>1 2023-03-07T00:08:26.540859+00:00 pf-firewall.local suricata 97216 - - [100119] <Notice> -- all 5 packet processing threads, 2 management threads initialized, engine started.
<27>1 2023-03-07T00:08:27.379542+00:00 pf-firewall.local php-fpm 349 - - /rc.linkup: Hotplug event detected for COMCAST(wan) dynamic IP address (4: 50.xxx.xxx.xxx, 6: dhcp6)
<27>1 2023-03-07T00:08:27.379732+00:00 pf-firewall.local php-fpm 349 - - /rc.linkup: DEVD Ethernet detached event for wan
<30>1 2023-03-07T00:08:27.419115+00:00 pf-firewall.local avahi-daemon 51185 - - Leaving mDNS multicast group on interface ix0.2.IPv6 with address xxxx.
<28>1 2023-03-07T00:08:27.419170+00:00 pf-firewall.local avahi-daemon 51185 - - IPV6_DROP_MEMBERSHIP failed: Can't assign requested address
<30>1 2023-03-07T00:08:27.419184+00:00 pf-firewall.local avahi-daemon 51185 - - Joining mDNS multicast group on interface ix0.2.IPv6 with address xxxx.
<30>1 2023-03-07T00:08:27.419342+00:00 pf-firewall.local avahi-daemon 51185 - - Leaving mDNS multicast group on interface ix0.3.IPv6 with address xxxx.
<28>1 2023-03-07T00:08:27.419358+00:00 pf-firewall.local avahi-daemon 51185 - - IPV6_DROP_MEMBERSHIP failed: Can't assign requested address
<30>1 2023-03-07T00:08:27.419370+00:00 pf-firewall.local avahi-daemon 51185 - - Joining mDNS multicast group on interface ix0.3.IPv6 with address xxxx.
<30>1 2023-03-07T00:08:27.419496+00:00 pf-firewall.local avahi-daemon 51185 - - Leaving mDNS multicast group on interface ix0.6.IPv6 with address xxxx.
<28>1 2023-03-07T00:08:27.419511+00:00 pf-firewall.local avahi-daemon 51185 - - IPV6_DROP_MEMBERSHIP failed: Can't assign requested address
<30>1 2023-03-07T00:08:27.419523+00:00 pf-firewall.local avahi-daemon 51185 - - Joining mDNS multicast group on interface ix0.6.IPv6 with address xxxx.
<27>1 2023-03-07T00:08:29.472933+00:00 pf-firewall.local php-fpm 349 - - /rc.linkup: Shutting down Router Advertisement daemon cleanly
<13>1 2023-03-07T00:08:30.485407+00:00 pf-firewall.local check_reload_status 387 - - Linkup starting igc0
<6>1 2023-03-07T00:08:30.485561+00:00 pf-firewall.local kernel - - - igc0: link state changed to UP
<11>1 2023-03-07T00:08:31.865777+00:00 pf-firewall.local dhcpleases 74897 - - Could not deliver signal HUP to process 51002: No such process.
<141>1 2023-03-07T00:08:42.519598+00:00 pf-firewall.local suricata 88999 - - [100201] <Notice> -- all 5 packet processing threads, 2 management threads initialized, engine started.
<27>1 2023-03-07T00:08:47.787651+00:00 pf-firewall.local php-fpm 349 - - /rc.linkup: Gateway, switch to: CRADLEPOINT_DHCP
<27>1 2023-03-07T00:08:47.791675+00:00 pf-firewall.local php-fpm 349 - - /rc.linkup: Default gateway setting Interface CRADLEPOINT_DHCP Gateway as default.

This causes a few seconds to a minute+ outage as the services bounce and unbound comes back online.

bmeeks

@pftr:
Go to the GLOBAL SETTINGS tab in Suricata and enable the Live Rule Swap on Update option. That will send Suricata a special signal to reload the rules without shutting down and restarting. However, this will cause a temporary increase in RAM usage during the rule swap. If you have tons of enabled rules and are low on free RAM, you might encounter an error with this setting enabled. If you do, simply uncheck the option.

TSE 0

@bmeeks cheers! I found this thread after analysing why I was losing data buffered by telegraf when I brought down my InfluxDB service overnight. I have more than enough metrics buffer configured …

Turns out it was a consequence of Suricata restarting the interfaces, which (based on pfsense logs) also causes all my packages to be shut down and restarted… so out go any buffered telegraf metrics…

So - TLDR - turning on live swap also cures the need for package reload … which may be important to some.

bmeeks

@tse-0 said in Suricata Rules Update Drops Internet Connection (briefly):

@bmeeks cheers! I found this thread after analysing why I was losing data buffered by telegraf when I brought down my InfluxDB service overnight. I have more than enough metrics buffer configured …

Turns out it was a consequence of Suricata restarting the interfaces, which (based on pfsense logs) also causes all my packages to be shut down and restarted… so out go any buffered telegraf metrics…

So - TLDR - turning on live swap also cures the need for package reload … which may be important to some.

Yes, this is a trick I have often recommended when using Inline IPS Mode as the native netmap device used by that mode will restart the underlying interface each time netmap is initialized. Suricata initializes netmap on each enabled interface as it starts when using Inline IPS Mode.

But recently some users have reported similar interface restarting behavior when using Legacy Mode Blocking. That mode uses libpcap, and in the past it never restarted the interface when Suricata initialized PCAP. But Suricata upstream made some libpcap modifications a while back to fix a bug, and one possible fallout from that fix might be restarting the interface when PCAP is activated.