Suricata ~ Updates Killing Network Connections
-
I am having issues with apparently random killing all network connections with 2.4.2p1 also, but I'm not sure it is Suricata doing it. For my system it happens at a seemingly random time from 1-10 days. I have been unable to correlate it to any cron job, and the logs do not show anything except the gateway is down when it happens. For mine the network interface outage usually lasts 1-2 minutes, though at least once I had to reboot. The system at the console thinks all is fine, and there are no crashes. I've tried different hardware including swapping servers and network cards, but the issue persists. Kinda hard to troubleshoot with the frequency so far apart, and no known way of inducing the error.
Hardware: HP DL380 G6, 48 Gig memory, Xeon 5650x2, Intel i340 4 port network cards x 2.
Packages: pfBlockerNG 2.1.2_2, mailreport 3.1, Suricata 4.0.3_1 -
So why is the gateway down? Do you have State killing on gateway failure enabled?
-
The gateways appear to go down because all of the Interfaces stop passing traffic (not just the WAN gateways). I currently have 4 physical interfaces configured. 2 WAN, 1 LAN, and 1 LAN that is just for management of the firewall (no NAT, no Suricata). The 2 LAN interfaces are configured as VLANS. When the issue occurs, I lose connectivity to the management LAN as well (no Suricata on that interface). Typically I will start to see packet loss on 1 gateway and within seconds the other, then all go down including the LAN's for about a minute. I know the LAN's go down as I have a separate Zabbix server pinging, and I cannot reach the web GUI until back up. I have verified that when this occurs the switch and server Ethernet ports do not lose physical connectivity.
As for "State killing on gateway failure enabled?", in Suricata I do have the "Kill States" checked which is the default. However, that should have no effect on the Interfaces that are not set up to be monitored by Suricata, and those go down as well. Do you recommend turning "Kill States" off?
Also in my case if it helps, using Suricata in Legacy mode.
Another anomaly which may/may not be related: After a server restart Suricata does not show any blocks. It shows alerts, but no blocks. If I restart the Suricata service, it works normally and shows blocks (I am only blocking mostly p2p). This morning I updated to Suricata 4.0.3_2 but it did not change this behavior.
I don't know if this is the same issue as Teken's or not, but it looked similar so I thought I'd throw this out there.
-
@RichH:
Another anomaly which may/may not be related: After a server restart Suricata does not show any blocks. It shows alerts, but no blocks. If I restart the Suricata service, it works normally and shows blocks (I am only blocking mostly p2p). This morning I updated to Suricata 4.0.3_2 but it did not change this behavior.
I don't know if this is the same issue as Teken's or not, but it looked similar so I thought I'd throw this out there.
What server are talking about restarting? Is it the box hosting pfSense and Suricata? When you say you "restart Suricata", is it showing as running on the interface when you click the restart icon or is it showing as not running and you click the start icon?
In regards to "showing alerts but no blocks", you do realize that the ALERTS tab is reading from a log file. So if that log file is not erased during a reboot of the firewall the alerts in the log file will be displayed on the ALERTS tab after the reboot. New alerts at the top and older ones at the bottom. The BLOCKS tab, on the other hand, literally reads the contents of the packet filter firewall's snort2c table and displays those. After a reboot, the snort2c table is cleared out until network traffic triggers more alerts which the result in blocks. So after a firewall reboot and before any new alerts have been received, the BLOCKS tab will be empty.
Bill
-
The "server" is the HP G6 box running pfSense, sorry I didn't make that clear. As for Suricata "showing alerts but no blocks" – they are not logs but current alerts that can be seen by the date and time. I even tested going to do a P2P download (which is set to be blocked by Suricata) just to be sure. It is logged in the alerts, but not in the "Blocks" section. I restart the Suricata service from Status-Services and then do the test again. The Block now shows. It is a reproducible issue. I just wish the killed network connections was something I knew how to reproduce, as I presume once it can be reproduced it can be fixed!
Thanks much!
-
@RichH:
The "server" is the HP G6 box running pfSense, sorry I didn't make that clear. As for Suricata "showing alerts but no blocks" – they are not logs but current alerts that can be seen by the date and time. I even tested going to do a P2P download (which is set to be blocked by Suricata) just to be sure. It is logged in the alerts, but not in the "Blocks" section. I restart the Suricata service from Status-Services and then do the test again. The Block now shows. It is a reproducible issue. I just wish the killed network connections was something I knew how to reproduce, as I presume once it can be reproduced it can be fixed!
Thanks much!
Yes, they are logs. Every single alert shown on that tab is read from an alert log file in /var/logs/suricata and sub-directories present there for each interface. The PHP code on the ALERTS tab opens the current log file and reads it into memory and displays the contents. That is the only place alerts are pulled from (the log files). As for blocks, when the BLOCKS tab loads and renders in browser, PHP code is executed that reads the snort2c table from pfSense, loads its contents (IP addresses) into an array and displays the IP addresses it found. Those are the "blocks" displayed. Here is a snippet of the actual code that does that:
exec("/sbin/pfctl -t snort2c -T show", $blocked_ips);I'm having a very difficult time reconciling what you describe with how the internals of the code actually works. Something is not quite right in your setup perhaps. For starters, when you see this behavior and go to the INTERFACES tab, what icon is showing for each of the Suricata-enabled interfaces? Is it a green check or is it a red X? Green means Suricata is running, red X means it is not or has died. Do you run any other packages on this firewall? One package known to cause issues with Snort and Suricata both is the Service Watchdog package. It will start multiple copies of Suricata that then fight with each other.
Bill
-
Thanks Bill. The 2 Suricata Interfaces I have set up to use are each showing a green check (1 WAN and 1 LAN), and have been green even when I have had issues. I presume Suricata is working as the current alerts show up even when the blocks do not. For packages I have only these 3:
pfBlockerNG 2.1.2_2
mailreport 3.1
Suricata 4.0.3_2 (though I see an update to 4.04 is available which I'll apply later today)I had to restart this morning but did not get to check right away; when I did it was working so maybe it is not consistent or takes a while to start working again or something.
-
Hi SuperTechie,
I'm having the same issues where the Suricata update is taking my interfaces down. Did you find a fix? Here are my pfSense/package versions:
Netgate SG-8860
2.4.2-RELEASE-p1
Suricata security 4.0.4My system logs make it seem like Suricata is the cause.
System Logs:
018-08-01 00:31:52.000 stjomocpfwm.npgco.com
SuricataStartup[56447]: Suricata STOP for WAN809(31583_igb3)...
2018-08-01 00:31:52.000 stjomocpfwm.npgco.com
suricata[12157]: [100193] <Notice> -- Signal Received. Stopping engine.
2018-08-01 00:31:51.000 stjomocpfwm.npgco.com
kernel: igb1: link state changed to DOWN
2018-08-01 00:31:51.000 stjomocpfwm.npgco.com
check_reload_status: Linkup starting igb1
2018-08-01 00:31:51.000 stjomocpfwm.npgco.com
suricata[10428]: [101459] <Notice> -- Stats for 'igb1': pkts: 134461382, drop: 256 (0.00%), invalid chksum: 0
2018-08-01 00:31:51.000 stjomocpfwm.npgco.com
kernel: igb1: promiscuous mode disabled
2018-08-01 00:31:51.000 stjomocpfwm.npgco.com
suricata[10428]: [101459] <Info> -- cleaning up signature grouping structure... complete
2018-08-01 00:31:50.000 stjomocpfwm.npgco.com
SuricataStartup[54412]: Suricata STOP for WAN801(9911_igb1)...
2018-08-01 00:31:50.000 stjomocpfwm.npgco.com
suricata[10428]: [101459] <Notice> -- Signal Received. Stopping engine.
2018-08-01 00:31:50.000 stjomocpfwm.npgco.com
suricata[10428]: [100791] <Info> -- (RX#01-igb1) Pcap Total:134458188 Recv:134457932 Drop:256 (0.0%).
2018-08-01 00:31:50.000 stjomocpfwm.npgco.com
php-cgi: suricata_check_for_rule_updates.php: [Suricata] Building new sid-msg.map file for WAN809...
2018-08-01 00:31:41.000 stjomocpfwm.npgco.com
php-cgi: suricata_check_for_rule_updates.php: [Suricata] Updating rules configuration for: WAN809 ...
2018-08-01 00:31:40.000 stjomocpfwm.npgco.com
php-cgi: suricata_check_for_rule_updates.php: [Suricata] Building new sid-msg.map file for WAN801...
2018-08-01 00:31:31.000 stjomocpfwm.npgco.com
php-cgi: suricata_check_for_rule_updates.php: [Suricata] Removed 51 obsoleted rules category files.
2018-08-01 00:31:31.000 stjomocpfwm.npgco.com
php-cgi: suricata_check_for_rule_updates.php: [Suricata] Updating rules configuration for: WAN801 ... -
@jaredw said in Suricata ~ Updates Killing Network Connections:
2018-08-01 00:31:51.000 stjomocpfwm.npgco.com
kernel: igb1: link state changed to DOWNLooks like it's actually taking the link down there.
However is there a reason you're running 2.4.2 still?
This is the 2.4 development board, currently 2.4.4a.Steve
-
Hi Steve,
No reason other than we have many of these firewalls in production and haven't gotten to updating. I'll update one and see if the problem goes away and keep you posted.
