Firewall *Stops* and Break in Logs For Extended Time
-
I came home today and my network was down. Rebooted pfSense, nothing. Rebooted internet, nothing. Rebooted switch, nothing. Unplugged my work PC, and everything came back to life. Not sure what the issue is there, but while looking at the logs I am discovering 12+ hour blocks of no log messages. Can anyone see what might be causing the logs to stop receiving input and then chug forward again after half a day?
Maybe something with Snort?
Jul 18 00:18:24 kernel igb0: promiscuous mode enabled Jul 18 00:18:22 php 59949 [Snort] The Rules update has finished. Jul 18 00:18:22 php 59949 [Snort] Snort has restarted with your new set of rules... Jul 18 00:18:20 snort 55453 AppInfo: AppId 2314 is UNKNOWN Jul 18 00:18:20 snort 55453 Invalid direct service AppId, 2314, for 0x80809f590 0x80cd84b40 Jul 18 00:18:20 snort 55453 AppInfo: AppId 2312 is UNKNOWN Jul 18 00:18:20 snort 55453 Invalid direct service AppId, 2312, for 0x80809f590 0x80cd84b40 Jul 18 00:18:20 snort 55453 AppInfo: AppId 4082 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 4519 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 4140 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 4082 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 4519 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 4140 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 4140 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 4314 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 3646 is UNKNOWN Jul 18 00:18:20 snort 55453 AppInfo: AppId 3646 is UNKNOWN Jul 18 00:18:20 snort 55453 AppId Jul 18 00:18:20 snort 55453 AppId Jul 18 00:18:20 snort 55453 AppId Jul 18 00:18:20 snort 55453 AppId Jul 18 00:18:20 snort 55453 AppId Jul 18 00:18:20 snort 55453 AppId Jul 18 00:18:19 kernel igb1.101: promiscuous mode enabled Jul 18 00:18:19 kernel igb1: promiscuous mode enabled Jul 18 00:18:19 snort 55453 AppId Jul 18 00:18:19 snort 55453 AppId Jul 18 00:18:19 snort 55453 AppId Jul 18 00:18:19 snort 55453 AppId Jul 18 00:18:19 snort 55453 AppId Jul 18 00:18:19 php 59949 [Snort] Snort START for VLAN101(igb1.101)... Jul 18 00:18:19 php 59949 [Snort] Snort START for LAN2(igb1)... Jul 18 00:18:19 php 59949 [Snort] Snort START for WAN1(igb0)... Jul 18 00:18:16 kernel igb1.101: promiscuous mode disabled Jul 18 00:18:16 kernel igb1: promiscuous mode disabled Jul 18 00:18:16 snort 27774 *** Caught Term-Signal Jul 18 00:18:15 php 59949 [Snort] Snort STOP for VLAN101(igb1.101)... Jul 18 00:18:14 snort 27479 *** Caught Term-Signal Jul 18 00:18:13 php 59949 [Snort] Snort STOP for LAN2(igb1)... Jul 18 00:18:12 kernel igb0: promiscuous mode disabled Jul 18 00:18:12 snort 27042 *** Caught Term-Signal Jul 18 00:18:11 php 59949 [Snort] Snort STOP for WAN1(igb0)... Jul 18 00:18:11 php 59949 [Snort] Building new sid-msg.map file for VLAN101... Jul 18 00:18:11 php 59949 [Snort] Enabling any flowbit-required rules for: VLAN101... Jul 18 00:18:10 php 59949 [Snort] Updating rules configuration for: VLAN101 ... Jul 18 00:18:10 php 59949 [Snort] Building new sid-msg.map file for LAN2... Jul 18 00:18:10 php 59949 [Snort] Enabling any flowbit-required rules for: LAN2... Jul 18 00:18:09 php 59949 [Snort] Updating rules configuration for: LAN2 ... Jul 18 00:18:09 php 59949 [Snort] Building new sid-msg.map file for WAN1... Jul 18 00:18:09 php 59949 [Snort] Enabling any flowbit-required rules for: WAN1... Jul 18 00:18:09 php 59949 [Snort] Checking flowbit rules dependent on disabled preprocessors for: WAN1... Jul 18 00:18:09 php 59949 [Snort] Enabling any flowbit-required rules for: WAN1... Jul 18 00:18:07 php 59949 [Snort] Checking for rules dependent on disabled preprocessors for: WAN1... Jul 18 00:18:06 php 59949 [Snort] Updating rules configuration for: WAN1 ... Jul 18 00:18:06 php 59949 [Snort] Removed 0 obsoleted rules category files. Jul 18 00:18:06 php 59949 [Snort] Hide Deprecated Rules is enabled. Removing obsoleted rules categories. Jul 18 00:18:06 php 59949 [Snort] Emerging Threats Open rules are up to date... Jul 18 00:18:06 php 59949 [Snort] Snort GPLv2 Community Rules file update downloaded successfully Jul 18 00:18:05 php 59949 [Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz... Jul 18 00:18:05 php 59949 [Snort] Snort AppID Open Text Rules are up to date... Jul 18 00:18:05 php 59949 [Snort] Snort OpenAppID detectors are up to date... Jul 18 00:18:05 php 59949 [Snort] Snort Subscriber rules are up to date... Jul 18 00:03:40 php 78198 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Jul 18 00:00:00 php 78198 [pfBlockerNG] Starting cron process. Jul 17 23:25:00 php 90300 [Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb111304/... Jul 17 23:25:00 php 90300 [Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb011268/... Jul 17 21:27:18 snort 27479 S5: Pruned 5 sessions from cache for memcap. 828 scbs remain. memcap: 8389036/8388608 Jul 17 21:27:18 snort 27042 S5: Pruned 5 sessions from cache for memcap. 652 scbs remain. memcap: 8389072/8388608 Jul 17 19:11:18 php-fpm 80722 /status_dhcp_leases.php: Successful login for user 'admin' from: 192.168.101.76 (Local Database) Jul 17 16:58:09 php-fpm 40582 /avahi_settings.php: Successful login for user 'admin' from: 192.168.101.76 (Local Database) Jul 17 16:16:50 php-fpm 92672 /index.php: Successful login for user 'admin' from: 192.168.101.76 (Local Database) Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_DNSBLIP_v4 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_Whitelist_v4 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_Level4_v4 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_Level3_v4 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_Level2_v4 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_Level1_v4 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v6 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating. Jul 17 12:30:55 php 27239 rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates Jul 17 12:30:00 php 27239 rc.update_urltables: /etc/rc.update_urltables: Sleeping for 55 seconds. Jul 17 12:30:00 php 27239 rc.update_urltables: /etc/rc.update_urltables: Starting up. Jul 17 01:01:03 php 19539 rc.dyndns.update: phpDynDNS (requests.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Jul 17 01:01:02 php 19539 rc.dyndns.update: phpDynDNS (plex.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Jul 17 01:01:01 php 19539 rc.dyndns.update: phpDynDNS (terraria.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Jul 17 01:01:00 php 19539 rc.dyndns.update: phpDynDNS (pfsense.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Jul 17 00:50:00 php 84916 [Snort] perfmon stats logs cleanup job removed 1 file(s) from /var/log/snort/snort_igb011268/... Jul 17 00:18:50 kernel igb0: promiscuous mode enabled Jul 17 00:18:48 php 15099 [Snort] The Rules update has finished. Jul 17 00:18:48 php 15099 [Snort] Snort has restarted with your new set of rules... Jul 17 00:18:46 snort 27042 AppInfo: AppId 2314 is UNKNOWN Jul 17 00:18:46 snort 27042 Invalid direct service AppId, 2314, for 0x80809f590 0x80cda4b40 Jul 17 00:18:46 snort 27042 AppInfo: AppId 2312 is UNKNOWN Jul 17 00:18:46 snort 27042 Invalid direct service AppId, 2312, for 0x80809f590 0x80cda4b40 Jul 17 00:18:46 snort 27042 AppInfo: AppId 3646 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 3646 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4082 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4519 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4140 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4082 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4519 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4140 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4140 is UNKNOWN Jul 17 00:18:46 snort 27042 AppInfo: AppId 4314 is UNKNOWN Jul 17 00:18:46 snort 27042 AppId Jul 17 00:18:46 snort 27042 AppId Jul 17 00:18:46 snort 27042 AppId Jul 17 00:18:46 snort 27042 AppId Jul 17 00:18:46 snort 27042 AppId Jul 17 00:18:46 snort 27042 AppId Jul 17 00:18:45 kernel igb1.101: promiscuous mode enabled Jul 17 00:18:45 kernel igb1: promiscuous mode enabled Jul 17 00:18:45 snort 27042 AppId Jul 17 00:18:45 snort 27042 AppId Jul 17 00:18:45 snort 27042 AppId Jul 17 00:18:45 snort 27042 AppId Jul 17 00:18:45 snort 27042 AppId Jul 17 00:18:45 php 15099 [Snort] Snort START for VLAN101(igb1.101)... Jul 17 00:18:45 php 15099 [Snort] Snort START for LAN2(igb1)... Jul 17 00:18:45 php 15099 [Snort] Snort START for WAN1(igb0)... Jul 17 00:18:42 kernel igb1.101: promiscuous mode disabled Jul 17 00:18:42 kernel igb1: promiscuous mode disabled Jul 17 00:18:42 snort 77136 *** Caught Term-Signal Jul 17 00:18:41 php 15099 [Snort] Snort STOP for VLAN101(igb1.101)... Jul 17 00:18:40 snort 76714 *** Caught Term-Signal Jul 17 00:18:39 php 15099 [Snort] Snort STOP for LAN2(igb1)... Jul 17 00:18:38 kernel igb0: promiscuous mode disabled Jul 17 00:18:38 snort 76440 *** Caught Term-Signal Jul 17 00:18:38 php 15099 [Snort] Snort STOP for WAN1(igb0)... Jul 17 00:18:38 php 15099 [Snort] Building new sid-msg.map file for VLAN101... Jul 17 00:18:38 php 15099 [Snort] Enabling any flowbit-required rules for: VLAN101... Jul 17 00:18:37 php 15099 [Snort] Updating rules configuration for: VLAN101 ... Jul 17 00:18:37 php 15099 [Snort] Building new sid-msg.map file for LAN2... Jul 17 00:18:37 php 15099 [Snort] Enabling any flowbit-required rules for: LAN2... Jul 17 00:18:36 php 15099 [Snort] Updating rules configuration for: LAN2 ... Jul 17 00:18:36 php 15099 [Snort] Building new sid-msg.map file for WAN1... Jul 17 00:18:36 php 15099 [Snort] Enabling any flowbit-required rules for: WAN1... Jul 17 00:18:36 php 15099 [Snort] Checking flowbit rules dependent on disabled preprocessors for: WAN1... Jul 17 00:18:36 php 15099 [Snort] Enabling any flowbit-required rules for: WAN1... Jul 17 00:18:35 php 15099 [Snort] Checking for rules dependent on disabled preprocessors for: WAN1... Jul 17 00:18:33 php 15099 [Snort] Updating rules configuration for: WAN1 ... Jul 17 00:18:33 php 15099 [Snort] Removed 0 obsoleted rules category files. Jul 17 00:18:33 php 15099 [Snort] Hide Deprecated Rules is enabled. Removing obsoleted rules categories. Jul 17 00:18:33 php 15099 [Snort] Emerging Threats Open rules file update downloaded successfully Jul 17 00:18:33 php 15099 [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz... Jul 17 00:18:33 php 15099 [Snort] Snort GPLv2 Community Rules file update downloaded successfully Jul 17 00:18:32 php 15099 [Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz... Jul 17 00:18:32 php 15099 [Snort] Snort AppID Open Text Rules are up to date... Jul 17 00:18:31 php 15099 [Snort] Snort OpenAppID detectors are up to date... Jul 17 00:18:31 php 15099 [Snort] Snort Subscriber rules are up to date... Jul 17 00:05:00 php 53023 [Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb011268/... Jul 17 00:04:12 php 83106 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Jul 17 00:00:00 php 83106 [pfBlockerNG] Starting cron process. Jul 16 23:25:00 php 58765 [Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb111304/... Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_DNSBLIP_v4 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_Whitelist_v4 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_Level4_v4 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_Level3_v4 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_Level2_v4 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_Level1_v4 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v6 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating. Jul 16 12:30:38 php 55281 rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates Jul 16 12:30:00 php 55281 rc.update_urltables: /etc/rc.update_urltables: Sleeping for 38 seconds. Jul 16 12:30:00 php 55281 rc.update_urltables: /etc/rc.update_urltables: Starting up. Jul 16 01:01:04 php 54108 rc.dyndns.update: phpDynDNS (requests.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. -
@imburr said in Firewall *Stops* and Break in Logs For Extended Time:
Maybe something with Snort?
Remove "maybe" !?
Disable snort for a while and see for yourself.Btw : snort tend so create huge files.
What if there is no space left on the device ? That would explain the 'no more logs'. -
Those Snort log messages you posted are perfectly normal. Snort is going out and checking for new rules on the interval you configured, and when finding a new set, it installs them and restarts the Snort daemon. That's what the Snort messages are about. The "unknown AppID" messages are expected. Some of the AppID rules are old and reference the wrong names for certain services. It is of no consequence in the operation of pfSense, though.
Snort is pretty good now about controlling the amount of disk space it uses. You notice a couple of messages in the log snippet from Snort showing it cleaned up some aged-out log files. This is handled by the LOGS MGMT tab settings.
I see you also have pfBlocker installed. It can create lots of log files as well, depending on the specific configuration you are using. But I see nothing in your logs indicating the box ran out of space.
Are you perhaps using RAM Disks? If so, try turning that off. It is not necessary anymore with modern disk drives (even SSD models).
But as @Gertjan says, you can certainly disable Snort as a test to see if that makes any difference.
The fact you say unplugging your work PC immediately restored your network (with, I assume, no further action on your part), I wonder if you had an IP address conflict happening? Or perhaps your work PC has a physical issue with its Ethernet port and was causing a loopback condition with your network. Were the lights on your LAN switch all flashing wildly?
-
-
@bmeeks RAM Disks are not enabled. There could be a problem with the physical NIC on the work PC. The work PC uses WebTitan OTG, which is a DNS proxy comparable to Cisco Umbrella. Thats the only thing I can think of that would cause the problem. While the problem exists, I think that there are crazy lights on the switch, I remember that happening the last time I troubleshot this error several months ago.
What is a good way to troubleshoot this? While the system is down, the entire network is hosed- no pings can reach the firewall even if I set a static IP address. No wifi, no DHCP, nothing. As soon as I unplug my work PC and reboot pfSense and switch, everything comes back up correctly and re-engages.
I guess running a wireshark on the PC while the network is down would be a next step?
When I look at Windows event log during the outage (which was at 12:45 ish on 7/18/21), I see a gap in the logs like the machine was offline or hibernating:
The only thing I see is the 7011 (several hours before outage), but cannot find any details on how this would cause what happened.
-
The behavior you describe smells a lot like a network loop. That could be due to a physical issue in the Ethernet port on the Work PC if using a cabled connection. If wireless, then I would be investigating installed software and its settings on the Work PC.
From your description, the Work PC, or software installed and running on it, appears to be the culprit -- and not pfSense itself.
A network loop literally means the TX and RX sides are connected directly to each other. It's like a feedback loop in an audio system (that loud sqeal from a PA, for example). A loop condition will totally freeze a network. Sometimes simply unplugging the faulty device will restore order, but sometimes things need a reboot, too.
-
@bmeeks Very interesting! The work PC is connected via ethernet to a 8 port switch, which then has one homerun into the main 24 port switch, which then has one ethernet cable going to LAN on the pfSense. I cannot see for a way for it to be a physical loop, can a loop still happen in this instance, due to some misconfiguration or software issues?
-
@imburr said in Firewall *Stops* and Break in Logs For Extended Time:
@bmeeks Very interesting! The work PC is connected via ethernet to a 8 port switch, which then has one homerun into the main 24 port switch, which then has one ethernet cable going to LAN on the pfSense. I cannot see for a way for it to be a physical loop, can a loop still happen in this instance, due to some misconfiguration or software issues?
Most often I've seen network loops created by plugging one port into another on the same switch accidentally. It's also possible for something like bent pins on an RJ45 connector, or a faulty cable with an internal short, to cause a loop.
It's just that a completely dead network where all devices on it seem "frozen", only for everything to return to normal when you remove, or power down, one device, is strong evidence for a network loop of some sort caused by that device. In your case, that seems to be your work PC.
I would start simple by replacing the Ethernet cable for the work PC. Next, take a flashlight and carefully examine the pins inside the RJ45 Ethernet port on the PC's network connection. Look for two pins touching, or any that seem obviously bent or damaged.
