Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Firewall *Stops* and Break in Logs For Extended Time

    Scheduled Pinned Locked Moved General pfSense Questions
    8 Posts 3 Posters 824 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • I Offline
      Imburr
      last edited by

      I came home today and my network was down. Rebooted pfSense, nothing. Rebooted internet, nothing. Rebooted switch, nothing. Unplugged my work PC, and everything came back to life. Not sure what the issue is there, but while looking at the logs I am discovering 12+ hour blocks of no log messages. Can anyone see what might be causing the logs to stop receiving input and then chug forward again after half a day?

      Maybe something with Snort?

      Jul 18 00:18:24	kernel		igb0: promiscuous mode enabled
      Jul 18 00:18:22	php	59949	[Snort] The Rules update has finished.
      Jul 18 00:18:22	php	59949	[Snort] Snort has restarted with your new set of rules...
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 2314 is UNKNOWN
      Jul 18 00:18:20	snort	55453	Invalid direct service AppId, 2314, for 0x80809f590 0x80cd84b40
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 2312 is UNKNOWN
      Jul 18 00:18:20	snort	55453	Invalid direct service AppId, 2312, for 0x80809f590 0x80cd84b40
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4082 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4519 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4140 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4082 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4519 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4140 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4140 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 4314 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 3646 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppInfo: AppId 3646 is UNKNOWN
      Jul 18 00:18:20	snort	55453	AppId
      Jul 18 00:18:20	snort	55453	AppId
      Jul 18 00:18:20	snort	55453	AppId
      Jul 18 00:18:20	snort	55453	AppId
      Jul 18 00:18:20	snort	55453	AppId
      Jul 18 00:18:20	snort	55453	AppId
      Jul 18 00:18:19	kernel		igb1.101: promiscuous mode enabled
      Jul 18 00:18:19	kernel		igb1: promiscuous mode enabled
      Jul 18 00:18:19	snort	55453	AppId
      Jul 18 00:18:19	snort	55453	AppId
      Jul 18 00:18:19	snort	55453	AppId
      Jul 18 00:18:19	snort	55453	AppId
      Jul 18 00:18:19	snort	55453	AppId
      Jul 18 00:18:19	php	59949	[Snort] Snort START for VLAN101(igb1.101)...
      Jul 18 00:18:19	php	59949	[Snort] Snort START for LAN2(igb1)...
      Jul 18 00:18:19	php	59949	[Snort] Snort START for WAN1(igb0)...
      Jul 18 00:18:16	kernel		igb1.101: promiscuous mode disabled
      Jul 18 00:18:16	kernel		igb1: promiscuous mode disabled
      Jul 18 00:18:16	snort	27774	*** Caught Term-Signal
      Jul 18 00:18:15	php	59949	[Snort] Snort STOP for VLAN101(igb1.101)...
      Jul 18 00:18:14	snort	27479	*** Caught Term-Signal
      Jul 18 00:18:13	php	59949	[Snort] Snort STOP for LAN2(igb1)...
      Jul 18 00:18:12	kernel		igb0: promiscuous mode disabled
      Jul 18 00:18:12	snort	27042	*** Caught Term-Signal
      Jul 18 00:18:11	php	59949	[Snort] Snort STOP for WAN1(igb0)...
      Jul 18 00:18:11	php	59949	[Snort] Building new sid-msg.map file for VLAN101...
      Jul 18 00:18:11	php	59949	[Snort] Enabling any flowbit-required rules for: VLAN101...
      Jul 18 00:18:10	php	59949	[Snort] Updating rules configuration for: VLAN101 ...
      Jul 18 00:18:10	php	59949	[Snort] Building new sid-msg.map file for LAN2...
      Jul 18 00:18:10	php	59949	[Snort] Enabling any flowbit-required rules for: LAN2...
      Jul 18 00:18:09	php	59949	[Snort] Updating rules configuration for: LAN2 ...
      Jul 18 00:18:09	php	59949	[Snort] Building new sid-msg.map file for WAN1...
      Jul 18 00:18:09	php	59949	[Snort] Enabling any flowbit-required rules for: WAN1...
      Jul 18 00:18:09	php	59949	[Snort] Checking flowbit rules dependent on disabled preprocessors for: WAN1...
      Jul 18 00:18:09	php	59949	[Snort] Enabling any flowbit-required rules for: WAN1...
      Jul 18 00:18:07	php	59949	[Snort] Checking for rules dependent on disabled preprocessors for: WAN1...
      Jul 18 00:18:06	php	59949	[Snort] Updating rules configuration for: WAN1 ...
      Jul 18 00:18:06	php	59949	[Snort] Removed 0 obsoleted rules category files.
      Jul 18 00:18:06	php	59949	[Snort] Hide Deprecated Rules is enabled. Removing obsoleted rules categories.
      Jul 18 00:18:06	php	59949	[Snort] Emerging Threats Open rules are up to date...
      Jul 18 00:18:06	php	59949	[Snort] Snort GPLv2 Community Rules file update downloaded successfully
      Jul 18 00:18:05	php	59949	[Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
      Jul 18 00:18:05	php	59949	[Snort] Snort AppID Open Text Rules are up to date...
      Jul 18 00:18:05	php	59949	[Snort] Snort OpenAppID detectors are up to date...
      Jul 18 00:18:05	php	59949	[Snort] Snort Subscriber rules are up to date...
      Jul 18 00:03:40	php	78198	[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
      Jul 18 00:00:00	php	78198	[pfBlockerNG] Starting cron process.
      Jul 17 23:25:00	php	90300	[Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb111304/...
      Jul 17 23:25:00	php	90300	[Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb011268/...
      Jul 17 21:27:18	snort	27479	S5: Pruned 5 sessions from cache for memcap. 828 scbs remain. memcap: 8389036/8388608
      Jul 17 21:27:18	snort	27042	S5: Pruned 5 sessions from cache for memcap. 652 scbs remain. memcap: 8389072/8388608
      Jul 17 19:11:18	php-fpm	80722	/status_dhcp_leases.php: Successful login for user 'admin' from: 192.168.101.76 (Local Database)
      Jul 17 16:58:09	php-fpm	40582	/avahi_settings.php: Successful login for user 'admin' from: 192.168.101.76 (Local Database)
      Jul 17 16:16:50	php-fpm	92672	/index.php: Successful login for user 'admin' from: 192.168.101.76 (Local Database)
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_DNSBLIP_v4 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_Whitelist_v4 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_Level4_v4 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_Level3_v4 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_Level2_v4 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_Level1_v4 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v6 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating.
      Jul 17 12:30:55	php	27239	rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates
      Jul 17 12:30:00	php	27239	rc.update_urltables: /etc/rc.update_urltables: Sleeping for 55 seconds.
      Jul 17 12:30:00	php	27239	rc.update_urltables: /etc/rc.update_urltables: Starting up.
      Jul 17 01:01:03	php	19539	rc.dyndns.update: phpDynDNS (requests.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
      Jul 17 01:01:02	php	19539	rc.dyndns.update: phpDynDNS (plex.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
      Jul 17 01:01:01	php	19539	rc.dyndns.update: phpDynDNS (terraria.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
      Jul 17 01:01:00	php	19539	rc.dyndns.update: phpDynDNS (pfsense.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
      Jul 17 00:50:00	php	84916	[Snort] perfmon stats logs cleanup job removed 1 file(s) from /var/log/snort/snort_igb011268/...
      Jul 17 00:18:50	kernel		igb0: promiscuous mode enabled
      Jul 17 00:18:48	php	15099	[Snort] The Rules update has finished.
      Jul 17 00:18:48	php	15099	[Snort] Snort has restarted with your new set of rules...
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 2314 is UNKNOWN
      Jul 17 00:18:46	snort	27042	Invalid direct service AppId, 2314, for 0x80809f590 0x80cda4b40
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 2312 is UNKNOWN
      Jul 17 00:18:46	snort	27042	Invalid direct service AppId, 2312, for 0x80809f590 0x80cda4b40
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 3646 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 3646 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4082 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4519 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4140 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4082 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4519 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4140 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4140 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppInfo: AppId 4314 is UNKNOWN
      Jul 17 00:18:46	snort	27042	AppId
      Jul 17 00:18:46	snort	27042	AppId
      Jul 17 00:18:46	snort	27042	AppId
      Jul 17 00:18:46	snort	27042	AppId
      Jul 17 00:18:46	snort	27042	AppId
      Jul 17 00:18:46	snort	27042	AppId
      Jul 17 00:18:45	kernel		igb1.101: promiscuous mode enabled
      Jul 17 00:18:45	kernel		igb1: promiscuous mode enabled
      Jul 17 00:18:45	snort	27042	AppId
      Jul 17 00:18:45	snort	27042	AppId
      Jul 17 00:18:45	snort	27042	AppId
      Jul 17 00:18:45	snort	27042	AppId
      Jul 17 00:18:45	snort	27042	AppId
      Jul 17 00:18:45	php	15099	[Snort] Snort START for VLAN101(igb1.101)...
      Jul 17 00:18:45	php	15099	[Snort] Snort START for LAN2(igb1)...
      Jul 17 00:18:45	php	15099	[Snort] Snort START for WAN1(igb0)...
      Jul 17 00:18:42	kernel		igb1.101: promiscuous mode disabled
      Jul 17 00:18:42	kernel		igb1: promiscuous mode disabled
      Jul 17 00:18:42	snort	77136	*** Caught Term-Signal
      Jul 17 00:18:41	php	15099	[Snort] Snort STOP for VLAN101(igb1.101)...
      Jul 17 00:18:40	snort	76714	*** Caught Term-Signal
      Jul 17 00:18:39	php	15099	[Snort] Snort STOP for LAN2(igb1)...
      Jul 17 00:18:38	kernel		igb0: promiscuous mode disabled
      Jul 17 00:18:38	snort	76440	*** Caught Term-Signal
      Jul 17 00:18:38	php	15099	[Snort] Snort STOP for WAN1(igb0)...
      Jul 17 00:18:38	php	15099	[Snort] Building new sid-msg.map file for VLAN101...
      Jul 17 00:18:38	php	15099	[Snort] Enabling any flowbit-required rules for: VLAN101...
      Jul 17 00:18:37	php	15099	[Snort] Updating rules configuration for: VLAN101 ...
      Jul 17 00:18:37	php	15099	[Snort] Building new sid-msg.map file for LAN2...
      Jul 17 00:18:37	php	15099	[Snort] Enabling any flowbit-required rules for: LAN2...
      Jul 17 00:18:36	php	15099	[Snort] Updating rules configuration for: LAN2 ...
      Jul 17 00:18:36	php	15099	[Snort] Building new sid-msg.map file for WAN1...
      Jul 17 00:18:36	php	15099	[Snort] Enabling any flowbit-required rules for: WAN1...
      Jul 17 00:18:36	php	15099	[Snort] Checking flowbit rules dependent on disabled preprocessors for: WAN1...
      Jul 17 00:18:36	php	15099	[Snort] Enabling any flowbit-required rules for: WAN1...
      Jul 17 00:18:35	php	15099	[Snort] Checking for rules dependent on disabled preprocessors for: WAN1...
      Jul 17 00:18:33	php	15099	[Snort] Updating rules configuration for: WAN1 ...
      Jul 17 00:18:33	php	15099	[Snort] Removed 0 obsoleted rules category files.
      Jul 17 00:18:33	php	15099	[Snort] Hide Deprecated Rules is enabled. Removing obsoleted rules categories.
      Jul 17 00:18:33	php	15099	[Snort] Emerging Threats Open rules file update downloaded successfully
      Jul 17 00:18:33	php	15099	[Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
      Jul 17 00:18:33	php	15099	[Snort] Snort GPLv2 Community Rules file update downloaded successfully
      Jul 17 00:18:32	php	15099	[Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
      Jul 17 00:18:32	php	15099	[Snort] Snort AppID Open Text Rules are up to date...
      Jul 17 00:18:31	php	15099	[Snort] Snort OpenAppID detectors are up to date...
      Jul 17 00:18:31	php	15099	[Snort] Snort Subscriber rules are up to date...
      Jul 17 00:05:00	php	53023	[Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb011268/...
      Jul 17 00:04:12	php	83106	[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
      Jul 17 00:00:00	php	83106	[pfBlockerNG] Starting cron process.
      Jul 16 23:25:00	php	58765	[Snort] Alert tcpdump packet capture file cleanup job removed 1 tcpdump packet capture file(s) from /var/log/snort/snort_igb111304/...
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_DNSBLIP_v4 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_Whitelist_v4 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_Level4_v4 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_Level3_v4 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_Level2_v4 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_Level1_v4 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v6 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating.
      Jul 16 12:30:38	php	55281	rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates
      Jul 16 12:30:00	php	55281	rc.update_urltables: /etc/rc.update_urltables: Sleeping for 38 seconds.
      Jul 16 12:30:00	php	55281	rc.update_urltables: /etc/rc.update_urltables: Starting up.
      Jul 16 01:01:04	php	54108	rc.dyndns.update: phpDynDNS (requests.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
      
      GertjanG 1 Reply Last reply Reply Quote 0
      • GertjanG Offline
        Gertjan @Imburr
        last edited by

        @imburr said in Firewall *Stops* and Break in Logs For Extended Time:

        Maybe something with Snort?

        Remove "maybe" !?
        Disable snort for a while and see for yourself.

        Btw : snort tend so create huge files.
        What if there is no space left on the device ? That would explain the 'no more logs'.

        No "help me" PM's please. Use the forum, the community will thank you.
        Edit : and where are the logs ??

        I 1 Reply Last reply Reply Quote 0
        • bmeeksB Offline
          bmeeks
          last edited by bmeeks

          Those Snort log messages you posted are perfectly normal. Snort is going out and checking for new rules on the interval you configured, and when finding a new set, it installs them and restarts the Snort daemon. That's what the Snort messages are about. The "unknown AppID" messages are expected. Some of the AppID rules are old and reference the wrong names for certain services. It is of no consequence in the operation of pfSense, though.

          Snort is pretty good now about controlling the amount of disk space it uses. You notice a couple of messages in the log snippet from Snort showing it cleaned up some aged-out log files. This is handled by the LOGS MGMT tab settings.

          I see you also have pfBlocker installed. It can create lots of log files as well, depending on the specific configuration you are using. But I see nothing in your logs indicating the box ran out of space.

          Are you perhaps using RAM Disks? If so, try turning that off. It is not necessary anymore with modern disk drives (even SSD models).

          But as @Gertjan says, you can certainly disable Snort as a test to see if that makes any difference.

          The fact you say unplugging your work PC immediately restored your network (with, I assume, no further action on your part), I wonder if you had an IP address conflict happening? Or perhaps your work PC has a physical issue with its Ethernet port and was causing a loopback condition with your network. Were the lights on your LAN switch all flashing wildly?

          I 1 Reply Last reply Reply Quote 0
          • I Offline
            Imburr @Gertjan
            last edited by

            @gertjan SNORT is in non-blocking mode only:

            22011efb-2593-4406-bc42-59df6db1bc48-image.png

            And as far as I can tell disk space is good to go:

            0d8f5b28-702d-4d45-8aae-da163f9e9fee-image.png

            1 Reply Last reply Reply Quote 0
            • I Offline
              Imburr @bmeeks
              last edited by

              @bmeeks RAM Disks are not enabled. There could be a problem with the physical NIC on the work PC. The work PC uses WebTitan OTG, which is a DNS proxy comparable to Cisco Umbrella. Thats the only thing I can think of that would cause the problem. While the problem exists, I think that there are crazy lights on the switch, I remember that happening the last time I troubleshot this error several months ago.

              What is a good way to troubleshoot this? While the system is down, the entire network is hosed- no pings can reach the firewall even if I set a static IP address. No wifi, no DHCP, nothing. As soon as I unplug my work PC and reboot pfSense and switch, everything comes back up correctly and re-engages.

              I guess running a wireshark on the PC while the network is down would be a next step?

              When I look at Windows event log during the outage (which was at 12:45 ish on 7/18/21), I see a gap in the logs like the machine was offline or hibernating:

              435c27b3-2431-41a6-bbd8-4a865f338472-image.png

              7e4fc638-f4aa-4c40-9058-96a2df8a3a57-image.png

              The only thing I see is the 7011 (several hours before outage), but cannot find any details on how this would cause what happened.

              1 Reply Last reply Reply Quote 0
              • bmeeksB Offline
                bmeeks
                last edited by bmeeks

                The behavior you describe smells a lot like a network loop. That could be due to a physical issue in the Ethernet port on the Work PC if using a cabled connection. If wireless, then I would be investigating installed software and its settings on the Work PC.

                From your description, the Work PC, or software installed and running on it, appears to be the culprit -- and not pfSense itself.

                A network loop literally means the TX and RX sides are connected directly to each other. It's like a feedback loop in an audio system (that loud sqeal from a PA, for example). A loop condition will totally freeze a network. Sometimes simply unplugging the faulty device will restore order, but sometimes things need a reboot, too.

                I 1 Reply Last reply Reply Quote 0
                • I Offline
                  Imburr @bmeeks
                  last edited by

                  @bmeeks Very interesting! The work PC is connected via ethernet to a 8 port switch, which then has one homerun into the main 24 port switch, which then has one ethernet cable going to LAN on the pfSense. I cannot see for a way for it to be a physical loop, can a loop still happen in this instance, due to some misconfiguration or software issues?

                  bmeeksB 1 Reply Last reply Reply Quote 0
                  • bmeeksB Offline
                    bmeeks @Imburr
                    last edited by

                    @imburr said in Firewall *Stops* and Break in Logs For Extended Time:

                    @bmeeks Very interesting! The work PC is connected via ethernet to a 8 port switch, which then has one homerun into the main 24 port switch, which then has one ethernet cable going to LAN on the pfSense. I cannot see for a way for it to be a physical loop, can a loop still happen in this instance, due to some misconfiguration or software issues?

                    Most often I've seen network loops created by plugging one port into another on the same switch accidentally. It's also possible for something like bent pins on an RJ45 connector, or a faulty cable with an internal short, to cause a loop.

                    It's just that a completely dead network where all devices on it seem "frozen", only for everything to return to normal when you remove, or power down, one device, is strong evidence for a network loop of some sort caused by that device. In your case, that seems to be your work PC.

                    I would start simple by replacing the Ethernet cable for the work PC. Next, take a flashlight and carefully examine the pins inside the RJ45 Ethernet port on the PC's network connection. Look for two pins touching, or any that seem obviously bent or damaged.

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.