Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Problems With WAN Loss Cobnection

    Scheduled Pinned Locked Moved General pfSense Questions
    57 Posts 4 Posters 2.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dcuadrados @Gertjan
      last edited by

      @Gertjan You're right, I have it enabled—it was carried over from another firewall backup that used a different driver. I'm going to disable it. That said, one of the devices with the same issue is using an em driver. Still, I’ve already disabled it, and I’ll reboot as soon as I can for the changes to take effect.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        That setting only does anything if you have hn NICs. So only in HyperV or Azure.

        igc does support altq, it should be fine.

        That CPU usage is all in the NIC. It looks like it's trying to push a load of traffic. Check the traffic and packet graphs in Status > Monitoring at the time. Is there a huge spike in traffic?

        Try to get a pcap of it at the time if you can. You might have something creating a loop or some sort of reflected traffic.

        @dcuadrados said in Problems With WAN Loss Cobnection:

        they are independent devices in different locations—they have nothing to do with each other.

        Sorry what I meant there was are all three locations using the same ISP? With the same type of ISP supplied router?
        Are all three using double NAT?

        D 1 Reply Last reply Reply Quote 0
        • D
          dcuadrados @stephenw10
          last edited by dcuadrados

          @stephenw10 said in Problems With WAN Loss Cobnection:

          Ese uso de CPU está todo en la NIC. Parece como si estuviera intentando impulsar una gran cantidad de tráfico. Verifique los gráficos de tráfico y paquetes en Estado > Monitoreo en ese momento. ¿Hay un gran aumento en el tráfico?

          I have the Grafana data, and there is indeed a traffic increase beforehand, followed by a noticeable drop at that moment. It happens around 11:00 PM, as shown, and at that time no one is working in the office. The Acronis Online Backup runs at 11:45 PM, and the loss begins around 11:30 PM, but I don’t see anything unusual aside from that. Additionally, all devices on the network are shut down at 10:30 PM by policy, and only the server remains on.

          fd2f7023-6c3a-4557-a4b7-df760e8f9dc6-image.png

          1f215657-fc65-4565-9753-fa6e7f2e17ac-image.png
          13fbc4e2-3998-44e7-89b8-96119110bb7b-image.png

          @stephenw10 said in Problems With WAN Loss Cobnection:

          Perdón, ¿qué quise decir con que había tres ubicaciones que usaban el mismo ISP? Con lo mismo tipo ¿del enrutador suministrado por el ISP?
          ¿Los tres utilizan NAT doble?

          All three use double NAT, and they’re with different ISPs, but in all cases I pass the traffic from the router to the firewall. As I mentioned, I have dozens of pfSense setups configured this way, but the ones failing are these—specifically the ones with i225 and i226 NICs.

          The difference with these setups is that I use:

          • Snort
          • Floating rules
          • pfBlocker
          • Telegraf

          The others are older and only use outbound hardening (secure outbound rules), VPN, IPsec... and they work flawlessly. All of those use em drivers, don’t use Snort, have regular per-interface rules, and I don’t monitor them with Influx + Grafana.

          And this is what I have in the cron jobs—nothing unusual as far as I can tell.
          Could this be the cause?

          0,15,30,45	*	*	*	*	root	/etc/rc.filter_configure_sync
          
          
          minute	hour	mday	month	wday	who	command	
          */1	*	*	*	*	root	/usr/sbin/newsyslog	  
          1	3	*	*	*	root	/etc/rc.periodic daily	  
          15	4	*	*	6	root	/etc/rc.periodic weekly	  
          30	5	1	*	*	root	/etc/rc.periodic monthly	  
          1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a	  
          1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh	  
          1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update	  
          */60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot	  
          30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables	  
          1	0	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_pkg_metadata	  
          */15	*	*	*	*	root	/usr/local/bin/php /root/scripts/pfsense_zbx.php speedtest_cron	  
          0,15,30,45	*	*	*	*	root	/etc/rc.filter_configure_sync	  
          *	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php /usr/local/sbin/acbupload.php	  
          0	16	*	*	7	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php bl shallalist,ut1 >> /var/log/pfblockerng/extras.log 2>&1	  
          36	0	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php /usr/local/sbin/execacb.php	  
          0	14	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1	  
          */1	*	*	*	*	root	/usr/local/pkg/servicewatchdog_cron.php	  
          16	3	*	*	*	root	/usr/local/pkg/acme/acme_command.sh "renewall" | /usr/bin/logger -t ACME 2>&1	  
          0	0	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1	  
          */5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc	  
          */5	*	*	*	*	root	/usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 1800	  
          20	2	*/28	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php	  
          

          EDITED:

          Alright, I just reviewed the last 30 days of Grafana data from the problematic firewalls, and I noticed that the WAN always drops at X:00, X:15, X:30, and X:45—so it clearly matches the reload that happens every 15 minutes.

          I’ve read that this can be caused by rules that use schedules, and in my case, I do use schedules for some rules:

          1. Certificate renewal — I haven’t switched all of them to ACME via API yet
          2. Some backup tasks
          3. Limiters in some firewalls for bandwidth management

          Could the issue be related to this?
          Is it really necessary to perform this reload every 15 minutes?

          GertjanG stephenw10S 2 Replies Last reply Reply Quote 0
          • GertjanG
            Gertjan @dcuadrados
            last edited by Gertjan

            @dcuadrados said in Problems With WAN Loss Cobnection:

            and I noticed that the WAN always drops at X:00, X:15, X:30, and X:45—so it clearly matches the reload that happens every 15 minutes.

            Yeah, you've shown it already :

            60249f07-6171-4e9c-a476-37c416852a2d-image.png

            Where did that cron task come from ?

            Btw :
            I can't figure out a good reason why a backup should do "do" things with interfaces and/or their connection.
            Neither for limiters.
            Certificate renewal is this one :

            4de4263a-9fb8-4e62-9c39-c134c94715cd-image.png

            and doesn't does nothing with interfaces.

            This one :

            347fce62-c8f8-43b2-80b0-406ef3df7971-image.png

            is the anti-quality-of-live pfSense package. It creates problems, hard to find issues and constant pain.

            This one :

            8672371b-8b0d-4ddb-9a41-b4c3c3811946-image.png

            isn't ideal neither ... Really, checking the speed of wan every 15 minutes ?
            in reality, the bandwidth available is always known : it's the bandwidth the ISP gave you, minus what you are using at that moment.

            No "help me" PM's please. Use the forum, the community will thank you.
            Edit : and where are the logs ??

            D 1 Reply Last reply Reply Quote 0
            • D
              dcuadrados @Gertjan
              last edited by dcuadrados

              @Gertjan said in Problems With WAN Loss Cobnection:

              It's automatically generated by the scheduler—that is, when I have rules enabled for specific time periods...
              When I remove the schedulers, it disappears.

              My question is: can it be deleted manually? Will it be recreated after a reboot?

              Supposedly, according to this, it should be resolved in these FreeBSD versions:
              https://redmine.pfsense.org/issues/10414

              #!/usr/local/bin/php-cgi -f
              <?php
              /*
               * rc.filter_configure_sync
               *
               * part of pfSense (https://www.pfsense.org)
               * Copyright (c) 2004-2013 BSD Perimeter
               * Copyright (c) 2013-2016 Electric Sheep Fencing
               * Copyright (c) 2014-2025 Rubicon Communications, LLC (Netgate)
               * All rights reserved.
               *
               * Licensed under the Apache License, Version 2.0 (the "License");
               * you may not use this file except in compliance with the License.
               * You may obtain a copy of the License at
               *
               * http://www.apache.org/licenses/LICENSE-2.0
               *
               * Unless required by applicable law or agreed to in writing, software
               * distributed under the License is distributed on an "AS IS" BASIS,
               * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
               * See the License for the specific language governing permissions and
               * limitations under the License.
               */
              
              require_once("config.inc");
              require_once("functions.inc");
              require_once("filter.inc");
              require_once("shaper.inc");
              require_once("ipsec.inc");
              require_once("vpn.inc");
              
              filter_configure_sync();
              
              ?>
              
              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator @dcuadrados
                last edited by

                @dcuadrados said in Problems With WAN Loss Cobnection:

                but the ones failing are these—specifically the ones with i225 and i226 NICs.

                What version of the NIC chips are they using? The early revision i225s especially were known for dropping connections. But I assume here you don't actually see the link loss?

                [25.03-BETA][admin@4100-2.stevew.lan]/root: pciconf -lv igc0
                igc0@pci0:4:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
                    vendor     = 'Intel Corporation'
                    device     = 'Ethernet Controller I225-V'
                    class      = network
                    subclass   = ethernet
                

                There you can see it's a rev.3 chip which had fixed the earlier issues.

                D 1 Reply Last reply Reply Quote 0
                • D
                  dcuadrados @stephenw10
                  last edited by dcuadrados

                  @stephenw10 said in Problems With WAN Loss Cobnection:

                  ¿Qué versión de los chips NIC están utilizando? Los primeros i225 de revisión eran especialmente conocidos por perder conexiones. ¿Pero supongo que aquí en realidad no ves la pérdida del enlace?

                  It seems that after removing that cron job, the firewall has completely stabilized. If you look at the graphs, there used to be spikes every 15 minutes—both on the WAN and CPU usage—and now they’re gone. I’m attaching the last 24 hours of graphs below. Now the key is to understand why that cron job is generated and whether it’s actually necessary.

                  As for the driver, it’s using the following version:

                  igc0@pci0:1:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
                      vendor     = 'Intel Corporation'
                      device     = 'Ethernet Controller I225-V'
                      class      = network
                      subclass   = ethernet
                  

                  @Gertjan said in Problems With WAN Loss Cobnection:

                  Este:

                  tampoco es ideal... ¿En serio, comprobar la velocidad de wan cada 15 minutos?
                  En realidad, el ancho de banda disponible siempre se conoce: es el ancho de banda que te dio el ISP, menos lo que estás usando en ese momento.

                  Regarding what you mentioned about the speed-related cron job, that was something I added to test a few things in Zabbix—but I don’t even use Zabbix anymore, since I now do monitoring with Grafana and Wazuh. I’ve removed those as well, and now I need to move all my ACME setups to API-based renewal using token authentication instead of relying on port-based renewal.

                  I’ll monitor how it behaves over the weekend, but it seems like we’ve made some progress. Huge thanks to both of you! I’ll keep you posted.

                  @Gertjan said in Problems With WAN Loss Cobnection:

                  Sí, ya lo has demostrado:

                  ¿De dónde surgió esa tarea cron?

                  As I mentioned earlier, this happens when schedules are used for the rules. It seems that every 15 minutes it kills all states to enforce the schedule and the associated rules.

                  e9f34c8e-6eb7-4e1e-9d27-3e2eb31b3a32-image.png

                  fbf97415-d03d-410c-ae22-23d9fcaf4010-image.png

                  bfe608d7-7d41-4f08-b499-48f45d01802f-image.png

                  21f6cf5f-a8b2-4dd1-99fd-06986853fde3-image.png

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    The filter configure sync cron job shouldn't cause a significant issue. It certainly shouldn't cause a complete loss of connectivity.

                    But let's see how it goes....

                    D 1 Reply Last reply Reply Quote 0
                    • D
                      dcuadrados @stephenw10
                      last edited by

                      @stephenw10 i see here the same problem https://redmine.pfsense.org/issues/10414#note-10

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        That was a very specific issue in pf and it was fixed 5 years ago. I assume you're not running 2.4.4?

                        D 1 Reply Last reply Reply Quote 0
                        • D
                          dcuadrados @stephenw10
                          last edited by dcuadrados

                          @stephenw10 said in Problems With WAN Loss Cobnection:

                          That was a very specific issue in pf and it was fixed 5 years ago. I assume you're not running 2.4.4?

                          Responder

                          No, I'm obviously not running that version, but as I mentioned, by analyzing the Grafana data, I noticed that the WAN always drops exactly when that cron job runs—at X:00, X:15, X:30, or X:45. So it clearly coincides with the cron execution.

                          Of course, I’m not considering the issue fully resolved yet. Right now, I’m monitoring to see if the problem happens again so I can gather more information. I’ll keep you updated over the next few days on how things go.

                          And interestingly, the firewalls that don’t experience any drops don’t have that cron job at all.

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            And definitely not that speedtest cronjob? That would be my first suspect.

                            D 1 Reply Last reply Reply Quote 0
                            • D
                              dcuadrados @stephenw10
                              last edited by

                              @stephenw10 said in Problems With WAN Loss Cobnection:

                              ¿Y definitivamente no ese cronjob de prueba de velocidad? Ese sería mi primer sospechoso.

                              That one wasn’t present on all systems—some had it, but others didn’t—so I don’t think that was the root cause. Still, I’ve cleaned up the cron jobs to leave only the essential ones, and for now, everything seems stable.

                              1 Reply Last reply Reply Quote 1
                              • D
                                dcuadrados
                                last edited by dcuadrados

                                I’m bringing an update: everything is running better now, with one exception. When Snort updates, something similar happens—the system becomes heavily loaded and somewhat unresponsive. I’ve noticed that two Snort processes remain running, and if it updates again, a third one appears. This causes the system to become overloaded, even though there should only be one Snort process running, since it's configured only on the LAN in Inline IPS mode.

                                I just ran the test right now.

                                Here are the logs shown in System:

                                Jun 29 18:34:00	php-cgi	91062	servicewatchdog_cron.php: Service Watchdog detected service telegraf stopped. Restarting telegraf (Telegraf daemon)
                                Jun 29 18:33:26	php_pfb	79326	[pfBlockerNG] filterlog daemon started
                                Jun 29 18:33:25	tail_pfb	79015	[pfBlockerNG] Firewall Filter Service started
                                Jun 29 18:33:25	lighttpd_pfb	76804	[pfBlockerNG] DNSBL Webserver started
                                Jun 29 18:33:25	lighttpd_pfb	74750	[pfBlockerNG] DNSBL Webserver stopped
                                Jun 29 18:33:25	php_pfb	73650	[pfBlockerNG] filterlog daemon stopped
                                Jun 29 18:33:25	tail_pfb	73212	[pfBlockerNG] Firewall Filter Service stopped
                                Jun 29 18:33:19	php-fpm	57400	/rc.start_packages: Restarting/Starting all packages.
                                Jun 29 18:33:18	check_reload_status	476	Reloading filter
                                Jun 29 18:33:18	check_reload_status	476	Starting packages
                                Jun 29 18:33:18	php-fpm	87641	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages.
                                Jun 29 18:33:16	php-fpm	87641	/rc.newwanip: Creating rrd update script
                                Jun 29 18:33:16	php-fpm	87641	/rc.newwanip: Resyncing OpenVPN instances for interface LAN.
                                Jun 29 18:33:16	php-fpm	87641	/rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan
                                Jun 29 18:33:13	php-fpm	87641	/rc.newwanip: Gateway, NONE AVAILABLE
                                Jun 29 18:33:12	php-fpm	87641	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.0.1
                                Jun 29 18:33:00	php-cgi	37854	servicewatchdog_cron.php: Service Watchdog detected service telegraf stopped. Restarting telegraf (Telegraf daemon)
                                Jun 29 18:32:59	php-fpm	87641	/rc.newwanip: rc.newwanip: on (IP address: 192.168.2.1) (interface: LAN[lan]) (real interface: em1).
                                Jun 29 18:32:59	php-fpm	87641	/rc.newwanip: rc.newwanip: Info: starting on em1.
                                Jun 29 18:32:58	check_reload_status	476	Reloading filter
                                Jun 29 18:32:58	check_reload_status	476	rc.newwanip starting em1
                                Jun 29 18:32:58	php-fpm	33297	/rc.linkup: HOTPLUG: Triggering address refresh on lan (em1)
                                Jun 29 18:32:58	php-fpm	33297	/rc.linkup: DEVD Ethernet attached event for lan
                                Jun 29 18:32:58	php-fpm	33297	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                Jun 29 18:32:58	check_reload_status	476	Reloading filter
                                Jun 29 18:32:56	php_pfb	12657	[pfBlockerNG] filterlog daemon started
                                Jun 29 18:32:56	tail_pfb	12409	[pfBlockerNG] Firewall Filter Service started
                                Jun 29 18:32:56	lighttpd_pfb	12380	[pfBlockerNG] DNSBL Webserver started
                                Jun 29 18:32:56	lighttpd_pfb	9243	[pfBlockerNG] DNSBL Webserver stopped
                                Jun 29 18:32:56	php_pfb	7656	[pfBlockerNG] filterlog daemon stopped
                                Jun 29 18:32:56	tail_pfb	7195	[pfBlockerNG] Firewall Filter Service stopped
                                Jun 29 18:32:52	kernel		em1: link state changed to UP
                                Jun 29 18:32:52	check_reload_status	476	Linkup starting em1
                                Jun 29 18:32:50	php-fpm	11935	/rc.start_packages: Restarting/Starting all packages.
                                Jun 29 18:32:49	php-fpm	89037	/rc.linkup: DEVD Ethernet detached event for lan
                                Jun 29 18:32:49	php-fpm	89037	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                Jun 29 18:32:49	check_reload_status	476	Reloading filter
                                Jun 29 18:32:49	check_reload_status	476	Starting packages
                                Jun 29 18:32:49	php-fpm	57400	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages.
                                Jun 29 18:32:48	kernel		em1: link state changed to DOWN
                                Jun 29 18:32:48	check_reload_status	476	Linkup starting em1
                                Jun 29 18:32:48	kernel		768.603146 [ 852] iflib_netmap_config txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
                                Jun 29 18:32:47	php-fpm	57400	/rc.newwanip: Creating rrd update script
                                Jun 29 18:32:47	php-fpm	57400	/rc.newwanip: Resyncing OpenVPN instances for interface LAN.
                                Jun 29 18:32:47	php-fpm	57400	/rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan
                                Jun 29 18:32:42	php-fpm	57400	/rc.newwanip: Gateway, NONE AVAILABLE
                                Jun 29 18:32:42	php-fpm	57400	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.0.1
                                Jun 29 18:32:24	php-fpm	57400	/rc.newwanip: rc.newwanip: on (IP address: 192.168.2.1) (interface: LAN[lan]) (real interface: em1).
                                Jun 29 18:32:24	php-fpm	57400	/rc.newwanip: rc.newwanip: Info: starting on em1.
                                Jun 29 18:32:23	check_reload_status	476	Reloading filter
                                Jun 29 18:32:23	check_reload_status	476	rc.newwanip starting em1
                                Jun 29 18:32:23	php-fpm	87641	/rc.linkup: HOTPLUG: Triggering address refresh on lan (em1)
                                Jun 29 18:32:23	php-fpm	87641	/rc.linkup: DEVD Ethernet attached event for lan
                                Jun 29 18:32:23	php-fpm	87641	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                Jun 29 18:32:22	kernel		em1: link state changed to UP
                                Jun 29 18:32:22	check_reload_status	476	Linkup starting em1
                                Jun 29 18:32:21	check_reload_status	476	Reloading filter
                                Jun 29 18:32:19	php-fpm	11935	/rc.linkup: DEVD Ethernet detached event for lan
                                Jun 29 18:32:19	php-fpm	11935	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                Jun 29 18:32:18	kernel		em1: link state changed to DOWN
                                Jun 29 18:32:18	check_reload_status	476	Linkup starting em1
                                Jun 29 18:32:18	php	37179	[Snort] The Rules update has finished.
                                Jun 29 18:32:18	php	37179	[Snort] Snort has restarted on LAN with your new set of rules...
                                Jun 29 18:32:18	kernel		738.705442 [ 852] iflib_netmap_config txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
                                Jun 29 18:32:00	php-cgi	78632	servicewatchdog_cron.php: Service Watchdog detected service telegraf stopped. Restarting telegraf (Telegraf daemon)
                                Jun 29 18:31:22	php_pfb	61839	[pfBlockerNG] filterlog daemon started
                                Jun 29 18:31:22	tail_pfb	61431	[pfBlockerNG] Firewall Filter Service started
                                Jun 29 18:31:22	lighttpd_pfb	59064	[pfBlockerNG] DNSBL Webserver started
                                Jun 29 18:31:22	lighttpd_pfb	57084	[pfBlockerNG] DNSBL Webserver stopped
                                Jun 29 18:31:22	php_pfb	56006	[pfBlockerNG] filterlog daemon stopped
                                Jun 29 18:31:22	tail_pfb	55454	[pfBlockerNG] Firewall Filter Service stopped
                                Jun 29 18:31:13	SnortStartup	93013	Snort START for LAN(em1)...
                                Jun 29 18:31:12	php-fpm	11935	/rc.start_packages: Restarting/Starting all packages.
                                Jun 29 18:31:11	check_reload_status	476	Reloading filter
                                Jun 29 18:31:11	check_reload_status	476	Starting packages
                                Jun 29 18:31:11	php-fpm	40346	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages.
                                Jun 29 18:31:09	php-fpm	40346	/rc.newwanip: Creating rrd update script
                                Jun 29 18:31:09	php-fpm	40346	/rc.newwanip: Resyncing OpenVPN instances for interface LAN.
                                Jun 29 18:31:09	php-fpm	40346	/rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan
                                Jun 29 18:31:04	php-fpm	40346	/rc.newwanip: Gateway, NONE AVAILABLE
                                Jun 29 18:31:04	php-fpm	40346	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.0.1
                                Jun 29 18:30:49	php-fpm	40346	/rc.newwanip: rc.newwanip: on (IP address: 192.168.2.1) (interface: LAN[lan]) (real interface: em1).
                                Jun 29 18:30:49	php-fpm	40346	/rc.newwanip: rc.newwanip: Info: starting on em1.
                                Jun 29 18:30:48	check_reload_status	476	Reloading filter
                                Jun 29 18:30:48	check_reload_status	476	rc.newwanip starting em1
                                Jun 29 18:30:48	php-fpm	40346	/rc.linkup: HOTPLUG: Triggering address refresh on lan (em1)
                                Jun 29 18:30:48	php-fpm	40346	/rc.linkup: DEVD Ethernet attached event for lan
                                Jun 29 18:30:48	php-fpm	40346	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                Jun 29 18:30:47	kernel		em1: link state changed to UP
                                Jun 29 18:30:47	check_reload_status	476	Linkup starting em1
                                Jun 29 18:30:46	check_reload_status	476	Reloading filter
                                Jun 29 18:30:45	snort	42755	AppInfo: AppId 5349 is UNKNOWN
                                Jun 29 18:30:45	snort	42755	Invalid direct service AppId, 5349, for 0x842745c70 0x1aa0e5f24800
                                Jun 29 18:30:45	snort	42755	AppInfo: AppId 4903 is UNKNOWN
                                Jun 29 18:30:45	snort	42755	AppId
                                Jun 29 18:30:45	snort	42755	AppId
                                Jun 29 18:30:44	snort	42755	AppId
                                Jun 29 18:30:44	php-fpm	11935	/rc.linkup: DEVD Ethernet detached event for lan
                                Jun 29 18:30:44	php-fpm	11935	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                Jun 29 18:30:44	php	37179	[Snort] Snort START for LAN(em1)...
                                Jun 29 18:30:43	kernel		em1: link state changed to DOWN
                                Jun 29 18:30:43	check_reload_status	476	Linkup starting em1
                                Jun 29 18:30:43	snort	98159	*** Caught Term-Signal
                                Jun 29 18:30:42	php	37179	[Snort] Snort STOP for LAN(em1)...
                                Jun 29 18:30:41	php	37179	[Snort] Building new sid-msg.map file for LAN...
                                Jun 29 18:30:41	php	37179	[Snort] Enabling any flowbit-required rules for: LAN...
                                Jun 29 18:30:41	php	37179	[Snort] Error - unable to find reject_sid list "none" specified for LAN
                                Jun 29 18:30:41	php	37179	[Snort] Error - unable to find drop_sid list "none" specified for LAN
                                Jun 29 18:30:38	php	37179	[Snort] Updating rules configuration for: LAN ...
                                Jun 29 18:30:38	php	37179	[Snort] Removed 0 obsoleted rules category files.
                                Jun 29 18:30:38	php	37179	[Snort] Hide Deprecated Rules is enabled. Removing obsoleted rules categories.
                                Jun 29 18:30:38	php	37179	[Snort] Feodo Tracker Botnet C2 IP rules were updated...
                                Jun 29 18:30:38	php	37179	[Snort] Feodo Tracker Botnet C2 IP rules file update downloaded successfully.
                                Jun 29 18:30:38	php	37179	[Snort] Emerging Threats Open rules are up to date...
                                Jun 29 18:30:38	php	37179	[Snort] Snort GPLv2 Community Rules are up to date...
                                Jun 29 18:30:37	php	37179	[Snort] Snort AppID Open Text Rules are up to date...
                                Jun 29 18:30:36	php	37179	[Snort] Snort OpenAppID detectors are up to date...
                                Jun 29 18:30:36	php	37179	[Snort] Snort Subscriber rules are up to date...
                                

                                Here you can see that Snort’s log shows the update completes very quickly:

                                Starting rules update...  Time: 2025-06-29 18:30:35
                                	Downloading Snort Subscriber rules md5 file snortrules-snapshot-29200.tar.gz.md5...
                                	Checking Snort Subscriber rules md5 file...
                                	Snort Subscriber rules are up to date.
                                	Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
                                	Checking Snort OpenAppID detectors md5 file...
                                	Snort OpenAppID detectors are up to date.
                                	Downloading Snort AppID Open Text Rules md5 file appid_rules.tar.gz.md5...
                                	Checking Snort AppID Open Text Rules md5 file...
                                	Snort AppID Open Text Rules are up to date.
                                	Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
                                	Checking Snort GPLv2 Community Rules md5 file...
                                	Snort GPLv2 Community Rules are up to date.
                                	Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
                                	Checking Emerging Threats Open rules md5 file...
                                	Emerging Threats Open rules are up to date.
                                	Downloading Feodo Tracker Botnet C2 IP rules file...
                                	Done downloading rules file.
                                	Extracting and installing Feodo Tracker Botnet C2 IP rules...
                                	Feodo Tracker Botnet C2 IP rules were updated.
                                	Copying new config and map files...
                                	Updating rules configuration for: LAN ...
                                	Restarting Snort on LAN to activate the new set of rules...
                                	Snort has restarted on LAN with your new set of rules.
                                The Rules update has finished.  Time: 2025-06-29 18:32:18
                                
                                

                                And the gateway:

                                Jun 29 18:43:39	dpinger	16187	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP "
                                Jun 29 18:43:39	dpinger	22451	exiting on signal 15
                                Jun 29 18:42:11	dpinger	22451	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP "
                                Jun 29 18:42:11	dpinger	32634	exiting on signal 15
                                Jun 29 18:41:55	dpinger	32634	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP "
                                Jun 29 18:41:55	dpinger	62920	exiting on signal 15
                                Jun 29 18:40:30	dpinger	62920	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP "
                                Jun 29 18:40:30	dpinger	61345	exiting on signal 15
                                Jun 29 18:33:13	dpinger	61345	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP "
                                Jun 29 18:33:12	dpinger	86243	exiting on signal 15
                                Jun 29 18:32:42	dpinger	86243	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP "
                                Jun 29 18:32:42	dpinger	26696	exiting on signal 15
                                Jun 29 18:31:04	dpinger	26696	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP "
                                Jun 29 18:31:04	dpinger	78346	exiting on signal 15
                                

                                It loses connectivity and everything starts behaving erratically.

                                0b39eb95-0732-42e0-9c64-4821c8380642-image.png

                                d9bbc902-985a-45c7-81a5-e445aab48978-image.png

                                Logs from anothe:

                                2025-06-29 18:15:37.362731+02:00	xinetd	5473	Swapping defaults
                                2025-06-29 18:15:37.362131+02:00	xinetd	5473	Starting reconfiguration
                                2025-06-29 18:15:31.770460+02:00	kernel	-	arp: 08:00:27:66:1d:d1 is using my IP address 192.168.255.254 on igc1!
                                2025-06-29 18:15:31.497019+02:00	php-fpm	38706	/rc.newwanip: rc.newwanip: on (IP address: 172.16.0.1) (interface: LAN[lan]) (real interface: igc1).
                                2025-06-29 18:15:31.496546+02:00	php-fpm	38706	/rc.newwanip: rc.newwanip: Info: starting on igc1.
                                2025-06-29 18:15:30.422269+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:15:30.416622+02:00	check_reload_status	674	rc.newwanip starting igc1
                                2025-06-29 18:15:30.416106+02:00	php-fpm	38706	/rc.linkup: HOTPLUG: Triggering address refresh on lan (igc1)
                                2025-06-29 18:15:30.415927+02:00	php-fpm	38706	/rc.linkup: DEVD Ethernet attached event for lan
                                2025-06-29 18:15:30.404027+02:00	php-fpm	38706	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1)
                                2025-06-29 18:15:29.755530+02:00	xinetd	5473	Reconfigured: new=0 old=6 dropped=0 (services)
                                2025-06-29 18:15:29.755518+02:00	xinetd	5473	readjusting service 19005-udp
                                2025-06-29 18:15:29.755507+02:00	xinetd	5473	readjusting service 19004-udp
                                2025-06-29 18:15:29.755495+02:00	xinetd	5473	readjusting service 19003-tcp
                                2025-06-29 18:15:29.755484+02:00	xinetd	5473	readjusting service 19002-tcp
                                2025-06-29 18:15:29.755473+02:00	xinetd	5473	readjusting service 19001-tcp
                                2025-06-29 18:15:29.755460+02:00	xinetd	5473	readjusting service 19000-tcp
                                2025-06-29 18:15:29.755443+02:00	xinetd	5473	Swapping defaults
                                2025-06-29 18:15:29.754868+02:00	xinetd	5473	Starting reconfiguration
                                2025-06-29 18:15:29.351604+02:00	kernel	-	igc1: link state changed to UP
                                2025-06-29 18:15:29.321806+02:00	check_reload_status	674	Linkup starting igc1
                                2025-06-29 18:15:28.610587+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:15:28.610258+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:15:26.838367+02:00	php-fpm	43924	/rc.linkup: DEVD Ethernet detached event for lan
                                2025-06-29 18:15:26.822283+02:00	php-fpm	43924	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1)
                                2025-06-29 18:15:25.912662+02:00	kernel	-	igc1: link state changed to DOWN
                                2025-06-29 18:15:25.763694+02:00	php	80125	[Snort] The Rules update has finished.
                                2025-06-29 18:15:25.762352+02:00	php	80125	[Snort] Snort has restarted on LAN with your new set of rules...
                                2025-06-29 18:15:25.731979+02:00	check_reload_status	674	Linkup starting igc1
                                2025-06-29 18:15:25.711985+02:00	kernel	-	725.693258 [ 852] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
                                2025-06-29 18:15:25.711692+02:00	kernel	-	725.656339 [ 852] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
                                ..............
                                2025-06-29 18:15:09.626165+02:00	snort	78173	AppInfo: AppId 4903 is UNKNOWN
                                2025-06-29 18:15:09.625169+02:00	snort	78173	AppId
                                2025-06-29 18:15:09.625141+02:00	snort	78173	AppId
                                2025-06-29 18:15:09.305900+02:00	php_pfb	89791	[pfBlockerNG] filterlog daemon started
                                2025-06-29 18:15:09.006181+02:00	snort	78173	AppId
                                2025-06-29 18:15:08.960122+02:00	tail_pfb	89291	[pfBlockerNG] Firewall Filter Service started
                                2025-06-29 18:15:08.910306+02:00	lighttpd_pfb	85817	[pfBlockerNG] DNSBL Webserver started
                                2025-06-29 18:15:08.900608+02:00	php_pfb	84265	[pfBlockerNG] filterlog daemon stopped
                                2025-06-29 18:15:08.891790+02:00	tail_pfb	83152	[pfBlockerNG] Firewall Filter Service stopped
                                2025-06-29 18:15:08.886751+02:00	lighttpd_pfb	82101	[pfBlockerNG] DNSBL Webserver stopped
                                2025-06-29 18:15:08.811564+02:00	SnortStartup	78044	Snort START for LAN(igc1)...
                                2025-06-29 18:15:01.391170+02:00	php-fpm	61163	/rc.start_packages: The command '/usr/local/bin/redis-cli shutdown save' returned exit code '1', the output was 'Could not connect to Redis at 127.0.0.1:6379: Connection refused'
                                2025-06-29 18:15:01.246646+02:00	xinetd	5473	Reconfigured: new=0 old=6 dropped=0 (services)
                                2025-06-29 18:15:01.246629+02:00	xinetd	5473	readjusting service 19005-udp
                                2025-06-29 18:15:01.246619+02:00	xinetd	5473	readjusting service 19004-udp
                                2025-06-29 18:15:01.246608+02:00	xinetd	5473	readjusting service 19003-tcp
                                2025-06-29 18:15:01.246598+02:00	xinetd	5473	readjusting service 19002-tcp
                                2025-06-29 18:15:01.246587+02:00	xinetd	5473	readjusting service 19001-tcp
                                2025-06-29 18:15:01.246573+02:00	xinetd	5473	readjusting service 19000-tcp
                                2025-06-29 18:15:01.246558+02:00	xinetd	5473	Swapping defaults
                                2025-06-29 18:15:01.246282+02:00	php-cgi	75901	servicewatchdog_cron.php: Service Watchdog detected service ntopng stopped. Restarting ntopng (ntopng Network Traffic Monitor)
                                2025-06-29 18:15:01.245996+02:00	xinetd	5473	Starting reconfiguration
                                2025-06-29 18:14:56.201498+02:00	php-fpm	61163	/rc.start_packages: [filer] XMLRPC sync completed.
                                2025-06-29 18:14:56.201329+02:00	php-fpm	61163	/rc.start_packages: XMLRPC reload data success with https://172.16.0.250:443/xmlrpc.php (pfsense.exec_php).
                                2025-06-29 18:14:56.143762+02:00	php-fpm	61163	/rc.start_packages: Beginning XMLRPC sync data to https://172.16.0.250:443/xmlrpc.php.
                                2025-06-29 18:14:56.143567+02:00	php-fpm	61163	/rc.start_packages: XMLRPC reload data success with https://172.16.0.250:443/xmlrpc.php (pfsense.merge_installedpackages_section).
                                2025-06-29 18:14:56.039883+02:00	xinetd	5473	Reconfigured: new=0 old=6 dropped=0 (services)
                                2025-06-29 18:14:56.039870+02:00	xinetd	5473	readjusting service 19005-udp
                                2025-06-29 18:14:56.039858+02:00	xinetd	5473	readjusting service 19004-udp
                                2025-06-29 18:14:56.039845+02:00	xinetd	5473	readjusting service 19003-tcp
                                2025-06-29 18:14:56.039833+02:00	xinetd	5473	readjusting service 19002-tcp
                                2025-06-29 18:14:56.039819+02:00	xinetd	5473	readjusting service 19001-tcp
                                2025-06-29 18:14:56.039803+02:00	xinetd	5473	readjusting service 19000-tcp
                                2025-06-29 18:14:56.039759+02:00	xinetd	5473	Swapping defaults
                                2025-06-29 18:14:56.038972+02:00	xinetd	5473	Starting reconfiguration
                                2025-06-29 18:14:56.008761+02:00	php-fpm	61163	/rc.start_packages: Beginning XMLRPC sync data to https://172.16.0.250:443/xmlrpc.php.
                                2025-06-29 18:14:56.008491+02:00	php-fpm	61163	/rc.start_packages: [filer] XMLRPC sync is starting.
                                2025-06-29 18:14:55.961051+02:00	php-fpm	61163	/rc.start_packages: Restarting/Starting all packages.
                                2025-06-29 18:14:55.335888+02:00	snmpd	31536	disk_OS_get_disks: adding device 'ada0' to device list
                                2025-06-29 18:14:54.886390+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:14:54.886167+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:14:54.880664+02:00	check_reload_status	674	Starting packages
                                2025-06-29 18:14:54.877863+02:00	php-fpm	43924	/rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 172.16.0.1 -> 172.16.0.1 - Restarting packages.
                                2025-06-29 18:14:52.824789+02:00	php-fpm	43924	/rc.newwanip: Creating rrd update script
                                2025-06-29 18:14:52.813951+02:00	php-fpm	43924	/rc.newwanip: Resyncing OpenVPN instances for interface LAN.
                                2025-06-29 18:14:52.813759+02:00	php-fpm	43924	/rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan
                                2025-06-29 18:14:51.000511+02:00	php-fpm	43924	/rc.newwanip: Gateway, NONE AVAILABLE
                                2025-06-29 18:14:50.992722+02:00	php-fpm	43924	/rc.newwanip: Gateway, NONE AVAILABLE
                                2025-06-29 18:14:50.431380+02:00	php-fpm	43924	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 10.0.11.213
                                2025-06-29 18:14:48.302102+02:00	xinetd	5473	Reconfigured: new=0 old=6 dropped=0 (services)
                                2025-06-29 18:14:42.208377+02:00	xinetd	5473	readjusting service 19005-udp
                                2025-06-29 18:14:42.208365+02:00	xinetd	5473	readjusting service 19004-udp
                                2025-06-29 18:14:42.208355+02:00	xinetd	5473	readjusting service 19003-tcp
                                2025-06-29 18:14:42.208344+02:00	xinetd	5473	readjusting service 19002-tcp
                                2025-06-29 18:14:42.208331+02:00	xinetd	5473	readjusting service 19001-tcp
                                2025-06-29 18:14:42.208318+02:00	xinetd	5473	readjusting service 19000-tcp
                                2025-06-29 18:14:42.208296+02:00	xinetd	5473	Swapping defaults
                                2025-06-29 18:14:42.207693+02:00	xinetd	5473	Starting reconfiguration
                                2025-06-29 18:14:36.769668+02:00	kernel	-	arp: 08:00:27:66:1d:d1 is using my IP address 192.168.255.254 on igc1!
                                2025-06-29 18:14:36.416400+02:00	php-fpm	43924	/rc.newwanip: rc.newwanip: on (IP address: 172.16.0.1) (interface: LAN[lan]) (real interface: igc1).
                                2025-06-29 18:14:36.415680+02:00	php-fpm	43924	/rc.newwanip: rc.newwanip: Info: starting on igc1.
                                2025-06-29 18:14:35.338815+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:14:35.333780+02:00	check_reload_status	674	rc.newwanip starting igc1
                                2025-06-29 18:14:35.333293+02:00	php-fpm	43924	/rc.linkup: HOTPLUG: Triggering address refresh on lan (igc1)
                                2025-06-29 18:14:35.333114+02:00	php-fpm	43924	/rc.linkup: DEVD Ethernet attached event for lan
                                2025-06-29 18:14:35.319985+02:00	php-fpm	43924	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1)
                                2025-06-29 18:14:34.964935+02:00	xinetd	5473	Reconfigured: new=0 old=6 dropped=0 (services)
                                2025-06-29 18:14:34.964923+02:00	xinetd	5473	readjusting service 19005-udp
                                2025-06-29 18:14:34.964912+02:00	xinetd	5473	readjusting service 19004-udp
                                2025-06-29 18:14:34.964901+02:00	xinetd	5473	readjusting service 19003-tcp
                                2025-06-29 18:14:34.964891+02:00	xinetd	5473	readjusting service 19002-tcp
                                2025-06-29 18:14:34.964880+02:00	xinetd	5473	readjusting service 19001-tcp
                                2025-06-29 18:14:34.964867+02:00	xinetd	5473	readjusting service 19000-tcp
                                2025-06-29 18:14:34.964849+02:00	xinetd	5473	Swapping defaults
                                2025-06-29 18:14:34.964295+02:00	xinetd	5473	Starting reconfiguration
                                2025-06-29 18:14:34.357670+02:00	kernel	-	igc1: link state changed to UP
                                2025-06-29 18:14:34.233345+02:00	check_reload_status	674	Linkup starting igc1
                                2025-06-29 18:14:33.851814+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:14:33.820590+02:00	check_reload_status	674	Reloading filter
                                2025-06-29 18:14:33.619210+02:00	snort	48132	AppInfo: AppId 2314 is UNKNOWN
                                2025-06-29 18:14:33.619161+02:00	snort	48132	Invalid direct service AppId, 2314, for 0x841db4d60 0x2cf3fffcd880
                                ........
                                2025-06-29 18:14:33.589595+02:00	snort	48132	AppInfo: AppId 4140 is UNKNOWN
                                2025-06-29 18:14:33.589520+02:00	snort	48132	AppInfo: AppId 4140 is UNKNOWN
                                2025-06-29 18:14:33.588731+02:00	snort	48132	AppId
                                2025-06-29 18:14:33.588575+02:00	snort	48132	AppId
                                2025-06-29 18:14:32.941276+02:00	snort	48132	AppId
                                2025-06-29 18:14:32.733089+02:00	php	80125	[Snort] Snort START for LAN(igc1)...
                                2025-06-29 18:14:31.926833+02:00	php-fpm	38706	/rc.linkup: DEVD Ethernet detached event for lan
                                2025-06-29 18:14:31.906676+02:00	php-fpm	38706	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1)
                                2025-06-29 18:14:30.713792+02:00	kernel	-	igc1: link state changed to DOWN
                                2025-06-29 18:14:30.672230+02:00	check_reload_status	674	Linkup starting igc1
                                2025-06-29 18:14:30.641688+02:00	snort	6713	*** Caught Term-Signal
                                2025-06-29 18:14:30.634302+02:00	php	80125	[Snort] Snort STOP for LAN(igc1)...
                                

                                Update:

                                This only happens when Inline IPS mode is enabled.
                                If you disable Blocking Mode or switch to Legacy Mode, the update runs fine—the interfaces restart correctly and everything works as expected.
                                However, when Blocking Mode is set to Inline IPS, that’s when the system hangs and behaves erratically.

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  When it runs in in-line mode it bounces the link which restarts all services. It looks like it does that several times and those restart processes cause significant load. On top of that you have the watchdog enabled and that is restarting services too. That's probably unecessary and those were restarting anyway.

                                  D 1 Reply Last reply Reply Quote 0
                                  • D
                                    dcuadrados @stephenw10
                                    last edited by

                                    @stephenw10 i only have for telegraf and ipsec:

                                    0189629a-c299-41c0-ab91-80667e2c18bd-image.png

                                    What I’ve noticed is that if I stop the service manually, it restarts automatically in less than 20 seconds. So the problem seems to be that during the update, it starts a new instance of the service, and then ends up spawning a second one.

                                    Jun 30 09:22:28	SnortStartup	65366	Snort START for LAN(em1)...
                                    Jun 30 09:22:28	php-fpm	435	/rc.start_packages: Restarting/Starting all packages.
                                    Jun 30 09:22:27	check_reload_status	479	Reloading filter
                                    Jun 30 09:22:27	check_reload_status	479	Starting packages
                                    Jun 30 09:22:27	php-fpm	8814	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages.
                                    Jun 30 09:22:25	php-fpm	8814	/rc.newwanip: Creating rrd update script
                                    Jun 30 09:22:25	php-fpm	8814	/rc.newwanip: Resyncing OpenVPN instances for interface LAN.
                                    Jun 30 09:22:25	php-fpm	8814	/rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan
                                    Jun 30 09:22:22	php-fpm	8814	/rc.newwanip: Gateway, NONE AVAILABLE
                                    Jun 30 09:22:21	php-fpm	8814	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.0.1
                                    Jun 30 09:22:09	php-fpm	8814	/rc.newwanip: rc.newwanip: on (IP address: 192.168.2.1) (interface: LAN[lan]) (real interface: em1).
                                    Jun 30 09:22:09	php-fpm	8814	/rc.newwanip: rc.newwanip: Info: starting on em1.
                                    Jun 30 09:22:08	check_reload_status	479	Reloading filter
                                    Jun 30 09:22:08	check_reload_status	479	rc.newwanip starting em1
                                    Jun 30 09:22:08	php-fpm	8814	/rc.linkup: HOTPLUG: Triggering address refresh on lan (em1)
                                    Jun 30 09:22:08	php-fpm	8814	/rc.linkup: DEVD Ethernet attached event for lan
                                    Jun 30 09:22:08	php-fpm	8814	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                    Jun 30 09:22:07	kernel		em1: link state changed to UP
                                    Jun 30 09:22:07	check_reload_status	479	Linkup starting em1
                                    Jun 30 09:22:07	check_reload_status	479	Reloading filter
                                    Jun 30 09:22:05	php-fpm	81789	/rc.linkup: DEVD Ethernet detached event for lan
                                    Jun 30 09:22:05	php-fpm	81789	/rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)
                                    Jun 30 09:22:04	kernel		em1: link state changed to DOWN
                                    Jun 30 09:22:04	check_reload_status	479	Linkup starting em1
                                    Jun 30 09:22:04	snort	96451	*** Caught Term-Signal
                                    Jun 30 09:22:03	SnortStartup	34465	Snort STOP for LAN(em1)...
                                    
                                    GertjanG 1 Reply Last reply Reply Quote 0
                                    • GertjanG
                                      Gertjan @dcuadrados
                                      last edited by Gertjan

                                      @dcuadrados said in Problems With WAN Loss Cobnection:

                                      i only have for telegraf and ipsec:

                                      IPSEC is an 'interface' bound process for sure.

                                      So, when an interface gets restarted, process gets restarted.
                                      The watchdog detects that IPSEC is restating (= stopped first), so it restart IPSEC, which will restart IPSEC so it restarts IPSEC so it restart .....

                                      You see the problem?

                                      @dcuadrados said in Problems With WAN Loss Cobnection:

                                      What I’ve noticed is that if I stop the service manually, it restarts automatically in less than 20 seconds.

                                      That's the job of watchdog.
                                      But, even that reaction is totally wrong.
                                      If you, as an admin, stops a process like IPSEC, then that's for a - your - reason, for example maintenance.
                                      You don't want it to restart all by itself.
                                      You stopped it.
                                      So ... you start it - when needed.

                                      and live will be easier.

                                      No "help me" PM's please. Use the forum, the community will thank you.
                                      Edit : and where are the logs ??

                                      D 1 Reply Last reply Reply Quote 0
                                      • D
                                        dcuadrados @Gertjan
                                        last edited by

                                        @Gertjan said in Problems With WAN Loss Cobnection:

                                        IPSEC es sin duda un proceso vinculado a una 'interfaz'.

                                        Entonces, cuando se reinicia una interfaz, se reinicia el proceso.
                                        El organismo de control detecta que IPSEC se está reiniciando (= se detuvo primero), por lo que reinicia IPSEC, que reiniciará IPSEC para que reinicie IPSEC para que se reinicie .....

                                        ¿Ves el problema?

                                        I’m testing this by completely removing Watchdog, and Snort still restarts on its own—even without an IPsec tunnel, on a freshly installed firewall. When you stop Snort, it automatically starts again.

                                        So during the update process, since Snort is already running after being restarted automatically, the update then tries to start it again—and voilà, you end up with two processes for a single interface and a highly unstable firewall.

                                        GertjanG 1 Reply Last reply Reply Quote 0
                                        • GertjanG
                                          Gertjan @dcuadrados
                                          last edited by

                                          @dcuadrados

                                          Also : what's connected to your LAN :
                                          Even if this :

                                          @dcuadrados said in Problems With WAN Loss Cobnection:

                                          Jun 29 18:32:58 php-fpm 33297 /rc.linkup: DEVD Ethernet attached event for lan
                                          Jun 29 18:32:58 php-fpm 33297 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1)

                                          isn't wrong per se, but you, as an pfSense admin, you really don't want your pfSense interfaces to get disconnected.
                                          Ok, sometimes, it can happen, or has to happen.
                                          Make it a one per month experience and you'll be way better.

                                          No "help me" PM's please. Use the forum, the community will thank you.
                                          Edit : and where are the logs ??

                                          D 1 Reply Last reply Reply Quote 0
                                          • D
                                            dcuadrados @Gertjan
                                            last edited by

                                            @Gertjan said in Problems With WAN Loss Cobnection:

                                            no está mal per se, pero usted, como administrador de pfSense, realmente no quiere que sus interfaces de pfSense se desconecten.
                                            Ok, a veces, puede suceder, o tiene que suceder.
                                            Haz que sea una experiencia de uno por mes y estarás mucho mejor.

                                            I currently have the update set to every 28 days and I'm running some tests, but I don’t understand why Snort restarts by itself. As you rightly said—as the administrator, if you stop a service, it’s for a reason. It shouldn't start again automatically. That behavior makes no sense at all.

                                            For now, I’ve scheduled it to update every 28 days. The issue is that every time it updates, it always creates a second “interface” and ends up spawning two processes.

                                            GertjanG 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.