Problems With WAN Loss Cobnection
-
It’s happening again… I have two systems right now with the same issue. It’s honestly very frustrating. The logs show this:
Jun 26 23:47:14 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:13 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:13 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:12 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:12 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:11 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:11 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:10 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:10 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:09 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:09 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:08 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:08 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:07 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:07 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:06 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:06 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:05 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:04 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:04 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:03 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:03 dpinger 12135 GW_Movistar 8.8.8.8: sendto error: 13 Jun 26 23:47:02 dpinger 12135 GW_Movistar 8.8.8.8: Alarm latency 0us stddev 0us loss 100% Jun 26 23:47:00 dpinger 12135 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.1.254 identifier "GW_Movistar "Other FW:
2025-06-26 23:41:59.210553+02:00 dpinger 39125 WANGW_nueva 8.8.8.8: Alarm latency 0us stddev 0us loss 100% 2025-06-26 23:41:57.200434+02:00 dpinger 39125 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.254 identifier "WANGW_nueva " 2025-06-26 23:41:57.187617+02:00 dpinger 38486 exiting on signal 15 2025-06-26 23:15:19.233616+02:00 dpinger 38486 WANGW_nueva 8.8.8.8: Alarm latency 13213us stddev 3126us loss 22%I have remote access to them, and Uptime Kuma reports they are up, but the WAN is down and they have no internet access. I’ve already removed Snort from the WAN—it's only on the LAN now. I really don’t know what else it could be. Could it be due to using floating rules? I can’t think of anything else that might be causing this. These are the rules:
top -HaSP:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -60 - 0B 1344K CPU2 2 43:40 100.00% [kernel{if_io_tqg_2}] 11 root 187 ki31 0B 64K CPU0 0 22.9H 99.56% [idle{idle: cpu0}] 11 root 187 ki31 0B 64K RUN 3 22.8H 90.45% [idle{idle: cpu3}] 11 root 187 ki31 0B 64K CPU1 1 23.0H 85.55% [idle{idle: cpu1}] 47858 root 20 0 1063M 934M bpf 1 6:06 0.29% /usr/local/bin/snort -R _10641 -M -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_igc010641 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 1 8220 root 20 0 17M 5156K CPU3 3 0:01 0.28% top -HaSP 61672 unbound 20 0 547M 496M kqread 1 2:53 0.19% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 33872 root 20 0 5655M 230M uwait 0 8:29 0.14% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 7 root -16 - 0B 16K pftm 3 1:15 0.13% [pf purge] 33872 root 20 0 5655M 230M uwait 0 3:27 0.12% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 0 root -60 - 0B 1344K - 1 1:49 0.08% [kernel{if_config_tqg_0}] 0 root -60 - 0B 1344K - 1 0:04 0.08% [kernel{softirq_1}] 0 root -60 - 0B 1344K - 3 0:03 0.05% [kernel{softirq_3}] 28381 root 20 0 22M 12M select 0 0:00 0.03% sshd: admin@pts/0 (sshd) 0 root -60 - 0B 1344K - 3 1:40 0.03% [kernel{if_io_tqg_3}] 97202 root 20 0 1187M 1005M bpf 1 7:06 0.03% /usr/local/bin/snort -R _5949 -M -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_igc15949 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 594 33872 root 20 0 5655M 230M uwait 3 3:17 0.03% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 33872 root 20 0 5655M 230M kqread 0 3:15 0.02% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 33872 root 20 0 5655M 230M kqread 0 3:02 0.02% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 57489 root 20 0 13M 2908K kqread 0 0:24 0.02% /usr/bin/tail_pfb -n0 -F /var/log/filter.log 9 root -16 - 0B 48K psleep 3 0:27 0.02% [pagedaemon{dom0}] 44329 root 20 0 13M 2908K kqread 1 0:05 0.02% tail -F -n 0 /var/log/auth.log 46933 root 20 0 13M 2748K select 0 0:04 0.01% tail: system.fileargs (tail) 58244 root 20 0 13M 2748K select 3 0:18 0.01% tail_pfb: system.fileargs (tail_pfb) 73593 root 20 0 13M 3876K bpf 0 0:37 0.01% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid 39125 root 20 0 17M 3160K nanslp 0 0:00 0.01% /usr/local/bin/dpinger -S -r 0 -i WANGW_nueva -B 192.168.0.254 -p /var/run/dpinger_WANGW_nueva~192.168.0.254~8.8.8.8.pid -u /var/run/dpinger_WANGW_nueva~192.168.0.254~8.8.8.8.sock -C /etc/rc.gateway_alarm -d 1 -s 0 root -60 - 0B 1344K - 0 0:03 0.01% [kernel{softirq_0}] 8 root -16 - 0B 16K - 3 0:14 0.01% [rand_harvestq] 33872 root 20 0 5655M 230M uwait 0 3:34 0.01% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 57781 root 20 0 83M 63M piperd 3 117:59 0.01% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog 2 root -60 - 0B 64K WAIT 0 0:18 0.01% [clock{clock (0)}] 0 root -60 - 0B 1344K - 0 2:02 0.01% [kernel{if_io_tqg_0}] 44643 root 20 0 13M 3196K kqread 1 0:07 0.01% /usr/sbin/syslogd -O rfc5424 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf 33872 root 20 0 5655M 230M uwait 3 1:22 0.01% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 34247 root 20 0 102M 28M uwait 1 0:01 0.01% /usr/local/libexec/ipsec/charon --use-syslog{charon} 32522 root 20 0 24M 9160K select 1 0:07 0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd} 0 root -12 - 0B 1344K - 3 1:45 0.01% [kernel{z_wr_iss_2}] 12 root -64 - 0B 224K WAIT 1 0:30 0.00% [intr{irq129: ahci0}] 34247 root 20 0 102M 28M uwait 3 0:02 0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon} 12 root -60 - 0B 224K WAIT 3 0:39 0.00% [intr{swi1: netisr 3}] 6013 root 20 0 1184M 974M bpf 1 0:04 0.00% /usr/local/bin/snort -R _57509 -M -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_ovpns157509 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 58427 root 20 0 22M 10M kqread 3 0:15 0.00% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf 628 root 20 0 108M 32M kqread 0 0:03 0.00% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) 0 root -16 - 0B 1344K - 3 0:22 0.00% [kernel{z_wr_int_1}] 2 root -60 - 0B 64K WAIT 1 0:04 0.00% [clock{clock (1)}] 61672 unbound 20 0 547M 496M kqread 3 2:11 0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 39125 root 20 0 17M 3160K nanslp 0 0:00 0.00% /usr/local/bin/dpinger -S -r 0 -i WANGW_nueva -B 192.168.0.254 -p /var/run/dpinger_WANGW_nueva~192.168.0.254~8.8.8.8.pid -u /var/run/dpinger_WANGW_nueva~192.168.0.254~8.8.8.8.sock -C /etc/rc.gateway_alarm -d 1 -s 16 root -60 - 0B 80K - 1 0:03 0.00% [usb{usbus0}] 33872 root 68 0 5655M 230M uwait 0 3:35 0.00% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf} 6013 root 20 0 1184M 974M nanslp 1 0:02 0.00% /usr/local/bin/snort -R _57509 -M -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_ovpns157509 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 0 root -16 - 0B 1344K - 3 0:02 0.00% [kernel{z_null_int}] 47858 root 20 0 1063M 934M nanslp 0 0:02 0.00% /usr/local/bin/snort -R _10641 -M -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_igc010641 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 1 2 root -60 - 0B 64K WAIT 3 0:05 0.00% [clock{clock (3)}] 97202 root 20 0 1187M 1005M nanslp 3 0:02 0.00% /usr/local/bin/snort -R _5949 -M -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_igc15949 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 594 0 root -60 - 0B 1344K - 1 1:56 0.00% [kernel{if_io_tqg_1}] 6 root -8 - 0B 896K mmp->m 1 0:01 0.00% [zfskern{mmp_thread_enter}] 21 root 16 - 0B 16K syncer 1 0:01 0.00% [syncer] 0 root -16 - 0B 1344K - 3 0:01 0.00% [kernel{z_flush_int}]My system tuneables:
Tunable Name Description Value net.inet.ip.intr_queue_maxlen Cola de interrupciones ampliada 3000 net.inet6.ip6.intr_queue_maxlen Cola de interrupciones IPv6 3000 hw.igc.max_interrupt_rate Tasa de interrupciones del driver IGC 12000 hw.igc.rx_process_limit Limitar el procesamiento de RX en el driver IGC -1 net.inet.tcp.soreceive_stream Ajuste TCP 1 net.isr.maxthreads Hilos ISR (paralelismo de red) 1 net.isr.bindthreads Vincular hilos ISR a CPUs 1 net.isr.dispatch Distribución directa del tráfico ISR direct net.inet.rss.enabled RSS habilitado 1 net.inet.rss.bits RSS Bits (distribuir en múltiples núcleos) 2 hw.ix.flow_control Desactivar Flow Control (i225/i226) 0 dev.igc.0.fc Desactivar Flow Control en interfaces i225/i226 WAN 0 dev.igc.1.fc Desactivar Flow Control en interfaces i225/i226 LAN 0 kern.ipc.nmbclusters Aumentar tamaño del buffer de red (mbufs) 1000000 kern.ipc.nmbclusters Clusters de memoria (recomendado por Netgate) 1000000 net.inet.ip.portrange.first 1024 net.inet.tcp.blackhole Do not send RST on segments to closed ports 2 net.inet.udp.blackhole Do not send port unreachables for refused connects 1 net.inet.ip.random_id Assign random ip_id values 1 net.inet.tcp.drop_synfin Drop TCP packets with SYN+FIN set 1 net.inet.ip.redirect Enable sending IP redirects 1 net.inet6.ip6.redirect Send ICMPv6 redirects for unforwardable IPv6 packets 1 net.inet6.ip6.use_tempaddr Create RFC3041 temporary addresses for autoconfigured addresses 0 net.inet6.ip6.prefer_tempaddr Prefer RFC3041 temporary addresses in source address selection 0 net.inet.tcp.syncookies Use TCP SYN cookies if the syncache overflows 1 net.inet.tcp.recvspace Initial receive socket buffer size 65228 net.inet.tcp.sendspace Initial send socket buffer size 65228 net.inet.tcp.delayed_ack Delay ACK to try and piggyback it onto a data packet 0 net.inet.udp.maxdgram Maximum outgoing UDP datagram size 57344 net.link.bridge.pfil_onlyip Only pass IP packets when pfil is enabled 0 net.link.bridge.pfil_member Packet filter on the member interface 1 net.link.bridge.pfil_bridge Packet filter on the bridge interface 0 net.link.tap.user_open Enable legacy devfs interface creation for all users 1 net.link.vlan.mtag_pcp Retain VLAN PCP information as packets are passed up the stack 1 kern.randompid Random PID modulus. Special values: 0: disable, 1: choose random value 347 hw.syscons.kbd_reboot enable keyboard reboot 0 net.inet.tcp.log_debug Log errors caused by incoming TCP segments 0 net.inet.tcp.tso Enable TCP Segmentation Offload 1 net.inet.icmp.icmplim Maximum number of ICMP responses per second 0 vfs.read_max Cluster read-ahead max block count 32 kern.ipc.maxsockbuf Maximum socket buffer size 4262144 net.inet.ip.process_options Enable IP options processing ([LS]SRR, RR, TS) 0 kern.random.harvest.mask Entropy harvesting mask 351 net.route.netisr_maxqlen maximum routing socket dispatch queue length 1024 net.inet.udp.checksum compute udp checksum 1 net.inet.icmp.reply_from_interface ICMP reply from incoming interface for non-local packets 1 net.inet6.ip6.rfc6204w3 Accept the default router list from ICMPv6 RA messages even when packet forwarding is enabled 1 net.key.preferred_oldsa 0 net.inet.carp.senderr_demotion_factor Send error demotion factor adjustment 0 net.pfsync.carp_demotion_factor pfsync's CARP demotion factor adjustment 0 net.raw.recvspace 65536 net.raw.sendspace 65536 net.inet.raw.recvspace Maximum space for incoming raw IP datagrams 131072 net.inet.raw.maxdgram Maximum outgoing raw IP datagram size 131072 kern.corefile Process corefile name format string /root/%N.core kern.crypto.iimb.enable_aescbc enable AES-CBC cipher 1 kern.crypto.iimb.enable_multiq enable multi-Q 1 kern.crypto.iimb.use_task use taskq 0 kern.crypto.iimb.arch Set architecture auto kern.crypto.iimb.prefetch prefetch key for jobs 1 kern.crypto.iimb.max_jobs max number of jobs in batch 256
loader.conf:
kern.cam.boot_delay=10000 kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" kern.ipc.nmbclusters="1000000" kern.ipc.nmbjumbo9="524288" kern.ipc.nmbjumbop="524288" opensolaris_load="YES" zfs_load="YES" opensolaris_load="YES" zfs_load="YES" kern.cam.boot_delay=10000 kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" kern.ipc.nmbclusters="1000000" kern.ipc.nmbjumbo9="524288" kern.ipc.nmbjumbop="524288" kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" cryptodev_load="YES" zfs_load="YES" kern.ipc.semopm=100 kern.ipc.semmni=128 kern.ipc.semmns=32000 kern.ipc.shmmni=4096 loader_conf_files="/boot/loader.conf.lua" boot_serial="NO" autoboot_delay="3" debug.ddb.capture.bufsize="524288" hw.e6000sw.default_disabled=1 hw.hn.vf_transparent="0" hw.hn.use_if_start="1" net.link.ifqmaxlen="128" machdep.hwpstate_pkg_ctrl="0" net.pf.states_hashsize="524288"Networking:
Rules:
WAN:
-
sendto error 13 is 'permission denied' which doesn't make much sense unless it's being blocked upstream. I might imagine something triggering on the monitoring pings and blocking your IP.
Try running packet capture on WAN whilst in that state. Do you see icmp denied packets being returned from upstream?
-
@stephenw10 I no longer have access—it’s completely frozen. Until it’s restarted, I won’t be able to connect, which means I won’t have access until tomorrow. I don’t know what it could be—whether it’s Snort, the rules, pfBlocker, or some misconfiguration. The best thing would be to reformat everything, leave it clean without any rules, and start adding them little by little. Clearly, it’s a configuration issue—some mistake on my part. There’s no other explanation.
-
Are all three devices behind the same ISP devices? This could be the upstream device blocking the traffic if it has some DOS prevention for example. Though if that was the case I'd expect it to remain blocked for some time whether or not pfSense is rebooted. Assuming it pulls the same WAN IP.
-
@stephenw10 said in Problems With WAN Loss Cobnection:
Are all three devices behind the same ISP devices? This could be the upstream device blocking the traffic if it has some DOS prevention for example. Though if that was the case I'd expect it to remain blocked for some time whether or not pfSense is rebooted. Assuming it pulls the same WAN IP.
No, they are independent devices in different locations—they have nothing to do with each other. What I did notice when running
topwas this:0 root -60 - 0B 1344K CPU2 2 43:40 100.00% [kernel{if_io_tqg_2}]In other words, something is maxing out the processor completely, while the other cores remain idle:
11 root 187 ki31 0B 64K CPU0 0 22.9H 99.56% [idle{idle: cpu0}] 11 root 187 ki31 0B 64K RUN 3 22.8H 90.45% [idle{idle: cpu3}] 11 root 187 ki31 0B 64K CPU1 1 23.0H 85.55% [idle{idle: cpu1}]So I’m not sure if it might be related to something I configured in System Tunables.
-
Just checking :
You've activated :
You use igc NICs, right ?
According to ALTQ support page : Network Interface Drivers with ALTQ Traffic Shaping Support where you can see the list of supported NIC drivers (click on altq(4)).
"igc" isn't present on that list.So ?
-
@Gertjan You're right, I have it enabled—it was carried over from another firewall backup that used a different driver. I'm going to disable it. That said, one of the devices with the same issue is using an em driver. Still, I’ve already disabled it, and I’ll reboot as soon as I can for the changes to take effect.
-
That setting only does anything if you have hn NICs. So only in HyperV or Azure.
igc does support altq, it should be fine.
That CPU usage is all in the NIC. It looks like it's trying to push a load of traffic. Check the traffic and packet graphs in Status > Monitoring at the time. Is there a huge spike in traffic?
Try to get a pcap of it at the time if you can. You might have something creating a loop or some sort of reflected traffic.
@dcuadrados said in Problems With WAN Loss Cobnection:
they are independent devices in different locations—they have nothing to do with each other.
Sorry what I meant there was are all three locations using the same ISP? With the same type of ISP supplied router?
Are all three using double NAT? -
@stephenw10 said in Problems With WAN Loss Cobnection:
Ese uso de CPU está todo en la NIC. Parece como si estuviera intentando impulsar una gran cantidad de tráfico. Verifique los gráficos de tráfico y paquetes en Estado > Monitoreo en ese momento. ¿Hay un gran aumento en el tráfico?
I have the Grafana data, and there is indeed a traffic increase beforehand, followed by a noticeable drop at that moment. It happens around 11:00 PM, as shown, and at that time no one is working in the office. The Acronis Online Backup runs at 11:45 PM, and the loss begins around 11:30 PM, but I don’t see anything unusual aside from that. Additionally, all devices on the network are shut down at 10:30 PM by policy, and only the server remains on.
@stephenw10 said in Problems With WAN Loss Cobnection:
Perdón, ¿qué quise decir con que había tres ubicaciones que usaban el mismo ISP? Con lo mismo tipo ¿del enrutador suministrado por el ISP?
¿Los tres utilizan NAT doble?All three use double NAT, and they’re with different ISPs, but in all cases I pass the traffic from the router to the firewall. As I mentioned, I have dozens of pfSense setups configured this way, but the ones failing are these—specifically the ones with i225 and i226 NICs.
The difference with these setups is that I use:
- Snort
- Floating rules
- pfBlocker
- Telegraf
The others are older and only use outbound hardening (secure outbound rules), VPN, IPsec... and they work flawlessly. All of those use em drivers, don’t use Snort, have regular per-interface rules, and I don’t monitor them with Influx + Grafana.
And this is what I have in the cron jobs—nothing unusual as far as I can tell.
Could this be the cause?0,15,30,45 * * * * root /etc/rc.filter_configure_syncminute hour mday month wday who command */1 * * * * root /usr/sbin/newsyslog 1 3 * * * root /etc/rc.periodic daily 15 4 * * 6 root /etc/rc.periodic weekly 30 5 1 * * root /etc/rc.periodic monthly 1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a 1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh 1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables 1 0 * * * root /usr/bin/nice -n20 /etc/rc.update_pkg_metadata */15 * * * * root /usr/local/bin/php /root/scripts/pfsense_zbx.php speedtest_cron 0,15,30,45 * * * * root /etc/rc.filter_configure_sync * * * * * root /usr/bin/nice -n20 /usr/local/bin/php /usr/local/sbin/acbupload.php 0 16 * * 7 root /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php bl shallalist,ut1 >> /var/log/pfblockerng/extras.log 2>&1 36 0 * * * root /usr/bin/nice -n20 /usr/local/bin/php /usr/local/sbin/execacb.php 0 14 * * * root /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1 */1 * * * * root /usr/local/pkg/servicewatchdog_cron.php 16 3 * * * root /usr/local/pkg/acme/acme_command.sh "renewall" | /usr/bin/logger -t ACME 2>&1 0 0 * * * root /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1 */5 * * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc */5 * * * * root /usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 1800 20 2 */28 * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.phpEDITED:
Alright, I just reviewed the last 30 days of Grafana data from the problematic firewalls, and I noticed that the WAN always drops at X:00, X:15, X:30, and X:45—so it clearly matches the reload that happens every 15 minutes.
I’ve read that this can be caused by rules that use schedules, and in my case, I do use schedules for some rules:
- Certificate renewal — I haven’t switched all of them to ACME via API yet
- Some backup tasks
- Limiters in some firewalls for bandwidth management
Could the issue be related to this?
Is it really necessary to perform this reload every 15 minutes? -
@dcuadrados said in Problems With WAN Loss Cobnection:
and I noticed that the WAN always drops at X:00, X:15, X:30, and X:45—so it clearly matches the reload that happens every 15 minutes.
Yeah, you've shown it already :
Where did that cron task come from ?
Btw :
I can't figure out a good reason why a backup should do "do" things with interfaces and/or their connection.
Neither for limiters.
Certificate renewal is this one :
and doesn't does nothing with interfaces.
This one :
is the anti-quality-of-live pfSense package. It creates problems, hard to find issues and constant pain.
This one :
isn't ideal neither ... Really, checking the speed of wan every 15 minutes ?
in reality, the bandwidth available is always known : it's the bandwidth the ISP gave you, minus what you are using at that moment. -
@Gertjan said in Problems With WAN Loss Cobnection:
It's automatically generated by the scheduler—that is, when I have rules enabled for specific time periods...
When I remove the schedulers, it disappears.My question is: can it be deleted manually? Will it be recreated after a reboot?
Supposedly, according to this, it should be resolved in these FreeBSD versions:
https://redmine.pfsense.org/issues/10414#!/usr/local/bin/php-cgi -f <?php /* * rc.filter_configure_sync * * part of pfSense (https://www.pfsense.org) * Copyright (c) 2004-2013 BSD Perimeter * Copyright (c) 2013-2016 Electric Sheep Fencing * Copyright (c) 2014-2025 Rubicon Communications, LLC (Netgate) * All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ require_once("config.inc"); require_once("functions.inc"); require_once("filter.inc"); require_once("shaper.inc"); require_once("ipsec.inc"); require_once("vpn.inc"); filter_configure_sync(); ?> -
@dcuadrados said in Problems With WAN Loss Cobnection:
but the ones failing are these—specifically the ones with i225 and i226 NICs.
What version of the NIC chips are they using? The early revision i225s especially were known for dropping connections. But I assume here you don't actually see the link loss?
[25.03-BETA][admin@4100-2.stevew.lan]/root: pciconf -lv igc0 igc0@pci0:4:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Ethernet Controller I225-V' class = network subclass = ethernetThere you can see it's a rev.3 chip which had fixed the earlier issues.
-
@stephenw10 said in Problems With WAN Loss Cobnection:
¿Qué versión de los chips NIC están utilizando? Los primeros i225 de revisión eran especialmente conocidos por perder conexiones. ¿Pero supongo que aquí en realidad no ves la pérdida del enlace?
It seems that after removing that cron job, the firewall has completely stabilized. If you look at the graphs, there used to be spikes every 15 minutes—both on the WAN and CPU usage—and now they’re gone. I’m attaching the last 24 hours of graphs below. Now the key is to understand why that cron job is generated and whether it’s actually necessary.
As for the driver, it’s using the following version:
igc0@pci0:1:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Ethernet Controller I225-V' class = network subclass = ethernet@Gertjan said in Problems With WAN Loss Cobnection:
Este:
tampoco es ideal... ¿En serio, comprobar la velocidad de wan cada 15 minutos?
En realidad, el ancho de banda disponible siempre se conoce: es el ancho de banda que te dio el ISP, menos lo que estás usando en ese momento.Regarding what you mentioned about the speed-related cron job, that was something I added to test a few things in Zabbix—but I don’t even use Zabbix anymore, since I now do monitoring with Grafana and Wazuh. I’ve removed those as well, and now I need to move all my ACME setups to API-based renewal using token authentication instead of relying on port-based renewal.
I’ll monitor how it behaves over the weekend, but it seems like we’ve made some progress. Huge thanks to both of you! I’ll keep you posted.
@Gertjan said in Problems With WAN Loss Cobnection:
Sí, ya lo has demostrado:
¿De dónde surgió esa tarea cron?
As I mentioned earlier, this happens when schedules are used for the rules. It seems that every 15 minutes it kills all states to enforce the schedule and the associated rules.
-
The filter configure sync cron job shouldn't cause a significant issue. It certainly shouldn't cause a complete loss of connectivity.
But let's see how it goes....
-
@stephenw10 i see here the same problem https://redmine.pfsense.org/issues/10414#note-10
-
That was a very specific issue in pf and it was fixed 5 years ago. I assume you're not running 2.4.4?
-
@stephenw10 said in Problems With WAN Loss Cobnection:
That was a very specific issue in pf and it was fixed 5 years ago. I assume you're not running 2.4.4?
Responder
No, I'm obviously not running that version, but as I mentioned, by analyzing the Grafana data, I noticed that the WAN always drops exactly when that cron job runs—at X:00, X:15, X:30, or X:45. So it clearly coincides with the cron execution.
Of course, I’m not considering the issue fully resolved yet. Right now, I’m monitoring to see if the problem happens again so I can gather more information. I’ll keep you updated over the next few days on how things go.
And interestingly, the firewalls that don’t experience any drops don’t have that cron job at all.
-
And definitely not that speedtest cronjob? That would be my first suspect.
-
@stephenw10 said in Problems With WAN Loss Cobnection:
¿Y definitivamente no ese cronjob de prueba de velocidad? Ese sería mi primer sospechoso.
That one wasn’t present on all systems—some had it, but others didn’t—so I don’t think that was the root cause. Still, I’ve cleaned up the cron jobs to leave only the essential ones, and for now, everything seems stable.
-
I’m bringing an update: everything is running better now, with one exception. When Snort updates, something similar happens—the system becomes heavily loaded and somewhat unresponsive. I’ve noticed that two Snort processes remain running, and if it updates again, a third one appears. This causes the system to become overloaded, even though there should only be one Snort process running, since it's configured only on the LAN in Inline IPS mode.
I just ran the test right now.
Here are the logs shown in System:
Jun 29 18:34:00 php-cgi 91062 servicewatchdog_cron.php: Service Watchdog detected service telegraf stopped. Restarting telegraf (Telegraf daemon) Jun 29 18:33:26 php_pfb 79326 [pfBlockerNG] filterlog daemon started Jun 29 18:33:25 tail_pfb 79015 [pfBlockerNG] Firewall Filter Service started Jun 29 18:33:25 lighttpd_pfb 76804 [pfBlockerNG] DNSBL Webserver started Jun 29 18:33:25 lighttpd_pfb 74750 [pfBlockerNG] DNSBL Webserver stopped Jun 29 18:33:25 php_pfb 73650 [pfBlockerNG] filterlog daemon stopped Jun 29 18:33:25 tail_pfb 73212 [pfBlockerNG] Firewall Filter Service stopped Jun 29 18:33:19 php-fpm 57400 /rc.start_packages: Restarting/Starting all packages. Jun 29 18:33:18 check_reload_status 476 Reloading filter Jun 29 18:33:18 check_reload_status 476 Starting packages Jun 29 18:33:18 php-fpm 87641 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages. Jun 29 18:33:16 php-fpm 87641 /rc.newwanip: Creating rrd update script Jun 29 18:33:16 php-fpm 87641 /rc.newwanip: Resyncing OpenVPN instances for interface LAN. Jun 29 18:33:16 php-fpm 87641 /rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan Jun 29 18:33:13 php-fpm 87641 /rc.newwanip: Gateway, NONE AVAILABLE Jun 29 18:33:12 php-fpm 87641 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.0.1 Jun 29 18:33:00 php-cgi 37854 servicewatchdog_cron.php: Service Watchdog detected service telegraf stopped. Restarting telegraf (Telegraf daemon) Jun 29 18:32:59 php-fpm 87641 /rc.newwanip: rc.newwanip: on (IP address: 192.168.2.1) (interface: LAN[lan]) (real interface: em1). Jun 29 18:32:59 php-fpm 87641 /rc.newwanip: rc.newwanip: Info: starting on em1. Jun 29 18:32:58 check_reload_status 476 Reloading filter Jun 29 18:32:58 check_reload_status 476 rc.newwanip starting em1 Jun 29 18:32:58 php-fpm 33297 /rc.linkup: HOTPLUG: Triggering address refresh on lan (em1) Jun 29 18:32:58 php-fpm 33297 /rc.linkup: DEVD Ethernet attached event for lan Jun 29 18:32:58 php-fpm 33297 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1) Jun 29 18:32:58 check_reload_status 476 Reloading filter Jun 29 18:32:56 php_pfb 12657 [pfBlockerNG] filterlog daemon started Jun 29 18:32:56 tail_pfb 12409 [pfBlockerNG] Firewall Filter Service started Jun 29 18:32:56 lighttpd_pfb 12380 [pfBlockerNG] DNSBL Webserver started Jun 29 18:32:56 lighttpd_pfb 9243 [pfBlockerNG] DNSBL Webserver stopped Jun 29 18:32:56 php_pfb 7656 [pfBlockerNG] filterlog daemon stopped Jun 29 18:32:56 tail_pfb 7195 [pfBlockerNG] Firewall Filter Service stopped Jun 29 18:32:52 kernel em1: link state changed to UP Jun 29 18:32:52 check_reload_status 476 Linkup starting em1 Jun 29 18:32:50 php-fpm 11935 /rc.start_packages: Restarting/Starting all packages. Jun 29 18:32:49 php-fpm 89037 /rc.linkup: DEVD Ethernet detached event for lan Jun 29 18:32:49 php-fpm 89037 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1) Jun 29 18:32:49 check_reload_status 476 Reloading filter Jun 29 18:32:49 check_reload_status 476 Starting packages Jun 29 18:32:49 php-fpm 57400 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages. Jun 29 18:32:48 kernel em1: link state changed to DOWN Jun 29 18:32:48 check_reload_status 476 Linkup starting em1 Jun 29 18:32:48 kernel 768.603146 [ 852] iflib_netmap_config txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048 Jun 29 18:32:47 php-fpm 57400 /rc.newwanip: Creating rrd update script Jun 29 18:32:47 php-fpm 57400 /rc.newwanip: Resyncing OpenVPN instances for interface LAN. Jun 29 18:32:47 php-fpm 57400 /rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan Jun 29 18:32:42 php-fpm 57400 /rc.newwanip: Gateway, NONE AVAILABLE Jun 29 18:32:42 php-fpm 57400 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.0.1 Jun 29 18:32:24 php-fpm 57400 /rc.newwanip: rc.newwanip: on (IP address: 192.168.2.1) (interface: LAN[lan]) (real interface: em1). Jun 29 18:32:24 php-fpm 57400 /rc.newwanip: rc.newwanip: Info: starting on em1. Jun 29 18:32:23 check_reload_status 476 Reloading filter Jun 29 18:32:23 check_reload_status 476 rc.newwanip starting em1 Jun 29 18:32:23 php-fpm 87641 /rc.linkup: HOTPLUG: Triggering address refresh on lan (em1) Jun 29 18:32:23 php-fpm 87641 /rc.linkup: DEVD Ethernet attached event for lan Jun 29 18:32:23 php-fpm 87641 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1) Jun 29 18:32:22 kernel em1: link state changed to UP Jun 29 18:32:22 check_reload_status 476 Linkup starting em1 Jun 29 18:32:21 check_reload_status 476 Reloading filter Jun 29 18:32:19 php-fpm 11935 /rc.linkup: DEVD Ethernet detached event for lan Jun 29 18:32:19 php-fpm 11935 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1) Jun 29 18:32:18 kernel em1: link state changed to DOWN Jun 29 18:32:18 check_reload_status 476 Linkup starting em1 Jun 29 18:32:18 php 37179 [Snort] The Rules update has finished. Jun 29 18:32:18 php 37179 [Snort] Snort has restarted on LAN with your new set of rules... Jun 29 18:32:18 kernel 738.705442 [ 852] iflib_netmap_config txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048 Jun 29 18:32:00 php-cgi 78632 servicewatchdog_cron.php: Service Watchdog detected service telegraf stopped. Restarting telegraf (Telegraf daemon) Jun 29 18:31:22 php_pfb 61839 [pfBlockerNG] filterlog daemon started Jun 29 18:31:22 tail_pfb 61431 [pfBlockerNG] Firewall Filter Service started Jun 29 18:31:22 lighttpd_pfb 59064 [pfBlockerNG] DNSBL Webserver started Jun 29 18:31:22 lighttpd_pfb 57084 [pfBlockerNG] DNSBL Webserver stopped Jun 29 18:31:22 php_pfb 56006 [pfBlockerNG] filterlog daemon stopped Jun 29 18:31:22 tail_pfb 55454 [pfBlockerNG] Firewall Filter Service stopped Jun 29 18:31:13 SnortStartup 93013 Snort START for LAN(em1)... Jun 29 18:31:12 php-fpm 11935 /rc.start_packages: Restarting/Starting all packages. Jun 29 18:31:11 check_reload_status 476 Reloading filter Jun 29 18:31:11 check_reload_status 476 Starting packages Jun 29 18:31:11 php-fpm 40346 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages. Jun 29 18:31:09 php-fpm 40346 /rc.newwanip: Creating rrd update script Jun 29 18:31:09 php-fpm 40346 /rc.newwanip: Resyncing OpenVPN instances for interface LAN. Jun 29 18:31:09 php-fpm 40346 /rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan Jun 29 18:31:04 php-fpm 40346 /rc.newwanip: Gateway, NONE AVAILABLE Jun 29 18:31:04 php-fpm 40346 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 192.168.0.1 Jun 29 18:30:49 php-fpm 40346 /rc.newwanip: rc.newwanip: on (IP address: 192.168.2.1) (interface: LAN[lan]) (real interface: em1). Jun 29 18:30:49 php-fpm 40346 /rc.newwanip: rc.newwanip: Info: starting on em1. Jun 29 18:30:48 check_reload_status 476 Reloading filter Jun 29 18:30:48 check_reload_status 476 rc.newwanip starting em1 Jun 29 18:30:48 php-fpm 40346 /rc.linkup: HOTPLUG: Triggering address refresh on lan (em1) Jun 29 18:30:48 php-fpm 40346 /rc.linkup: DEVD Ethernet attached event for lan Jun 29 18:30:48 php-fpm 40346 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1) Jun 29 18:30:47 kernel em1: link state changed to UP Jun 29 18:30:47 check_reload_status 476 Linkup starting em1 Jun 29 18:30:46 check_reload_status 476 Reloading filter Jun 29 18:30:45 snort 42755 AppInfo: AppId 5349 is UNKNOWN Jun 29 18:30:45 snort 42755 Invalid direct service AppId, 5349, for 0x842745c70 0x1aa0e5f24800 Jun 29 18:30:45 snort 42755 AppInfo: AppId 4903 is UNKNOWN Jun 29 18:30:45 snort 42755 AppId Jun 29 18:30:45 snort 42755 AppId Jun 29 18:30:44 snort 42755 AppId Jun 29 18:30:44 php-fpm 11935 /rc.linkup: DEVD Ethernet detached event for lan Jun 29 18:30:44 php-fpm 11935 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 192.168.2.1) Jun 29 18:30:44 php 37179 [Snort] Snort START for LAN(em1)... Jun 29 18:30:43 kernel em1: link state changed to DOWN Jun 29 18:30:43 check_reload_status 476 Linkup starting em1 Jun 29 18:30:43 snort 98159 *** Caught Term-Signal Jun 29 18:30:42 php 37179 [Snort] Snort STOP for LAN(em1)... Jun 29 18:30:41 php 37179 [Snort] Building new sid-msg.map file for LAN... Jun 29 18:30:41 php 37179 [Snort] Enabling any flowbit-required rules for: LAN... Jun 29 18:30:41 php 37179 [Snort] Error - unable to find reject_sid list "none" specified for LAN Jun 29 18:30:41 php 37179 [Snort] Error - unable to find drop_sid list "none" specified for LAN Jun 29 18:30:38 php 37179 [Snort] Updating rules configuration for: LAN ... Jun 29 18:30:38 php 37179 [Snort] Removed 0 obsoleted rules category files. Jun 29 18:30:38 php 37179 [Snort] Hide Deprecated Rules is enabled. Removing obsoleted rules categories. Jun 29 18:30:38 php 37179 [Snort] Feodo Tracker Botnet C2 IP rules were updated... Jun 29 18:30:38 php 37179 [Snort] Feodo Tracker Botnet C2 IP rules file update downloaded successfully. Jun 29 18:30:38 php 37179 [Snort] Emerging Threats Open rules are up to date... Jun 29 18:30:38 php 37179 [Snort] Snort GPLv2 Community Rules are up to date... Jun 29 18:30:37 php 37179 [Snort] Snort AppID Open Text Rules are up to date... Jun 29 18:30:36 php 37179 [Snort] Snort OpenAppID detectors are up to date... Jun 29 18:30:36 php 37179 [Snort] Snort Subscriber rules are up to date...Here you can see that Snort’s log shows the update completes very quickly:
Starting rules update... Time: 2025-06-29 18:30:35 Downloading Snort Subscriber rules md5 file snortrules-snapshot-29200.tar.gz.md5... Checking Snort Subscriber rules md5 file... Snort Subscriber rules are up to date. Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5... Checking Snort OpenAppID detectors md5 file... Snort OpenAppID detectors are up to date. Downloading Snort AppID Open Text Rules md5 file appid_rules.tar.gz.md5... Checking Snort AppID Open Text Rules md5 file... Snort AppID Open Text Rules are up to date. Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5... Checking Snort GPLv2 Community Rules md5 file... Snort GPLv2 Community Rules are up to date. Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5... Checking Emerging Threats Open rules md5 file... Emerging Threats Open rules are up to date. Downloading Feodo Tracker Botnet C2 IP rules file... Done downloading rules file. Extracting and installing Feodo Tracker Botnet C2 IP rules... Feodo Tracker Botnet C2 IP rules were updated. Copying new config and map files... Updating rules configuration for: LAN ... Restarting Snort on LAN to activate the new set of rules... Snort has restarted on LAN with your new set of rules. The Rules update has finished. Time: 2025-06-29 18:32:18And the gateway:
Jun 29 18:43:39 dpinger 16187 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP " Jun 29 18:43:39 dpinger 22451 exiting on signal 15 Jun 29 18:42:11 dpinger 22451 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP " Jun 29 18:42:11 dpinger 32634 exiting on signal 15 Jun 29 18:41:55 dpinger 32634 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP " Jun 29 18:41:55 dpinger 62920 exiting on signal 15 Jun 29 18:40:30 dpinger 62920 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP " Jun 29 18:40:30 dpinger 61345 exiting on signal 15 Jun 29 18:33:13 dpinger 61345 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP " Jun 29 18:33:12 dpinger 86243 exiting on signal 15 Jun 29 18:32:42 dpinger 86243 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP " Jun 29 18:32:42 dpinger 26696 exiting on signal 15 Jun 29 18:31:04 dpinger 26696 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 192.168.0.20 identifier "WAN_DHCP " Jun 29 18:31:04 dpinger 78346 exiting on signal 15It loses connectivity and everything starts behaving erratically.
Logs from anothe:
2025-06-29 18:15:37.362731+02:00 xinetd 5473 Swapping defaults 2025-06-29 18:15:37.362131+02:00 xinetd 5473 Starting reconfiguration 2025-06-29 18:15:31.770460+02:00 kernel - arp: 08:00:27:66:1d:d1 is using my IP address 192.168.255.254 on igc1! 2025-06-29 18:15:31.497019+02:00 php-fpm 38706 /rc.newwanip: rc.newwanip: on (IP address: 172.16.0.1) (interface: LAN[lan]) (real interface: igc1). 2025-06-29 18:15:31.496546+02:00 php-fpm 38706 /rc.newwanip: rc.newwanip: Info: starting on igc1. 2025-06-29 18:15:30.422269+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:15:30.416622+02:00 check_reload_status 674 rc.newwanip starting igc1 2025-06-29 18:15:30.416106+02:00 php-fpm 38706 /rc.linkup: HOTPLUG: Triggering address refresh on lan (igc1) 2025-06-29 18:15:30.415927+02:00 php-fpm 38706 /rc.linkup: DEVD Ethernet attached event for lan 2025-06-29 18:15:30.404027+02:00 php-fpm 38706 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1) 2025-06-29 18:15:29.755530+02:00 xinetd 5473 Reconfigured: new=0 old=6 dropped=0 (services) 2025-06-29 18:15:29.755518+02:00 xinetd 5473 readjusting service 19005-udp 2025-06-29 18:15:29.755507+02:00 xinetd 5473 readjusting service 19004-udp 2025-06-29 18:15:29.755495+02:00 xinetd 5473 readjusting service 19003-tcp 2025-06-29 18:15:29.755484+02:00 xinetd 5473 readjusting service 19002-tcp 2025-06-29 18:15:29.755473+02:00 xinetd 5473 readjusting service 19001-tcp 2025-06-29 18:15:29.755460+02:00 xinetd 5473 readjusting service 19000-tcp 2025-06-29 18:15:29.755443+02:00 xinetd 5473 Swapping defaults 2025-06-29 18:15:29.754868+02:00 xinetd 5473 Starting reconfiguration 2025-06-29 18:15:29.351604+02:00 kernel - igc1: link state changed to UP 2025-06-29 18:15:29.321806+02:00 check_reload_status 674 Linkup starting igc1 2025-06-29 18:15:28.610587+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:15:28.610258+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:15:26.838367+02:00 php-fpm 43924 /rc.linkup: DEVD Ethernet detached event for lan 2025-06-29 18:15:26.822283+02:00 php-fpm 43924 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1) 2025-06-29 18:15:25.912662+02:00 kernel - igc1: link state changed to DOWN 2025-06-29 18:15:25.763694+02:00 php 80125 [Snort] The Rules update has finished. 2025-06-29 18:15:25.762352+02:00 php 80125 [Snort] Snort has restarted on LAN with your new set of rules... 2025-06-29 18:15:25.731979+02:00 check_reload_status 674 Linkup starting igc1 2025-06-29 18:15:25.711985+02:00 kernel - 725.693258 [ 852] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048 2025-06-29 18:15:25.711692+02:00 kernel - 725.656339 [ 852] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048 .............. 2025-06-29 18:15:09.626165+02:00 snort 78173 AppInfo: AppId 4903 is UNKNOWN 2025-06-29 18:15:09.625169+02:00 snort 78173 AppId 2025-06-29 18:15:09.625141+02:00 snort 78173 AppId 2025-06-29 18:15:09.305900+02:00 php_pfb 89791 [pfBlockerNG] filterlog daemon started 2025-06-29 18:15:09.006181+02:00 snort 78173 AppId 2025-06-29 18:15:08.960122+02:00 tail_pfb 89291 [pfBlockerNG] Firewall Filter Service started 2025-06-29 18:15:08.910306+02:00 lighttpd_pfb 85817 [pfBlockerNG] DNSBL Webserver started 2025-06-29 18:15:08.900608+02:00 php_pfb 84265 [pfBlockerNG] filterlog daemon stopped 2025-06-29 18:15:08.891790+02:00 tail_pfb 83152 [pfBlockerNG] Firewall Filter Service stopped 2025-06-29 18:15:08.886751+02:00 lighttpd_pfb 82101 [pfBlockerNG] DNSBL Webserver stopped 2025-06-29 18:15:08.811564+02:00 SnortStartup 78044 Snort START for LAN(igc1)... 2025-06-29 18:15:01.391170+02:00 php-fpm 61163 /rc.start_packages: The command '/usr/local/bin/redis-cli shutdown save' returned exit code '1', the output was 'Could not connect to Redis at 127.0.0.1:6379: Connection refused' 2025-06-29 18:15:01.246646+02:00 xinetd 5473 Reconfigured: new=0 old=6 dropped=0 (services) 2025-06-29 18:15:01.246629+02:00 xinetd 5473 readjusting service 19005-udp 2025-06-29 18:15:01.246619+02:00 xinetd 5473 readjusting service 19004-udp 2025-06-29 18:15:01.246608+02:00 xinetd 5473 readjusting service 19003-tcp 2025-06-29 18:15:01.246598+02:00 xinetd 5473 readjusting service 19002-tcp 2025-06-29 18:15:01.246587+02:00 xinetd 5473 readjusting service 19001-tcp 2025-06-29 18:15:01.246573+02:00 xinetd 5473 readjusting service 19000-tcp 2025-06-29 18:15:01.246558+02:00 xinetd 5473 Swapping defaults 2025-06-29 18:15:01.246282+02:00 php-cgi 75901 servicewatchdog_cron.php: Service Watchdog detected service ntopng stopped. Restarting ntopng (ntopng Network Traffic Monitor) 2025-06-29 18:15:01.245996+02:00 xinetd 5473 Starting reconfiguration 2025-06-29 18:14:56.201498+02:00 php-fpm 61163 /rc.start_packages: [filer] XMLRPC sync completed. 2025-06-29 18:14:56.201329+02:00 php-fpm 61163 /rc.start_packages: XMLRPC reload data success with https://172.16.0.250:443/xmlrpc.php (pfsense.exec_php). 2025-06-29 18:14:56.143762+02:00 php-fpm 61163 /rc.start_packages: Beginning XMLRPC sync data to https://172.16.0.250:443/xmlrpc.php. 2025-06-29 18:14:56.143567+02:00 php-fpm 61163 /rc.start_packages: XMLRPC reload data success with https://172.16.0.250:443/xmlrpc.php (pfsense.merge_installedpackages_section). 2025-06-29 18:14:56.039883+02:00 xinetd 5473 Reconfigured: new=0 old=6 dropped=0 (services) 2025-06-29 18:14:56.039870+02:00 xinetd 5473 readjusting service 19005-udp 2025-06-29 18:14:56.039858+02:00 xinetd 5473 readjusting service 19004-udp 2025-06-29 18:14:56.039845+02:00 xinetd 5473 readjusting service 19003-tcp 2025-06-29 18:14:56.039833+02:00 xinetd 5473 readjusting service 19002-tcp 2025-06-29 18:14:56.039819+02:00 xinetd 5473 readjusting service 19001-tcp 2025-06-29 18:14:56.039803+02:00 xinetd 5473 readjusting service 19000-tcp 2025-06-29 18:14:56.039759+02:00 xinetd 5473 Swapping defaults 2025-06-29 18:14:56.038972+02:00 xinetd 5473 Starting reconfiguration 2025-06-29 18:14:56.008761+02:00 php-fpm 61163 /rc.start_packages: Beginning XMLRPC sync data to https://172.16.0.250:443/xmlrpc.php. 2025-06-29 18:14:56.008491+02:00 php-fpm 61163 /rc.start_packages: [filer] XMLRPC sync is starting. 2025-06-29 18:14:55.961051+02:00 php-fpm 61163 /rc.start_packages: Restarting/Starting all packages. 2025-06-29 18:14:55.335888+02:00 snmpd 31536 disk_OS_get_disks: adding device 'ada0' to device list 2025-06-29 18:14:54.886390+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:14:54.886167+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:14:54.880664+02:00 check_reload_status 674 Starting packages 2025-06-29 18:14:54.877863+02:00 php-fpm 43924 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 172.16.0.1 -> 172.16.0.1 - Restarting packages. 2025-06-29 18:14:52.824789+02:00 php-fpm 43924 /rc.newwanip: Creating rrd update script 2025-06-29 18:14:52.813951+02:00 php-fpm 43924 /rc.newwanip: Resyncing OpenVPN instances for interface LAN. 2025-06-29 18:14:52.813759+02:00 php-fpm 43924 /rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface lan 2025-06-29 18:14:51.000511+02:00 php-fpm 43924 /rc.newwanip: Gateway, NONE AVAILABLE 2025-06-29 18:14:50.992722+02:00 php-fpm 43924 /rc.newwanip: Gateway, NONE AVAILABLE 2025-06-29 18:14:50.431380+02:00 php-fpm 43924 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 10.0.11.213 2025-06-29 18:14:48.302102+02:00 xinetd 5473 Reconfigured: new=0 old=6 dropped=0 (services) 2025-06-29 18:14:42.208377+02:00 xinetd 5473 readjusting service 19005-udp 2025-06-29 18:14:42.208365+02:00 xinetd 5473 readjusting service 19004-udp 2025-06-29 18:14:42.208355+02:00 xinetd 5473 readjusting service 19003-tcp 2025-06-29 18:14:42.208344+02:00 xinetd 5473 readjusting service 19002-tcp 2025-06-29 18:14:42.208331+02:00 xinetd 5473 readjusting service 19001-tcp 2025-06-29 18:14:42.208318+02:00 xinetd 5473 readjusting service 19000-tcp 2025-06-29 18:14:42.208296+02:00 xinetd 5473 Swapping defaults 2025-06-29 18:14:42.207693+02:00 xinetd 5473 Starting reconfiguration 2025-06-29 18:14:36.769668+02:00 kernel - arp: 08:00:27:66:1d:d1 is using my IP address 192.168.255.254 on igc1! 2025-06-29 18:14:36.416400+02:00 php-fpm 43924 /rc.newwanip: rc.newwanip: on (IP address: 172.16.0.1) (interface: LAN[lan]) (real interface: igc1). 2025-06-29 18:14:36.415680+02:00 php-fpm 43924 /rc.newwanip: rc.newwanip: Info: starting on igc1. 2025-06-29 18:14:35.338815+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:14:35.333780+02:00 check_reload_status 674 rc.newwanip starting igc1 2025-06-29 18:14:35.333293+02:00 php-fpm 43924 /rc.linkup: HOTPLUG: Triggering address refresh on lan (igc1) 2025-06-29 18:14:35.333114+02:00 php-fpm 43924 /rc.linkup: DEVD Ethernet attached event for lan 2025-06-29 18:14:35.319985+02:00 php-fpm 43924 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1) 2025-06-29 18:14:34.964935+02:00 xinetd 5473 Reconfigured: new=0 old=6 dropped=0 (services) 2025-06-29 18:14:34.964923+02:00 xinetd 5473 readjusting service 19005-udp 2025-06-29 18:14:34.964912+02:00 xinetd 5473 readjusting service 19004-udp 2025-06-29 18:14:34.964901+02:00 xinetd 5473 readjusting service 19003-tcp 2025-06-29 18:14:34.964891+02:00 xinetd 5473 readjusting service 19002-tcp 2025-06-29 18:14:34.964880+02:00 xinetd 5473 readjusting service 19001-tcp 2025-06-29 18:14:34.964867+02:00 xinetd 5473 readjusting service 19000-tcp 2025-06-29 18:14:34.964849+02:00 xinetd 5473 Swapping defaults 2025-06-29 18:14:34.964295+02:00 xinetd 5473 Starting reconfiguration 2025-06-29 18:14:34.357670+02:00 kernel - igc1: link state changed to UP 2025-06-29 18:14:34.233345+02:00 check_reload_status 674 Linkup starting igc1 2025-06-29 18:14:33.851814+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:14:33.820590+02:00 check_reload_status 674 Reloading filter 2025-06-29 18:14:33.619210+02:00 snort 48132 AppInfo: AppId 2314 is UNKNOWN 2025-06-29 18:14:33.619161+02:00 snort 48132 Invalid direct service AppId, 2314, for 0x841db4d60 0x2cf3fffcd880 ........ 2025-06-29 18:14:33.589595+02:00 snort 48132 AppInfo: AppId 4140 is UNKNOWN 2025-06-29 18:14:33.589520+02:00 snort 48132 AppInfo: AppId 4140 is UNKNOWN 2025-06-29 18:14:33.588731+02:00 snort 48132 AppId 2025-06-29 18:14:33.588575+02:00 snort 48132 AppId 2025-06-29 18:14:32.941276+02:00 snort 48132 AppId 2025-06-29 18:14:32.733089+02:00 php 80125 [Snort] Snort START for LAN(igc1)... 2025-06-29 18:14:31.926833+02:00 php-fpm 38706 /rc.linkup: DEVD Ethernet detached event for lan 2025-06-29 18:14:31.906676+02:00 php-fpm 38706 /rc.linkup: Hotplug event detected for LAN(lan) static IP address (4: 172.16.0.1) 2025-06-29 18:14:30.713792+02:00 kernel - igc1: link state changed to DOWN 2025-06-29 18:14:30.672230+02:00 check_reload_status 674 Linkup starting igc1 2025-06-29 18:14:30.641688+02:00 snort 6713 *** Caught Term-Signal 2025-06-29 18:14:30.634302+02:00 php 80125 [Snort] Snort STOP for LAN(igc1)...Update:
This only happens when Inline IPS mode is enabled.
If you disable Blocking Mode or switch to Legacy Mode, the update runs fine—the interfaces restart correctly and everything works as expected.
However, when Blocking Mode is set to Inline IPS, that’s when the system hangs and behaves erratically.
