Multiple issues, firewall freezes and whole network goes down.

Laxarus

@stephenw10 I have removed the captive portal from service watchdog list and I don't see it failing to start. Btw what do you mean by this?

@stephenw10 said in Multiple issues, firewall freezes and whole network goes down.:

The service watchdog should really only be used for troubleshooting.

I am using service watchdog for every service I employ in case something crashes. Is it wrong?

For ipv6, why might be the reason that the firewall is trying to forward a package especially from a link local?

I use AOC-S25G-i2S with intel XXV710 with LAG and symptoms seem to match exactly but is it normal that it freezes the firewall? This become a critical issue for me now as all my vlans and lan as well as most of the communication go through this LAG.

stephenw10

Yes, do not use the service watchdog unless you're seeing specific issues. It will trigger unnecessary restarts for services when they are restarted normally and you can end up with a far higher processes restart load and time then required.

Check what device is using the link-local address. Why is it sending traffic using that?

I only know what's on the bug there. As I understand it it should only prevent multicast traffic. However if you have something dependent on that it might cause other issues.
I've never managed to hit it locally to dig deeper.

Laxarus

@stephenw10 I see, thanks for the information. I used service watchdog in case something crashes and need a restart and I guess this was unnecessary.

As for this bug, I use PIMD, avahi and udp broadcast simultaneously so is it possible that these services might freeze up the firewall? I am especially suspicious of avahi as it sometimes fails to start with below error.

Sep  3 14:30:05 FIREWALL avahi-daemon[69665]: Loading service file /usr/local/etc/avahi/services/ssh.service.
Sep  3 14:30:05 FIREWALL avahi-daemon[69665]: *** WARNING: Detected another IPv4 mDNS stack running on this host. This makes mDNS unreliable and is thus not recommended. ***
Sep  3 14:30:05 FIREWALL avahi-daemon[69665]: bind() failed: Address already in use
Sep  3 14:30:05 FIREWALL avahi-daemon[69665]: Failed to create server: No suitable network protocol available

So, how about this theory,
1- Modem restarts and WAN goes down
2- Modem and WAN comes back again
3- firewall establishes pppoe link
4- the firewall for some reason restarts these packages (extract from pastenin log)

Sep  3 06:03:12 FIREWALL check_reload_status[635]: updating dyndns MNG_DHCP
Sep  3 06:03:12 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 06:03:12 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 06:03:12 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 06:03:13 FIREWALL php-fpm[50644]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use MNG_DHCP.
Sep  3 06:03:17 FIREWALL php-fpm[49205]: /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1725332597] unbound[7884:0] error: bind: address already in use [1725332597] unbound[7884:0] fatal error: could not open ports' 
Sep  3 06:03:18 FIREWALL php-fpm[49205]: /rc.newwanip: Resyncing OpenVPN instances for interface MODEM.
Sep  3 06:03:18 FIREWALL php-fpm[49205]: /rc.newwanip: Creating rrd update script
Sep  3 06:03:20 FIREWALL php-fpm[49205]: /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 ->  192.168.0.2 - Restarting packages.
Sep  3 06:03:20 FIREWALL check_reload_status[635]: Starting packages
Sep  3 06:03:20 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 06:03:20 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 06:03:21 FIREWALL php-fpm[87388]: /rc.start_packages: Restarting/Starting all packages.
Sep  3 06:03:21 FIREWALL php-fpm[87388]: /rc.start_packages: Stopping service avahi
Sep  3 06:03:21 FIREWALL avahi-daemon[48473]: Got SIGTERM, quitting.
Sep  3 06:03:21 FIREWALL php-fpm[87388]: /rc.start_packages: Starting service avahi
Sep  3 06:03:21 FIREWALL avahi-daemon[48473]: Leaving mDNS multicast group on interface lagg0.150.IPv4 with address 192.168.150.1.
Sep  3 06:03:21 FIREWALL avahi-daemon[48473]: Leaving mDNS multicast group on interface lagg0.25.IPv4 with address 192.168.25.1.
Sep  3 06:03:21 FIREWALL avahi-daemon[48473]: Leaving mDNS multicast group on interface lagg0.135.IPv4 with address 192.168.135.1.
Sep  3 06:03:21 FIREWALL avahi-daemon[48473]: Leaving mDNS multicast group on interface lagg0.33.IPv4 with address 192.168.33.1.
Sep  3 06:03:21 FIREWALL avahi-daemon[48473]: Leaving mDNS multicast group on interface lagg0.IPv4 with address 10.10.10.10.
Sep  3 06:03:21 FIREWALL avahi-daemon[48473]: avahi-daemon 0.8 exiting.
Sep  3 06:03:21 FIREWALL php-fpm[87388]: 
Sep  3 06:03:21 FIREWALL php-fpm[87388]: 
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
Sep  3 06:03:21 FIREWALL kernel: ixl0: Disabled multicast promiscuous mode
.
.
.

then the package crashes and system watchdog tries to restart them again and they crash again because of this bug, then we have a loop, then firewall freezes.

stephenw10

Yes and if you have the watchdog enabled for all services it will almost certainly trigger for some of them during the normal package restart that happens when the WAN IP changes. That can lead to either the services being restarted again or, worse, multiple processes started causing conflicts. Possibly more restarts and more watchdog triggers! That can easily escalate into a multi minute process or, as you say, a loop that eventually consumes enough resources the firewall stops responding.

Yes generally running 3 services all trying to listen to the same traffic on the same interface isn't going to end well!

Laxarus

@stephenw10 it is good to know that. In the meantime, I had another freeze again and this time with a more weird logs.
I did as you suggested and removed all the service from the service watchdog. Then around 22:40, I have lost contact with the remote site again. After again doing a power painfully hard power reboot. I see these logs.

Sep  3 22:38:00 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:38:00 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:38:01 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:39:12 FIREWALL rc.gateway_alarm[43932]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:0 RTT:475.895ms RTTsd:101.631ms Loss:1%)
Sep  3 22:39:12 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:39:12 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:39:12 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:39:12 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:39:13 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue over limit
Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
.
.
.

then reading more logs
I see UPS data stale logs

Sep  3 22:39:35 FIREWALL kernel: fq_codel_enqueue maxidx = 163
Sep  3 22:39:38 FIREWALL rc.gateway_alarm[32956]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:501.320ms RTTsd:115.292ms Loss:2%)
Sep  3 22:39:38 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:39:38 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:39:38 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:39:38 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:39:39 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:40:48 FIREWALL rc.gateway_alarm[45337]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:0 RTT:493.722ms RTTsd:131.424ms Loss:0%)
Sep  3 22:40:48 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:40:48 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:40:48 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:40:48 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:40:49 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:40:58 FIREWALL rc.gateway_alarm[94206]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:503.502ms RTTsd:132.421ms Loss:0%)
Sep  3 22:40:58 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:40:58 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:40:58 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:40:58 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:40:59 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:41:00 FIREWALL sshguard[94827]: Exiting on signal.
Sep  3 22:41:00 FIREWALL sshguard[37387]: Now monitoring attacks.
Sep  3 22:44:00 FIREWALL sshguard[37387]: Exiting on signal.
Sep  3 22:44:00 FIREWALL sshguard[46412]: Now monitoring attacks.
Sep  3 22:44:01 FIREWALL rc.gateway_alarm[51747]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:0 RTT:493.256ms RTTsd:100.247ms Loss:1%)
Sep  3 22:44:01 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:44:01 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:44:01 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:44:01 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:44:02 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:44:08 FIREWALL rc.gateway_alarm[10181]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:500.345ms RTTsd:92.345ms Loss:2%)
Sep  3 22:44:08 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:44:08 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:44:08 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:44:08 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:44:10 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:46:00 FIREWALL sshguard[46412]: Exiting on signal.
Sep  3 22:46:00 FIREWALL sshguard[94101]: Now monitoring attacks.
Sep  3 22:50:00 FIREWALL sshguard[94101]: Exiting on signal.
Sep  3 22:50:00 FIREWALL sshguard[50856]: Now monitoring attacks.
Sep  3 22:51:37 FIREWALL rc.gateway_alarm[69138]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:0 RTT:487.545ms RTTsd:95.405ms Loss:1%)
Sep  3 22:51:37 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:51:37 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:51:37 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:51:37 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:51:38 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:54:00 FIREWALL sshguard[50856]: Exiting on signal.
Sep  3 22:54:00 FIREWALL sshguard[87930]: Now monitoring attacks.
Sep  3 22:54:59 FIREWALL rc.gateway_alarm[50114]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:501.285ms RTTsd:76.221ms Loss:0%)
Sep  3 22:54:59 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:54:59 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:54:59 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:54:59 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:55:01 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:55:53 FIREWALL rc.gateway_alarm[39552]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:0 RTT:487.635ms RTTsd:176.075ms Loss:7%)
Sep  3 22:55:53 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:55:53 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:55:53 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:55:53 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:55:54 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:57:00 FIREWALL sshguard[87930]: Exiting on signal.
Sep  3 22:57:00 FIREWALL sshguard[77852]: Now monitoring attacks.
Sep  3 22:57:51 FIREWALL rc.gateway_alarm[18591]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:501.239ms RTTsd:74.600ms Loss:2%)
Sep  3 22:57:51 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:57:51 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:57:51 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:57:51 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:57:52 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:58:00 FIREWALL sshguard[77852]: Exiting on signal.
Sep  3 22:58:00 FIREWALL sshguard[90215]: Now monitoring attacks.
Sep  3 22:58:16 FIREWALL rc.gateway_alarm[75986]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:0 RTT:488.383ms RTTsd:116.574ms Loss:1%)
Sep  3 22:58:16 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:58:16 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:58:16 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:58:16 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:58:17 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:58:38 FIREWALL rc.gateway_alarm[18506]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:503.587ms RTTsd:166.697ms Loss:1%)
Sep  3 22:58:38 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  3 22:58:38 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:58:38 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:58:38 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:58:39 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  3 22:59:00 FIREWALL rc.gateway_alarm[52935]: >>> Gateway alarm: VPNAC_WG (Addr:10.11.0.1 Alarm:1 RTT:107.041ms RTTsd:128.454ms Loss:21%)
Sep  3 22:59:00 FIREWALL check_reload_status[635]: updating dyndns VPNAC_WG
Sep  3 22:59:00 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:59:00 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:59:00 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:59:01 FIREWALL rc.gateway_alarm[55652]: >>> Gateway alarm: WANV6_TUNNELV6 (Addr:2001:470:1f1a:46d::1 Alarm:1 RTT:108.930ms RTTsd:200.628ms Loss:21%)
Sep  3 22:59:01 FIREWALL check_reload_status[635]: updating dyndns WANV6_TUNNELV6
Sep  3 22:59:01 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:59:01 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:59:01 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:59:01 FIREWALL php-fpm[67909]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Sep  3 22:59:02 FIREWALL rc.gateway_alarm[8021]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:19.134ms RTTsd:100.354ms Loss:22%)
Sep  3 22:59:02 FIREWALL php-fpm[67909]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WANV6_TUNNELV6.
Sep  3 22:59:10 FIREWALL upsd[88284]: Data for UPS [UPS] is stale - check driver
Sep  3 22:59:16 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for device.description
Sep  3 22:59:19 FIREWALL rc.gateway_alarm[69140]: >>> Gateway alarm: MNG_DHCP (Addr:192.168.2.1 Alarm:1 RTT:17.536ms RTTsd:100.282ms Loss:21%)
Sep  3 22:59:19 FIREWALL check_reload_status[635]: updating dyndns MNG_DHCP
Sep  3 22:59:19 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:59:19 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:59:19 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:59:20 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use MNG_DHCP.
Sep  3 22:59:24 FIREWALL rc.gateway_alarm[84753]: >>> Gateway alarm: MODEM_DHCP (Addr:192.168.0.1 Alarm:1 RTT:3.491ms RTTsd:15.092ms Loss:22%)
Sep  3 22:59:24 FIREWALL check_reload_status[635]: updating dyndns MODEM_DHCP
Sep  3 22:59:24 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:59:24 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:59:24 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:59:25 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use MODEM_DHCP.
Sep  3 22:59:39 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for device.contact
Sep  3 22:59:43 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Server disconnected
Sep  3 22:59:43 FIREWALL upsmon[92822]: Communications with UPS UPS lost
Sep  3 22:59:46 FIREWALL rc.gateway_alarm[8513]: >>> Gateway alarm: WANV6_TUNNELV6 (Addr:2001:470:1f1a:46d::1 Alarm:1 RTT:688.758ms RTTsd:625.886ms Loss:98%)
Sep  3 22:59:46 FIREWALL check_reload_status[635]: updating dyndns WANV6_TUNNELV6
Sep  3 22:59:46 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:59:46 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:59:46 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:59:47 FIREWALL php-fpm[19267]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WANV6_TUNNELV6.
Sep  3 22:59:48 FIREWALL upsd[88284]: User local-monitor@127.0.0.1 logged into UPS [UPS]
Sep  3 22:59:48 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 22:59:53 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 22:59:56 FIREWALL rc.gateway_alarm[24827]: >>> Gateway alarm: WANV6_TUNNELV6 (Addr:2001:470:1f1a:46d::1 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Sep  3 22:59:56 FIREWALL check_reload_status[635]: updating dyndns WANV6_TUNNELV6
Sep  3 22:59:56 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  3 22:59:56 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  3 22:59:56 FIREWALL check_reload_status[635]: Reloading filter
Sep  3 22:59:58 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 22:59:58 FIREWALL php-fpm[30071]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WANV6_TUNNELV6.
Sep  3 23:00:01 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for device.location
Sep  3 23:00:03 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:08 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:13 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:18 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:23 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for input.voltage
Sep  3 23:00:23 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:28 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:33 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:38 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:43 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:45 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for input.voltage.maximum
Sep  3 23:00:48 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:53 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:00:58 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:03 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:07 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for input.voltage.minimum
Sep  3 23:01:08 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:11 FIREWALL php-cgi[5147]: notify_monitor.php: Could not send the message to "redacted mail"@gmail.com -- Error: Failed to connect to ssl://smtp.gmail.com:465 [SMTP: Failed to connect socket: php_network_getaddresses: getaddrinfo for smtp.gmail.com failed: Address family for hostname not supported (code: -1, response: )]
Sep  3 23:01:13 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:18 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:23 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:28 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:29 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for input.frequency
Sep  3 23:01:33 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:38 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:43 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:48 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:51 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for input.transfer.low
Sep  3 23:01:53 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:01:58 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:02:03 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:02:08 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:02:13 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:02:13 FIREWALL snmp-ups[92044]: [UPS] snmp_ups_walk: data stale for input.transfer.high

and console again not responsive with below output

until reboot

Sep  3 23:53:57 FIREWALL upsmon[92822]: UPS UPS is unavailable
Sep  3 23:54:02 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:07 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:12 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:17 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:22 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:27 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:32 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:38 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:43 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:48 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:53 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:54:58 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:03 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:08 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:13 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:16 FIREWALL php-cgi[96287]: notify_monitor.php: Could not send the message to laxarus@gmail.com -- Error: Failed to connect to ssl://smtp.gmail.com:465 [SMTP: Failed to connect socket: php_network_getaddresses: getaddrinfo for smtp.gmail.com failed: Address family for hostname not supported (code: -1, response: )]
Sep  3 23:55:18 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:23 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:28 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:33 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:38 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:42 FIREWALL login[14908]: login on ttyv0 as root
Sep  3 23:55:42 FIREWALL login[24544]: login on ttyv0 as root
Sep  3 23:55:43 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:48 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:53 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:55:58 FIREWALL upsmon[92822]: Poll UPS [UPS] failed - Data stale
Sep  3 23:57:52 FIREWALL syslogd: kernel boot file is /boot/kernel/kernel
Sep  3 23:57:52 FIREWALL kernel: ---<<BOOT>>---
Sep  3 23:57:52 FIREWALL kernel: Copyright (c) 1992-2024 The FreeBSD Project.
Sep  3 23:57:52 FIREWALL kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Sep  3 23:57:52 FIREWALL kernel: 	The Regents of the University of California. All rights reserved.
Sep  3 23:57:52 FIREWALL kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Sep  3 23:57:52 FIREWALL kernel: FreeBSD 15.0-CURRENT #0 plus-RELENG_24_03-n256311-e71f834dd81: Fri Apr 19 00:28:14 UTC 2024
Sep  3 23:57:52 FIREWALL kernel:

What is wrong with fq_codel and UPS I have no idea? The console froze and did not register the UPS data stale errors after fq_codel errors. I did not have this issue before and nothing changed for these settings for around a year.
The only limiter is on WAN as described in the netgate cookbook limiter for bufer bloating.

stephenw10

No idea about the UPS, it appears to just not be responding. How is it connected.

There are a bunch of gateway alarms on multiple gateways for latency and packet loss. Check the Status > Monitoring graphs. Was the firewall under huge load at that time? >500ms across a VPN is very high.

Laxarus

@stephenw10 UPS is snmp so my guess is fq_codel errors froze the network and the firewall lost comms with UPS as a result then the "ups stale" errors.

The 500ms across a VPN is normal as the internet connection on the other end of the tunnel is very bad.

stephenw10

Ah then you should definitely tune the gateway alert settings so it doesn't cause an alarm at that latency. It's causing a lot of unnecessary scripts to run.

Laxarus

@stephenw10 got it. Btw thanks for bearing with me until now (:. But is this somehow related to freeze and crash?

stephenw10

It could be if it's somehow ending up with scripts looping until it uses all available resources of some kind. The monitoring graphs should show that though as I mentioned

Laxarus

@stephenw10 I have checked the graphs, but nothing seems wrong with the values. I had also another freeze. There is nothing useful on the logs other than some gateway alarms.
I have noticed a common issue across all these crashes/freezes though. As soon as something goes wrong with my WAN, there is a big chance that the firewall also freezes/crashes. I cannot make sense of it though.
So far,
I have removed all the watchdog items
tweaked the gatewayvalues a little bit
and after this crash I have removed the HE Tunnel since it was useless.

Sep  8 17:36:24 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:36:25 FIREWALL php-fpm[34190]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Sep  8 17:36:27 FIREWALL rc.gateway_alarm[40780]: >>> Gateway alarm: WANV6_TUNNELV6 (Addr:2001:470:1f1a:46d::1 Alarm:1 RTT:510.886ms RTTsd:361.034ms Loss:11%)
Sep  8 17:36:27 FIREWALL check_reload_status[635]: updating dyndns WANV6_TUNNELV6
Sep  8 17:36:27 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:36:27 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:36:27 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:36:28 FIREWALL php-fpm[34190]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WANV6_TUNNELV6.
Sep  8 17:36:30 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  8 17:36:30 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:36:30 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:36:30 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:36:30 FIREWALL rc.gateway_alarm[58334]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:707.150ms RTTsd:342.738ms Loss:11%)
Sep  8 17:36:31 FIREWALL php-fpm[40188]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  8 17:36:47 FIREWALL rc.gateway_alarm[20939]: >>> Gateway alarm: VPNAC_WG (Addr:10.11.0.1 Alarm:1 RTT:574.422ms RTTsd:376.129ms Loss:21%)
Sep  8 17:36:47 FIREWALL check_reload_status[635]: updating dyndns VPNAC_WG
Sep  8 17:36:47 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:36:47 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:36:47 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:36:48 FIREWALL rc.gateway_alarm[24045]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:500.131ms RTTsd:368.558ms Loss:16%)
Sep  8 17:36:48 FIREWALL check_reload_status[635]: updating dyndns WAN_PPPOE
Sep  8 17:36:48 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:36:48 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:36:48 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:36:48 FIREWALL php-fpm[40188]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Sep  8 17:36:49 FIREWALL php-fpm[36133]: /rc.dyndns.update: phpDynDNS (@.mydomain.org): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep  8 17:36:49 FIREWALL php-fpm[40188]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_PPPOE.
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS: updatedns() starting
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _checkIP() starting.
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): "redacted ip" extracted from local system.
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS (931952): running get_failover_interface for wan. found pppoe0
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _detectChange() starting.
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _checkIP() starting.
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): "redacted ip" extracted from local system.
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic Dns (931952): Current WAN IP: "redacted ip" Cached IP: "redacted ip"
Sep  8 17:36:50 FIREWALL php-fpm[36133]: /rc.dyndns.update: phpDynDNS (931952): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep  8 17:36:51 FIREWALL rc.gateway_alarm[79606]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:518.989ms RTTsd:369.829ms Loss:21%)
Sep  8 17:36:51 FIREWALL check_reload_status[635]: updating dyndns WAN_PPPOE
Sep  8 17:36:51 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:36:51 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:36:51 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:36:51 FIREWALL rc.gateway_alarm[80565]: >>> Gateway alarm: WANV6_TUNNELV6 (Addr:2001:470:1f1a:46d::1 Alarm:1 RTT:578.630ms RTTsd:391.247ms Loss:22%)
Sep  8 17:36:51 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:36:51 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:36:52 FIREWALL php-fpm[639]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WANV6_TUNNELV6.
Sep  8 17:36:52 FIREWALL php-fpm[36133]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_PPPOE.
Sep  8 17:37:02 FIREWALL rc.gateway_alarm[4317]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:735.071ms RTTsd:303.770ms Loss:37%)
Sep  8 17:37:02 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  8 17:37:02 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:02 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:02 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:37:03 FIREWALL php-fpm[639]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  8 17:37:07 FIREWALL rc.gateway_alarm[60969]: >>> Gateway alarm: MNG_DHCP (Addr:192.168.2.1 Alarm:1 RTT:24.549ms RTTsd:124.820ms Loss:21%)
Sep  8 17:37:07 FIREWALL check_reload_status[635]: updating dyndns MNG_DHCP
Sep  8 17:37:07 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:07 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:08 FIREWALL php-fpm[40188]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use MNG_DHCP.
Sep  8 17:37:12 FIREWALL rc.gateway_alarm[27585]: >>> Gateway alarm: MODEM_DHCP (Addr:192.168.0.1 Alarm:1 RTT:23.050ms RTTsd:124.997ms Loss:22%)
Sep  8 17:37:12 FIREWALL check_reload_status[635]: updating dyndns MODEM_DHCP
Sep  8 17:37:12 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:12 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:12 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:37:13 FIREWALL php-fpm[592]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use MODEM_DHCP.
Sep  8 17:37:16 FIREWALL upsd[6640]: Data for UPS [UPS] is stale - check driver
Sep  8 17:37:19 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:19 FIREWALL upsmon[19346]: Communications with UPS UPS lost
Sep  8 17:37:22 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for device.description
Sep  8 17:37:24 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:29 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:31 FIREWALL rc.gateway_alarm[47891]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:383.257ms RTTsd:388.087ms Loss:81%)
Sep  8 17:37:31 FIREWALL check_reload_status[635]: updating dyndns WAN_PPPOE
Sep  8 17:37:31 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:31 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:31 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:37:32 FIREWALL php-fpm[36133]: /rc.dyndns.update: phpDynDNS (@.mydomain.org): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep  8 17:37:32 FIREWALL php-fpm[639]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_PPPOE.
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS: updatedns() starting
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _checkIP() starting.
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): "redacted ip" extracted from local system.
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS (931952): running get_failover_interface for wan. found pppoe0
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _detectChange() starting.
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _checkIP() starting.
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): "redacted ip" extracted from local system.
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: Dynamic Dns (931952): Current WAN IP: "redacted ip" Cached IP: "redacted ip"
Sep  8 17:37:33 FIREWALL php-fpm[36133]: /rc.dyndns.update: phpDynDNS (931952): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep  8 17:37:34 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:38 FIREWALL rc.gateway_alarm[62940]: >>> Gateway alarm: WANV6_TUNNELV6 (Addr:2001:470:1f1a:46d::1 Alarm:1 RTT:266.937ms RTTsd:155.977ms Loss:93%)
Sep  8 17:37:38 FIREWALL check_reload_status[635]: updating dyndns WANV6_TUNNELV6
Sep  8 17:37:38 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:38 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:38 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:37:39 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:39 FIREWALL php-fpm[639]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WANV6_TUNNELV6.
Sep  8 17:37:40 FIREWALL rc.gateway_alarm[79115]: >>> Gateway alarm: VPNAC_WG (Addr:10.11.0.1 Alarm:1 RTT:225.111ms RTTsd:55.236ms Loss:96%)
Sep  8 17:37:40 FIREWALL check_reload_status[635]: updating dyndns VPNAC_WG
Sep  8 17:37:40 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:40 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:40 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:37:40 FIREWALL rc.gateway_alarm[81903]: >>> Gateway alarm: OVPN_S2S_VPNV4 (Addr:10.25.25.2 Alarm:1 RTT:436.319ms RTTsd:43.501ms Loss:97%)
Sep  8 17:37:40 FIREWALL check_reload_status[635]: updating dyndns OVPN_S2S_VPNV4
Sep  8 17:37:40 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:40 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:41 FIREWALL php-fpm[90111]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPNAC_WG.
Sep  8 17:37:41 FIREWALL rc.gateway_alarm[96096]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:806.992ms RTTsd:750.675ms Loss:98%)
Sep  8 17:37:41 FIREWALL php-fpm[639]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use OVPN_S2S_VPNV4.
Sep  8 17:37:44 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:44 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for device.contact
Sep  8 17:37:49 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:51 FIREWALL rc.gateway_alarm[12424]: >>> Gateway alarm: WAN_PPPOE (Addr:10.98.238.224 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
Sep  8 17:37:51 FIREWALL check_reload_status[635]: updating dyndns WAN_PPPOE
Sep  8 17:37:51 FIREWALL check_reload_status[635]: Restarting IPsec tunnels
Sep  8 17:37:51 FIREWALL check_reload_status[635]: Restarting OpenVPN tunnels/interfaces
Sep  8 17:37:51 FIREWALL check_reload_status[635]: Reloading filter
Sep  8 17:37:52 FIREWALL php-fpm[90111]: /rc.dyndns.update: phpDynDNS (@.mydomain.org): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep  8 17:37:52 FIREWALL php-fpm[36133]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_PPPOE.
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic DNS: updatedns() starting
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _checkIP() starting.
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): "redacted ip" extracted from local system.
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic DNS (931952): running get_failover_interface for wan. found pppoe0
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _detectChange() starting.
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): _checkIP() starting.
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic DNS he-net-tunnelbroker (931952): "redacted ip" extracted from local system.
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: Dynamic Dns (931952): Current WAN IP: "redacted ip" Cached IP: "redacted ip"
Sep  8 17:37:53 FIREWALL php-fpm[90111]: /rc.dyndns.update: phpDynDNS (931952): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep  8 17:37:54 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:37:59 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:04 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:06 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for device.location
Sep  8 17:38:09 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:14 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:19 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:24 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:28 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for input.voltage
Sep  8 17:38:29 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:34 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:39 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:44 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:47 FIREWALL php-cgi[43129]: notify_monitor.php: Could not send the message to mymail@gmail.com -- Error: Failed to connect to ssl://smtp.gmail.com:465 [SMTP: Failed to connect socket: php_network_getaddresses: getaddrinfo for smtp.gmail.com failed: Address family for hostname not supported (code: -1, response: )]
Sep  8 17:38:49 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:50 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for input.voltage.maximum
Sep  8 17:38:54 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:38:59 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:04 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:09 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:13 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for input.voltage.minimum
Sep  8 17:39:14 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:19 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:24 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:29 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:34 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:35 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for input.frequency
Sep  8 17:39:39 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:44 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:49 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:54 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:39:57 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for input.transfer.low
Sep  8 17:39:59 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:04 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:09 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:14 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:19 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for input.transfer.high
Sep  8 17:40:19 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:24 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:29 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:34 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:39 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:41 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for input.transfer.reason
Sep  8 17:40:44 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:49 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:54 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:40:59 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:03 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.power
Sep  8 17:41:04 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:09 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:15 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:20 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:25 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:25 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.realpower
Sep  8 17:41:30 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:35 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:40 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:45 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:48 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.status
Sep  8 17:41:50 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:41:55 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:00 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:05 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:10 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.status
Sep  8 17:42:10 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:15 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:15 FIREWALL upsmon[19346]: UPS UPS is unavailable
Sep  8 17:42:20 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:25 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:30 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:32 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.status
Sep  8 17:42:35 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:40 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:45 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:50 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:42:54 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.status
Sep  8 17:42:55 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:00 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:05 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:10 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:15 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:16 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.temperature
Sep  8 17:43:20 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:25 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:30 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:35 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:38 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for ups.load
Sep  8 17:43:40 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:44 FIREWALL php-cgi[63749]: notify_monitor.php: Could not send the message to mymail@gmail.com -- Error: Failed to connect to ssl://smtp.gmail.com:465 [SMTP: Failed to connect socket: php_network_getaddresses: getaddrinfo for smtp.gmail.com failed: Address family for hostname not supported (code: -1, response: )]
Sep  8 17:43:45 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:50 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:43:55 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:00 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:00 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for battery.charge
Sep  8 17:44:05 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:10 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:15 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:20 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:23 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for battery.runtime
Sep  8 17:44:25 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:30 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:35 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:40 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:45 FIREWALL snmp-ups[18117]: [UPS] snmp_ups_walk: data stale for battery.runtime.low
Sep  8 17:44:45 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale
Sep  8 17:44:50 FIREWALL upsmon[19346]: Poll UPS [UPS] failed - Data stale

Laxarus

A short time later, this time I got an another crash with crash report.

Dump header from device: /dev/nda0p2
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 617472
  Blocksize: 512
  Compression: none
  Dumptime: 2024-09-08 20:08:22 +0300
  Hostname: FIREWALL.mydomain.org
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 15.0-CURRENT #0 plus-RELENG_24_03-n256311-e71f834dd81: Fri Apr 19 00:28:14 UTC 2024
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_03-main/obj/amd64/Y4MAEJ2R/var/j
  Panic String: page fault
  Dump Parity: 44932402
  Bounds: 0
  Dump Status: good

Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 08
fault virtual address	= 0x1c
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80f246e2
stack pointer	        = 0x28:0xfffffe00e1f3bae0
frame pointer	        = 0x28:0xfffffe00e1f3bb70
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 2 (clock (6))
rdi: 0000000000000000 rsi: 0000000000000000 rdx: fffffe00e1f3bcf8
rcx: 0000000000000000  r8: 0000000000000528  r9: 0000000000000000
rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe00e1f3bb70
r10: 000000000000300f r11: 0000000000015069 r12: 0000000000000000
r13: 0000000000000528 r14: fffff8027dfb5000 r15: 0000000000000034
trap number		= 12
panic: page fault
cpuid = 6
time = 1725815302
KDB: enter: panic
                                                                        panic.txt                                                                                           0600    0       0       12          14667355006   7145                                                                                                       ustar   root                            wheel                                                                                                                                                                                                                  page fault                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      version.txt                                                                                         0600    0       0       457         14667355006   7635                                                                                                       ustar   root                            wheel                                                                                                                                                                                                                  FreeBSD 15.0-CURRENT #0 plus-RELENG_24_03-n256311-e71f834dd81: Fri Apr 19 00:28:14 UTC 2024
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_03-main/obj/amd64/Y4MAEJ2R/var/jenkins/workspace/pfSense-Plus-snapshots-24_03-main/sources/FreeBSD-src-plus-RELENG_24_03/amd64.amd64/sys/pfSense

full crash dump here
I see a lot of "Disabled multicast promiscuous mode" outputs here.
textdump.tar.0

right now, my ISP is working on the cables in the neighborhood and I am having frequent WAN downtime but for some reason, this is crashing the firewall.

stephenw10

Ok that crash is this: https://redmine.pfsense.org/issues/15684

Try setting the workaround suggested there: https://redmine.pfsense.org/issues/15684#note-12

The logs show all gateways going down including what looks like an internal gateway?

Sep  8 17:37:07 FIREWALL rc.gateway_alarm[60969]: >>> Gateway alarm: MNG_DHCP (Addr:192.168.2.1 Alarm:1 RTT:24.549ms RTTsd:124.820ms Loss:21%)

Are all those gateways using the same NIC(s)?

Laxarus

@stephenw10 I have set the workaround though I had to set it manualy from system tunables sine it was not there by default.

There are 5 gateways with corresponding Interfaces

and the interfaces below

stephenw10

Ok so 4 of those gateways are all using igb1 but the MNG gtaeway uses igb0. So you would not expect to see all 5 throwing packet loss unless they go through the same switch maybe?

Laxarus

@stephenw10 yep, igb1 goes to modem port and igb0 goes to different switch. MNG is a management network with a separate switch with dhcp server not connected to the internet. It has all the IPMI and critical management connections. The purpose is to provide an environment where even if the pfsense crashes, management interface should stay up to reach pfsense UI (if possible) and IPMI

stephenw10

Hmm, what hardware is this?

Not much can cause two NICs to stop passing traffic like that. Especially igb NICs.

Laxarus

@stephenw10 it is Supermicro SuperServer 5019D-4C-FN8TP with 32GB ECC RAM and with addon card AOC-S25G-I2S-O PCIe SFPP28 25gbps

Laxarus

@stephenw10 to make it clear, the firewall just freezes itself, even directly connecting to the console, no inputs are registered by the firewall through console. Until reboot, it is just at stuck at something.

stephenw10

Hmm, so all 4 of those ports are on-board.

Does it not respond even to ctl+t?