DNS/DHCP stop working suddenly
-
Yes, exactly I don't expect to just get hung up in the interface/network like that. You certainly can end up with a lot of script 'churn' when an interface drops out. Especially if there are other interfaces on it like VPNs etc. But it will stop after a few minutes unless the interface changes again.
-
@michmoor said in DNS/DHCP stop working suddenly:
@stephenw10
When @cyberconsultants mentioned suricata i did immediately go to my service watchdog and I have the following enabled.
This is a set up Ive had for years now. I don't mind removing these services from monitoring but i don't see how it prevented a reboot. Ive upgraded/rebooted many times with these enabled.
edit: I would also say that its likely something environmentally changed here but I have no idea what it could be. Short of adding a firewall rule, the config is static. Goes down for an upgrade every few months. So whatever is causing a cascade of headaches for me (love the term) its proving difficult to isolate.
One of the issues with Service Watchdog is that the package is not super smart. It simply looks for the presence or absence of the monitored daemon, and sends it a start command if missing. Lots of things can result in normal restarts of certain daemons --
unboundbeing one of those daemons. Consider this scenario:unboundcan be normally restarted by several things:-
If you have DHCP hostname registration enabled (most don't, but if you do) it will restart
unboundeach time a DHCP lease renews. But Service Watchdog is not aware of that. It will simply blindly seeunboundnot running at the instant it checks and then send it a restart command without knowing the service is already in the middle of restarting. -
A temporary issue with your WAN (packet loss, for example) might trigger a gateway alarm. If that deteriorates into a down condition, pfSense will issue a command to restart a bunch of processes including
unbound. Again, Service Watchdog is ignorant of this process. It will seeunboundmissing from the list of running daemons when it spot checks, and thus issue a restart command. But pfSense itself is already restartingunbound. That can lead to problems. -
If you run pfBlockerNG, it can also issue restart commands to
unboundin some update scenarios. Service Watchdog is ignorant of this process as well, and will just blindly issue a restart command when it fails to see a runningunbounddaemon.
So, looking at the scenarios above you can see that it would be possible for bad things to happen if two
unboundinstances are trying to both start simultaneously. And whether or not this might occur would be random as it would depend on the exact timing differences between the natural restart process and when Service Watchdog sends it own independent restart signal.As @stephenw10 mentioned, Service Watchdog is really only intended for use when debugging a problem or working around some known issue. It is not a great solution for routine monitoring - especially monitoring of processes that can be restarted for legitimate reasons. This is why I preach to never use it with the IDS/IPS packages. Service Watchdog just does not consider if a service is legitimately restarting and thus the missing daemon instance is expected (and to be fair, it can't know this). It just says "no daemon present at this instant, so restart!".
-
-
Well lets dissect this a bit then. As far as Gateway actions are concerned its disabled. Has been for a while
So if the trigger is my WAN address releasing/renewing an address why all of a sudden?
Based on the alerts from monitoring this problem started at 12:47am ET. Lets review what i could dig up in graylog.
Ping monitors to my WAN address also fail at this time. So the WAN address was not accessible.
During this time the filterlog still shows traffic inbound to my old IP so i am inferring that traffic was still making it into the firewall. In fact traffic continues to come inbound until 2:47:58am ET where there is a dhclient process starting which of course is the DHCP call out on the WAN.
The link flaps a few times
Thats when things crap the bed and no more inbound flows
timestamp source message 2024-03-07T01:01:00.000-05:00 php-cgi[67141]: php-cgi[67141]: rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:36:45.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:36:45.000-05:00 kernel: kernel: ix3: link state changed to DOWN 2024-03-07T02:36:45.000-05:00 GAFW GAFW dhclient[32095]: ix3 link state up -> down 2024-03-07T02:36:47.000-05:00 charon[71782]: charon[71782]: 06[KNL] 162.193.210.96 disappeared from ix3 2024-03-07T02:36:47.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Failed to get IP for interface ix3 2024-03-07T02:36:47.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down 2024-03-07T02:36:49.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:36:49.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:36:53.000-05:00 ntpd[86868]: ntpd[86868]: Deleting interface #55 ix3, 162.193.210.96#123, interface stats: received=1173, sent=1257, dropped=0, active_time=127360 secs 2024-03-07T02:36:55.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:36:55.000-05:00 kernel: kernel: ix3: link state changed to UP 2024-03-07T02:36:56.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:36:57.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:36:58.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:36:59.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:01.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:05.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:09.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:37:10.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:37:11.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2 2024-03-07T02:37:13.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 4 2024-03-07T02:37:17.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 6 2024-03-07T02:37:23.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 8 2024-03-07T02:37:26.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:26.000-05:00 kernel: kernel: ix3: link state changed to DOWN 2024-03-07T02:37:26.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state up -> down 2024-03-07T02:37:31.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 15 2024-03-07T02:37:46.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10 2024-03-07T02:37:50.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:50.000-05:00 kernel: kernel: ix3: link state changed to UP 2024-03-07T02:37:50.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:50.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state down -> up 2024-03-07T02:37:52.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:56.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:56.000-05:00 kernel: kernel: ix3: link state changed to DOWN 2024-03-07T02:37:56.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:56.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10 2024-03-07T02:37:56.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state up -> down 2024-03-07T02:37:58.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:58.000-05:00 kernel: kernel: ix3: link state changed to UP 2024-03-07T02:37:58.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:58.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state down -> up 2024-03-07T02:37:59.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:00.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:01.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:03.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:06.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:06.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 7 2024-03-07T02:38:13.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 9 2024-03-07T02:38:17.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:38:18.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:38:19.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:38:20.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2 2024-03-07T02:38:22.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2 2024-03-07T02:38:24.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 3 2024-03-07T02:38:26.000-05:00 check_reload_status[460]: check_reload_status[460]: rc.newwanip starting ix3 2024-03-07T02:38:26.000-05:00 charon[71782]: charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[89711]: /sbin/route add -host 192.168.1.254 -iface ix3 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[87947]: Adding new routes to interface: ix3 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[87082]: New Routers (ix3): 192.168.1.254 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[86571]: New Broadcast Address (ix3): 192.168.1.255 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[85955]: New Subnet Mask (ix3): 255.255.255.0 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[85320]: New IP Address (ix3): 192.168.1.94 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[84415]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:27.000-05:00 php-fpm[44338]: php-fpm[44338]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3). 2024-03-07T02:38:27.000-05:00 php-fpm[44338]: php-fpm[44338]: /rc.newwanip: rc.newwanip: Info: starting on ix3. 2024-03-07T02:38:29.000-05:00 ntpd[86868]: ntpd[86868]: Listen normally on 58 ix3 192.168.1.94:123 2024-03-07T02:38:31.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Failed to get IP for interface ix3 2024-03-07T02:38:31.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down 2024-03-07T02:38:31.000-05:00 charon[71782]: charon[71782]: 01[KNL] 192.168.1.94 disappeared from ix3 2024-03-07T02:38:34.000-05:00 ntpd[86868]: ntpd[86868]: Deleting interface #58 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs 2024-03-07T02:38:34.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:34.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:35.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:35.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:36.000-05:00 check_reload_status[460]: check_reload_status[460]: rc.newwanip starting ix3 2024-03-07T02:38:36.000-05:00 charon[71782]: charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[42980]: /sbin/route add -host 192.168.1.254 -iface ix3 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[40764]: Adding new routes to interface: ix3 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[39766]: New Routers (ix3): 192.168.1.254 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[38095]: New Broadcast Address (ix3): 192.168.1.255 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[37026]: New Subnet Mask (ix3): 255.255.255.0 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[35894]: New IP Address (ix3): 192.168.1.94 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[34295]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[28646]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:37.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3). 2024-03-07T02:38:37.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.newwanip: rc.newwanip: Info: starting on ix3. 2024-03-07T02:38:39.000-05:00 ntpd[86868]: ntpd[86868]: Listen normally on 59 ix3 192.168.1.94:123 2024-03-07T02:38:41.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Failed to get IP for interface ix3 2024-03-07T02:38:41.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down 2024-03-07T02:38:41.000-05:00 charon[71782]: charon[71782]: 08[KNL] 192.168.1.94 disappeared from ix3 2024-03-07T02:38:44.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:44.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:44.000-05:00 ntpd[86868]: ntpd[86868]: Deleting interface #59 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs 2024-03-07T02:38:45.000-05:00 check_reload_status[460]: check_reload_status[460]: rc.newwanip starting ix3 2024-03-07T02:38:45.000-05:00 charon[71782]: charon[71782]: 10[KNL] 192.168.1.94 appeared on ix3 2024-03-07T02:38:45.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:45.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[25149]: /sbin/route add -host 192.168.1.254 -iface ix3 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[23218]: Adding new routes to interface: ix3 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[22332]: New Routers (ix3): 192.168.1.254 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[20657]: New Broadcast Address (ix3): 192.168.1.255 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[19212]: New Subnet Mask (ix3): 255.255.255.0 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[17977]: New IP Address (ix3): 192.168.1.94 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[15786]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[8968]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:46.000-05:00 php-fpm[19319]: php-fpm[19319]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3). 2024-03-07T02:38:46.000-05:00 php-fpm[19319]: php-fpm[19319]: /rc.newwanip: rc.newwanip: Info: starting on ix3. 2024-03-07T02:38:48.000-05:00 ntpd[86868]: ntpd[86868]: Listen normally on 60 ix3 192.168.1.94:123 2024-03-07T02:38:57.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:57.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:58.000-05:00 php-fpm[44338]: php-fpm[44338]: /rc.newwanip: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:39:00.000-05:00 GAFW GAFW dhclient[8968]: DHCPREQUEST on ix3 to 192.168.1.254 port 67 -
Its highly likely that multiple scenrios are happening that are negatively impacting.
Watchdog is no longer monitoring any serviceIts probable that something did occur that caused the firewall to be inaccessible from the LAN side and may be related to the watchdog process and restarting unbound. Suricata was never part of the watchdog daemon monitoring.
That said, i will just have to wait to see if this happens again.
I appreciate everyone chiming in on this one. -
@michmoor said in DNS/DHCP stop working suddenly:
Based on the alerts from monitoring this problem started at 12:47am ET. Lets review what i could dig up in graylog.
Ping monitors to my WAN address also fail at this time. So the WAN address was not accessible.
During this time the filterlog still shows traffic inbound to my old IP so i am inferring that traffic was still making it into the firewall. In fact traffic continues to come inbound until 2:47:58am ET where there is a dhclient process starting which of course is the DHCP call out on the WAN.
The link flaps a few times
Thats when things crap the bed and no more inbound flows
timestamp source message 2024-03-07T01:01:00.000-05:00 php-cgi[67141]: php-cgi[67141]: rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:36:45.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:36:45.000-05:00 kernel: kernel: ix3: link state changed to DOWN 2024-03-07T02:36:45.000-05:00 GAFW GAFW dhclient[32095]: ix3 link state up -> down 2024-03-07T02:36:47.000-05:00 charon[71782]: charon[71782]: 06[KNL] 162.193.210.96 disappeared from ix3 2024-03-07T02:36:47.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Failed to get IP for interface ix3 2024-03-07T02:36:47.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down 2024-03-07T02:36:49.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:36:49.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:36:53.000-05:00 ntpd[86868]: ntpd[86868]: Deleting interface #55 ix3, 162.193.210.96#123, interface stats: received=1173, sent=1257, dropped=0, active_time=127360 secs 2024-03-07T02:36:55.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:36:55.000-05:00 kernel: kernel: ix3: link state changed to UP 2024-03-07T02:36:56.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:36:57.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:36:58.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:36:59.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:01.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:05.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:09.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:37:10.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:37:11.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2 2024-03-07T02:37:13.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 4 2024-03-07T02:37:17.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 6 2024-03-07T02:37:23.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 8 2024-03-07T02:37:26.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:26.000-05:00 kernel: kernel: ix3: link state changed to DOWN 2024-03-07T02:37:26.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state up -> down 2024-03-07T02:37:31.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 15 2024-03-07T02:37:46.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10 2024-03-07T02:37:50.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:50.000-05:00 kernel: kernel: ix3: link state changed to UP 2024-03-07T02:37:50.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:50.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state down -> up 2024-03-07T02:37:52.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:56.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:56.000-05:00 kernel: kernel: ix3: link state changed to DOWN 2024-03-07T02:37:56.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:56.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10 2024-03-07T02:37:56.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state up -> down 2024-03-07T02:37:58.000-05:00 check_reload_status[460]: check_reload_status[460]: Linkup starting ix3 2024-03-07T02:37:58.000-05:00 kernel: kernel: ix3: link state changed to UP 2024-03-07T02:37:58.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:37:58.000-05:00 GAFW GAFW dhclient[14415]: ix3 link state down -> up 2024-03-07T02:37:59.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:00.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:01.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:03.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:06.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:06.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 7 2024-03-07T02:38:13.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 9 2024-03-07T02:38:17.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:38:18.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:38:19.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1 2024-03-07T02:38:20.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2 2024-03-07T02:38:22.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2 2024-03-07T02:38:24.000-05:00 GAFW GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 3 2024-03-07T02:38:26.000-05:00 check_reload_status[460]: check_reload_status[460]: rc.newwanip starting ix3 2024-03-07T02:38:26.000-05:00 charon[71782]: charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[89711]: /sbin/route add -host 192.168.1.254 -iface ix3 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[87947]: Adding new routes to interface: ix3 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[87082]: New Routers (ix3): 192.168.1.254 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[86571]: New Broadcast Address (ix3): 192.168.1.255 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[85955]: New Subnet Mask (ix3): 255.255.255.0 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[85320]: New IP Address (ix3): 192.168.1.94 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[84415]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255 2024-03-07T02:38:26.000-05:00 GAFW GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:27.000-05:00 php-fpm[44338]: php-fpm[44338]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3). 2024-03-07T02:38:27.000-05:00 php-fpm[44338]: php-fpm[44338]: /rc.newwanip: rc.newwanip: Info: starting on ix3. 2024-03-07T02:38:29.000-05:00 ntpd[86868]: ntpd[86868]: Listen normally on 58 ix3 192.168.1.94:123 2024-03-07T02:38:31.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Failed to get IP for interface ix3 2024-03-07T02:38:31.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down 2024-03-07T02:38:31.000-05:00 charon[71782]: charon[71782]: 01[KNL] 192.168.1.94 disappeared from ix3 2024-03-07T02:38:34.000-05:00 ntpd[86868]: ntpd[86868]: Deleting interface #58 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs 2024-03-07T02:38:34.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:34.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:35.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:35.000-05:00 php-fpm[78999]: php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:36.000-05:00 check_reload_status[460]: check_reload_status[460]: rc.newwanip starting ix3 2024-03-07T02:38:36.000-05:00 charon[71782]: charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[42980]: /sbin/route add -host 192.168.1.254 -iface ix3 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[40764]: Adding new routes to interface: ix3 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[39766]: New Routers (ix3): 192.168.1.254 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[38095]: New Broadcast Address (ix3): 192.168.1.255 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[37026]: New Subnet Mask (ix3): 255.255.255.0 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[35894]: New IP Address (ix3): 192.168.1.94 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[34295]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255 2024-03-07T02:38:36.000-05:00 GAFW GAFW dhclient[28646]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:37.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3). 2024-03-07T02:38:37.000-05:00 php-fpm[5633]: php-fpm[5633]: /rc.newwanip: rc.newwanip: Info: starting on ix3. 2024-03-07T02:38:39.000-05:00 ntpd[86868]: ntpd[86868]: Listen normally on 59 ix3 192.168.1.94:123 2024-03-07T02:38:41.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Failed to get IP for interface ix3 2024-03-07T02:38:41.000-05:00 miniupnpd[92455]: miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down 2024-03-07T02:38:41.000-05:00 charon[71782]: charon[71782]: 08[KNL] 192.168.1.94 disappeared from ix3 2024-03-07T02:38:44.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:44.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:44.000-05:00 ntpd[86868]: ntpd[86868]: Deleting interface #59 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs 2024-03-07T02:38:45.000-05:00 check_reload_status[460]: check_reload_status[460]: rc.newwanip starting ix3 2024-03-07T02:38:45.000-05:00 charon[71782]: charon[71782]: 10[KNL] 192.168.1.94 appeared on ix3 2024-03-07T02:38:45.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:45.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[25149]: /sbin/route add -host 192.168.1.254 -iface ix3 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[23218]: Adding new routes to interface: ix3 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[22332]: New Routers (ix3): 192.168.1.254 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[20657]: New Broadcast Address (ix3): 192.168.1.255 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[19212]: New Subnet Mask (ix3): 255.255.255.0 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[17977]: New IP Address (ix3): 192.168.1.94 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[15786]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255 2024-03-07T02:38:45.000-05:00 GAFW GAFW dhclient[8968]: DHCPREQUEST on ix3 to 255.255.255.255 port 67 2024-03-07T02:38:46.000-05:00 php-fpm[19319]: php-fpm[19319]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3). 2024-03-07T02:38:46.000-05:00 php-fpm[19319]: php-fpm[19319]: /rc.newwanip: rc.newwanip: Info: starting on ix3. 2024-03-07T02:38:48.000-05:00 ntpd[86868]: ntpd[86868]: Listen normally on 60 ix3 192.168.1.94:123 2024-03-07T02:38:57.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ). 2024-03-07T02:38:57.000-05:00 php-fpm[93379]: php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:38:58.000-05:00 php-fpm[44338]: php-fpm[44338]: /rc.newwanip: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3 2024-03-07T02:39:00.000-05:00 GAFW GAFW dhclient[8968]: DHCPREQUEST on ix3 to 192.168.1.254 port 67based on some of that timestamping and behavior, ISP maintenance seems within the realm of possibilites, too (if you haven't already confirmed or not with them).
again, it's the outage that casues the cascade of restarts/retries/timeout-counters counting/etc., etc. disabling gateway monitoring action doesn't mean that other processes/services and packages won't still 'react' to a link going down.
-
Mmm, can you try a switch between the WAN and the modem so it never actually loses link?
-
Ok try running this from the console when it stops responding:
dtrace -n 'fbt::_task_fn_rx:entry' -n 'fbt::_task_fn_tx:entry' -n 'fbt::_task_fn_iov:entry' -n 'fbt::_task_fn_admin:entry' -n 'fbt::_task_fn_rx_watchdog:entry'One of those things is probably stuck and firing off numerous times.
-
@stephenw10
Got it. Lets hope it doesnt happen
-
@stephenw10
It happened again. Got the trace for you.dtrace -n 'fbt::_task_fn_rx:entry' -n 'fbt::_task_fn_tx:entry' -n 'fbt::_task_fn_iov:entry' -n 'fbt::_task_fn_admin:entry' -n 'fbt::_task_fn_rx_watchdog:entry' dtrace: description 'fbt::_task_fn_rx:entry' matched 1 probe dtrace: description 'fbt::_task_fn_tx:entry' matched 1 probe dtrace: description 'fbt::_task_fn_iov:entry' matched 1 probe dtrace: description 'fbt::_task_fn_admin:entry' matched 1 probe dtrace: description 'fbt::_task_fn_rx_watchdog:entry' matched 1 probe CPU ID FUNCTION:NAME 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 2 25033 _task_fn_tx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 2 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 2 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 3 25031 _task_fn_rx:entry 3 25033 _task_fn_tx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 3 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry 1 25031 _task_fn_rx:entry 1 25033 _task_fn_tx:entry dtrace: processing aborted: Abort due to systemic unresponsiveness -
Hmm, I'll point that at out devs but as I understand it that's the normal output.
-
@stephenw10
Thanks. Seeing how the dtrace stopped no more than 15s into it due to the high load.."dtrace: processing aborted: Abort due to systemic unresponsiveness"Let me know what you hear back and what further traces i can run.
Watch it be something so ridiculous..edit: checked syslogs afterward. Nothing out of the ordinary was happening. Anything related to ix3 (WAN) previously did not show up here. I was not home at the time so this was just a spontaneous event without me logging in or performing any administrative action.
Firewall: NetGate,Palo Alto-VM,Juniper SRX
Routing: Juniper, Arista, Cisco
Switching: Juniper, Arista, Cisco
Wireless: Unifi, Aruba IAP
JNCIP,CCNP Enterprise -
@stephenw10
Happened again a few hours later.Only difference is that kernel{if_io_tqg_3}] is now kernel{if_io_tqg_0}]
dtrace was the same and stopped abruptly.
-
You may have an ISP line problem. Something like that happened to me about a year ago. It turned out that the old external coax line was really bad. pfSense did not handle those short and frequent outages too well. My old ASUS router was better at it. Initially I thought that my new NETGATE box was failing completely, but no. It was actually working but very, very slowly. It was taking 3-4 minutes to log into the Web UI and the Internet download speed was 2-3 Mb/s. I think I would be a good idea to check the internal Internet line first and, if that’s fine, ask the ISP to check the external line.
-
@michmoor said in DNS/DHCP stop working suddenly:
Seeing how the dtrace stopped no more than 15s into it due to the high load.."dtrace: processing aborted: Abort due to systemic unresponsiveness"
Ah sorry I completely missed that. That's obviously not normal!
Seeing the load on a different task queue is probably just coincidence.
-
@stephenw10
Could we be looking at something corrupted in the firmware? Would a re-install be a pathway here to resolving this? -
You could try that but it doesn't look like that to me. A damaged filesystem would almost certainly fail immediately.
-
@stephenw10
Incident just happened about an hour ago.
Ive checked each interface and there was no sudden sharp increase in bandwidth so this doesn't seem to be traffic-related.
I think....I may have found something very interesting and to your point traffic related....
Restricted vlan is time based. It is not available yet.... that's a lot of packets being processed? how?
Assuming the RRD graphs are accurate in terms of counting, the Restricted VLAN is the only vlan with millions of packets being recorded even compared to the WAN which doesn't register that many packets.
-
Hmm, I think that's actually milli-packets per second not millions. Compared to the in-block value it's very low on the graph.
-
Update
Moving the WAN connection from IX3 to IX1 i continued to have the firewall lock up.
Moving the internet connectivity to the igc* interface seems to have resolved the issue. pfSense has remained stable with no lockups.
IX and IGC are different nics on the 6100.
At first, this may seem like perhaps a failing ix* nic card. That's what i thought. Why else does everything work great on the igc nics?
As a test, i found another PC and connected it directly to ix3. Ran a series of speedtests daily. pfSense hasnt locked up at all.The physical connectivity had the ATT Cable modem directly attached to ix3. No patch panel, no switch. The connectivity today is the same except its directly connected to an igc interface.
The only conclusion that i can reach is that for some odd reason, there is something strange taking place on the wire between the modem and pfsense ix* interface which causes the system to freak out and lock up. The cable modem is directly connected to igc* and its normal operation.I cant explain this odd behavior. I appreciate everyones input. @stephenw10 appreciate the additional assistance out of band..
-
Hmm, yeah that's odd. You also tested with a switch on the WAN side between the IX NIC and modem?
And you are still using the IX NICs internally?
