Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    DNS/DHCP stop working suddenly

    Scheduled Pinned Locked Moved General pfSense Questions
    59 Posts 6 Posters 8.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Not sure about that. Come back to me if/when we find the root cause. 😉

      tinfoilmattT 1 Reply Last reply Reply Quote 0
      • tinfoilmattT
        tinfoilmatt @stephenw10
        last edited by

        @stephenw10 it's pretty obviously, based on OP's detailed posts and a fair bit of 'reading between the lines,' triggered gateway monitoring action following either transiest latency/loss of connecitivty or DHCP lease renewal.

        the rest of it is just system tailspin—which in fairness would, in all likelihood, eventually recover given enough time to do so. BSD and pfSense be solid like that.

        M 1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Yes, exactly I don't expect to just get hung up in the interface/network like that. You certainly can end up with a lot of script 'churn' when an interface drops out. Especially if there are other interfaces on it like VPNs etc. But it will stop after a few minutes unless the interface changes again.

          1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks @michmoor
            last edited by bmeeks

            @michmoor said in DNS/DHCP stop working suddenly:

            @stephenw10
            When @cyberconsultants mentioned suricata i did immediately go to my service watchdog and I have the following enabled.

            2a81fa14-6f7f-47e7-94b9-1fb1f9ac5b1b-image.png

            This is a set up Ive had for years now. I don't mind removing these services from monitoring but i don't see how it prevented a reboot. Ive upgraded/rebooted many times with these enabled.

            edit: I would also say that its likely something environmentally changed here but I have no idea what it could be. Short of adding a firewall rule, the config is static. Goes down for an upgrade every few months. So whatever is causing a cascade of headaches for me (love the term) its proving difficult to isolate.

            One of the issues with Service Watchdog is that the package is not super smart. It simply looks for the presence or absence of the monitored daemon, and sends it a start command if missing. Lots of things can result in normal restarts of certain daemons -- unbound being one of those daemons. Consider this scenario:

            unbound can be normally restarted by several things:

            1. If you have DHCP hostname registration enabled (most don't, but if you do) it will restart unbound each time a DHCP lease renews. But Service Watchdog is not aware of that. It will simply blindly see unbound not running at the instant it checks and then send it a restart command without knowing the service is already in the middle of restarting.

            2. A temporary issue with your WAN (packet loss, for example) might trigger a gateway alarm. If that deteriorates into a down condition, pfSense will issue a command to restart a bunch of processes including unbound. Again, Service Watchdog is ignorant of this process. It will see unbound missing from the list of running daemons when it spot checks, and thus issue a restart command. But pfSense itself is already restarting unbound. That can lead to problems.

            3. If you run pfBlockerNG, it can also issue restart commands to unbound in some update scenarios. Service Watchdog is ignorant of this process as well, and will just blindly issue a restart command when it fails to see a running unbound daemon.

            So, looking at the scenarios above you can see that it would be possible for bad things to happen if two unbound instances are trying to both start simultaneously. And whether or not this might occur would be random as it would depend on the exact timing differences between the natural restart process and when Service Watchdog sends it own independent restart signal.

            As @stephenw10 mentioned, Service Watchdog is really only intended for use when debugging a problem or working around some known issue. It is not a great solution for routine monitoring - especially monitoring of processes that can be restarted for legitimate reasons. This is why I preach to never use it with the IDS/IPS packages. Service Watchdog just does not consider if a service is legitimately restarting and thus the missing daemon instance is expected (and to be fair, it can't know this). It just says "no daemon present at this instant, so restart!".

            1 Reply Last reply Reply Quote 3
            • M
              michmoor LAYER 8 Rebel Alliance @tinfoilmatt
              last edited by michmoor

              @cyberconsultants @stephenw10

              Well lets dissect this a bit then. As far as Gateway actions are concerned its disabled. Has been for a while
              49dd2964-647c-4edd-82c6-f968eeb5605b-image.png

              So if the trigger is my WAN address releasing/renewing an address why all of a sudden?

              Based on the alerts from monitoring this problem started at 12:47am ET. Lets review what i could dig up in graylog.

              81b8d136-dcba-4d6f-9877-42e734f4556f-image.png

              Ping monitors to my WAN address also fail at this time. So the WAN address was not accessible.

              92280c32-03e6-4fa7-b7e1-2bf688cb5464-image.png

              During this time the filterlog still shows traffic inbound to my old IP so i am inferring that traffic was still making it into the firewall. In fact traffic continues to come inbound until 2:47:58am ET where there is a dhclient process starting which of course is the DHCP call out on the WAN.

              The link flaps a few times
              38ae9d8c-44fe-4c92-af52-451ce0ee018e-image.png

              Thats when things crap the bed and no more inbound flows

              timestamp	source	message
2024-03-07T01:01:00.000-05:00	php-cgi[67141]:	php-cgi[67141]: rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:36:45.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:36:45.000-05:00	kernel:	kernel: ix3: link state changed to DOWN
2024-03-07T02:36:45.000-05:00	GAFW	GAFW dhclient[32095]: ix3 link state up -> down
2024-03-07T02:36:47.000-05:00	charon[71782]:	charon[71782]: 06[KNL] 162.193.210.96 disappeared from ix3
2024-03-07T02:36:47.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Failed to get IP for interface ix3
2024-03-07T02:36:47.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down
2024-03-07T02:36:49.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:36:49.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:36:53.000-05:00	ntpd[86868]:	ntpd[86868]: Deleting interface #55 ix3, 162.193.210.96#123, interface stats: received=1173, sent=1257, dropped=0, active_time=127360 secs
2024-03-07T02:36:55.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:36:55.000-05:00	kernel:	kernel: ix3: link state changed to UP
2024-03-07T02:36:56.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:36:57.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:36:58.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:36:59.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:01.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:05.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:09.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:37:10.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:37:11.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2
2024-03-07T02:37:13.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 4
2024-03-07T02:37:17.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 6
2024-03-07T02:37:23.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 8
2024-03-07T02:37:26.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:26.000-05:00	kernel:	kernel: ix3: link state changed to DOWN
2024-03-07T02:37:26.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state up -> down
2024-03-07T02:37:31.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 15
2024-03-07T02:37:46.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10
2024-03-07T02:37:50.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:50.000-05:00	kernel:	kernel: ix3: link state changed to UP
2024-03-07T02:37:50.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:50.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state down -> up
2024-03-07T02:37:52.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:56.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:56.000-05:00	kernel:	kernel: ix3: link state changed to DOWN
2024-03-07T02:37:56.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:56.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10
2024-03-07T02:37:56.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state up -> down
2024-03-07T02:37:58.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:58.000-05:00	kernel:	kernel: ix3: link state changed to UP
2024-03-07T02:37:58.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:58.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state down -> up
2024-03-07T02:37:59.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:00.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:01.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:03.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:06.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:06.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 7
2024-03-07T02:38:13.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 9
2024-03-07T02:38:17.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:38:18.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:38:19.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:38:20.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2
2024-03-07T02:38:22.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2
2024-03-07T02:38:24.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 3
2024-03-07T02:38:26.000-05:00	check_reload_status[460]:	check_reload_status[460]: rc.newwanip starting ix3
2024-03-07T02:38:26.000-05:00	charon[71782]:	charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[89711]: /sbin/route add -host 192.168.1.254 -iface ix3
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[87947]: Adding new routes to interface: ix3
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[87082]: New Routers (ix3): 192.168.1.254
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[86571]: New Broadcast Address (ix3): 192.168.1.255
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[85955]: New Subnet Mask (ix3): 255.255.255.0
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[85320]: New IP Address (ix3): 192.168.1.94
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[84415]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:27.000-05:00	php-fpm[44338]:	php-fpm[44338]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3).
2024-03-07T02:38:27.000-05:00	php-fpm[44338]:	php-fpm[44338]: /rc.newwanip: rc.newwanip: Info: starting on ix3.
2024-03-07T02:38:29.000-05:00	ntpd[86868]:	ntpd[86868]: Listen normally on 58 ix3 192.168.1.94:123
2024-03-07T02:38:31.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Failed to get IP for interface ix3
2024-03-07T02:38:31.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down
2024-03-07T02:38:31.000-05:00	charon[71782]:	charon[71782]: 01[KNL] 192.168.1.94 disappeared from ix3
2024-03-07T02:38:34.000-05:00	ntpd[86868]:	ntpd[86868]: Deleting interface #58 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs
2024-03-07T02:38:34.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:34.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:35.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:35.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:36.000-05:00	check_reload_status[460]:	check_reload_status[460]: rc.newwanip starting ix3
2024-03-07T02:38:36.000-05:00	charon[71782]:	charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[42980]: /sbin/route add -host 192.168.1.254 -iface ix3
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[40764]: Adding new routes to interface: ix3
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[39766]: New Routers (ix3): 192.168.1.254
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[38095]: New Broadcast Address (ix3): 192.168.1.255
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[37026]: New Subnet Mask (ix3): 255.255.255.0
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[35894]: New IP Address (ix3): 192.168.1.94
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[34295]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[28646]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:37.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3).
2024-03-07T02:38:37.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.newwanip: rc.newwanip: Info: starting on ix3.
2024-03-07T02:38:39.000-05:00	ntpd[86868]:	ntpd[86868]: Listen normally on 59 ix3 192.168.1.94:123
2024-03-07T02:38:41.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Failed to get IP for interface ix3
2024-03-07T02:38:41.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down
2024-03-07T02:38:41.000-05:00	charon[71782]:	charon[71782]: 08[KNL] 192.168.1.94 disappeared from ix3
2024-03-07T02:38:44.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:44.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:44.000-05:00	ntpd[86868]:	ntpd[86868]: Deleting interface #59 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs
2024-03-07T02:38:45.000-05:00	check_reload_status[460]:	check_reload_status[460]: rc.newwanip starting ix3
2024-03-07T02:38:45.000-05:00	charon[71782]:	charon[71782]: 10[KNL] 192.168.1.94 appeared on ix3
2024-03-07T02:38:45.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:45.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[25149]: /sbin/route add -host 192.168.1.254 -iface ix3
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[23218]: Adding new routes to interface: ix3
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[22332]: New Routers (ix3): 192.168.1.254
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[20657]: New Broadcast Address (ix3): 192.168.1.255
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[19212]: New Subnet Mask (ix3): 255.255.255.0
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[17977]: New IP Address (ix3): 192.168.1.94
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[15786]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[8968]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:46.000-05:00	php-fpm[19319]:	php-fpm[19319]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3).
2024-03-07T02:38:46.000-05:00	php-fpm[19319]:	php-fpm[19319]: /rc.newwanip: rc.newwanip: Info: starting on ix3.
2024-03-07T02:38:48.000-05:00	ntpd[86868]:	ntpd[86868]: Listen normally on 60 ix3 192.168.1.94:123
2024-03-07T02:38:57.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:57.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:58.000-05:00	php-fpm[44338]:	php-fpm[44338]: /rc.newwanip: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:39:00.000-05:00	GAFW	GAFW dhclient[8968]: DHCPREQUEST on ix3 to 192.168.1.254 port 67

              Firewall: NetGate,Palo Alto-VM,Juniper SRX
              Routing: Juniper, Arista, Cisco
              Switching: Juniper, Arista, Cisco
              Wireless: Unifi, Aruba IAP
              JNCIP,CCNP Enterprise

              tinfoilmattT 1 Reply Last reply Reply Quote 0
              • M
                michmoor LAYER 8 Rebel Alliance
                last edited by

                Its highly likely that multiple scenrios are happening that are negatively impacting.
                Watchdog is no longer monitoring any service

                Its probable that something did occur that caused the firewall to be inaccessible from the LAN side and may be related to the watchdog process and restarting unbound. Suricata was never part of the watchdog daemon monitoring.

                That said, i will just have to wait to see if this happens again.
                I appreciate everyone chiming in on this one.

                Firewall: NetGate,Palo Alto-VM,Juniper SRX
                Routing: Juniper, Arista, Cisco
                Switching: Juniper, Arista, Cisco
                Wireless: Unifi, Aruba IAP
                JNCIP,CCNP Enterprise

                1 Reply Last reply Reply Quote 0
                • tinfoilmattT
                  tinfoilmatt @michmoor
                  last edited by

                  @michmoor said in DNS/DHCP stop working suddenly:

                  Based on the alerts from monitoring this problem started at 12:47am ET. Lets review what i could dig up in graylog.

                  81b8d136-dcba-4d6f-9877-42e734f4556f-image.png

                  Ping monitors to my WAN address also fail at this time. So the WAN address was not accessible.

                  92280c32-03e6-4fa7-b7e1-2bf688cb5464-image.png

                  During this time the filterlog still shows traffic inbound to my old IP so i am inferring that traffic was still making it into the firewall. In fact traffic continues to come inbound until 2:47:58am ET where there is a dhclient process starting which of course is the DHCP call out on the WAN.

                  The link flaps a few times
                  38ae9d8c-44fe-4c92-af52-451ce0ee018e-image.png

                  Thats when things crap the bed and no more inbound flows

                  timestamp	source	message
2024-03-07T01:01:00.000-05:00	php-cgi[67141]:	php-cgi[67141]: rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:36:45.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:36:45.000-05:00	kernel:	kernel: ix3: link state changed to DOWN
2024-03-07T02:36:45.000-05:00	GAFW	GAFW dhclient[32095]: ix3 link state up -> down
2024-03-07T02:36:47.000-05:00	charon[71782]:	charon[71782]: 06[KNL] 162.193.210.96 disappeared from ix3
2024-03-07T02:36:47.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Failed to get IP for interface ix3
2024-03-07T02:36:47.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down
2024-03-07T02:36:49.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:36:49.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:36:53.000-05:00	ntpd[86868]:	ntpd[86868]: Deleting interface #55 ix3, 162.193.210.96#123, interface stats: received=1173, sent=1257, dropped=0, active_time=127360 secs
2024-03-07T02:36:55.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:36:55.000-05:00	kernel:	kernel: ix3: link state changed to UP
2024-03-07T02:36:56.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:36:57.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:36:58.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:36:59.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:01.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:05.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:09.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:37:10.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:37:11.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2
2024-03-07T02:37:13.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 4
2024-03-07T02:37:17.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 6
2024-03-07T02:37:23.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 8
2024-03-07T02:37:26.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:26.000-05:00	kernel:	kernel: ix3: link state changed to DOWN
2024-03-07T02:37:26.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state up -> down
2024-03-07T02:37:31.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 15
2024-03-07T02:37:46.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10
2024-03-07T02:37:50.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:50.000-05:00	kernel:	kernel: ix3: link state changed to UP
2024-03-07T02:37:50.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:50.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state down -> up
2024-03-07T02:37:52.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:56.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:56.000-05:00	kernel:	kernel: ix3: link state changed to DOWN
2024-03-07T02:37:56.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:56.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 10
2024-03-07T02:37:56.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state up -> down
2024-03-07T02:37:58.000-05:00	check_reload_status[460]:	check_reload_status[460]: Linkup starting ix3
2024-03-07T02:37:58.000-05:00	kernel:	kernel: ix3: link state changed to UP
2024-03-07T02:37:58.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:37:58.000-05:00	GAFW	GAFW dhclient[14415]: ix3 link state down -> up
2024-03-07T02:37:59.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:00.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:01.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:03.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:06.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:06.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 7
2024-03-07T02:38:13.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 9
2024-03-07T02:38:17.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:38:18.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:38:19.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 1
2024-03-07T02:38:20.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2
2024-03-07T02:38:22.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 2
2024-03-07T02:38:24.000-05:00	GAFW	GAFW dhclient[14415]: DHCPDISCOVER on ix3 to 255.255.255.255 port 67 interval 3
2024-03-07T02:38:26.000-05:00	check_reload_status[460]:	check_reload_status[460]: rc.newwanip starting ix3
2024-03-07T02:38:26.000-05:00	charon[71782]:	charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[89711]: /sbin/route add -host 192.168.1.254 -iface ix3
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[87947]: Adding new routes to interface: ix3
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[87082]: New Routers (ix3): 192.168.1.254
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[86571]: New Broadcast Address (ix3): 192.168.1.255
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[85955]: New Subnet Mask (ix3): 255.255.255.0
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[85320]: New IP Address (ix3): 192.168.1.94
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[84415]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255
2024-03-07T02:38:26.000-05:00	GAFW	GAFW dhclient[14415]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:27.000-05:00	php-fpm[44338]:	php-fpm[44338]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3).
2024-03-07T02:38:27.000-05:00	php-fpm[44338]:	php-fpm[44338]: /rc.newwanip: rc.newwanip: Info: starting on ix3.
2024-03-07T02:38:29.000-05:00	ntpd[86868]:	ntpd[86868]: Listen normally on 58 ix3 192.168.1.94:123
2024-03-07T02:38:31.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Failed to get IP for interface ix3
2024-03-07T02:38:31.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down
2024-03-07T02:38:31.000-05:00	charon[71782]:	charon[71782]: 01[KNL] 192.168.1.94 disappeared from ix3
2024-03-07T02:38:34.000-05:00	ntpd[86868]:	ntpd[86868]: Deleting interface #58 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs
2024-03-07T02:38:34.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:34.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:35.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:35.000-05:00	php-fpm[78999]:	php-fpm[78999]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:36.000-05:00	check_reload_status[460]:	check_reload_status[460]: rc.newwanip starting ix3
2024-03-07T02:38:36.000-05:00	charon[71782]:	charon[71782]: 01[KNL] 192.168.1.94 appeared on ix3
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[42980]: /sbin/route add -host 192.168.1.254 -iface ix3
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[40764]: Adding new routes to interface: ix3
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[39766]: New Routers (ix3): 192.168.1.254
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[38095]: New Broadcast Address (ix3): 192.168.1.255
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[37026]: New Subnet Mask (ix3): 255.255.255.0
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[35894]: New IP Address (ix3): 192.168.1.94
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[34295]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255
2024-03-07T02:38:36.000-05:00	GAFW	GAFW dhclient[28646]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:37.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3).
2024-03-07T02:38:37.000-05:00	php-fpm[5633]:	php-fpm[5633]: /rc.newwanip: rc.newwanip: Info: starting on ix3.
2024-03-07T02:38:39.000-05:00	ntpd[86868]:	ntpd[86868]: Listen normally on 59 ix3 192.168.1.94:123
2024-03-07T02:38:41.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Failed to get IP for interface ix3
2024-03-07T02:38:41.000-05:00	miniupnpd[92455]:	miniupnpd[92455]: Cannot get IP address for ext interface ix3. Network is down
2024-03-07T02:38:41.000-05:00	charon[71782]:	charon[71782]: 08[KNL] 192.168.1.94 disappeared from ix3
2024-03-07T02:38:44.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:44.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:44.000-05:00	ntpd[86868]:	ntpd[86868]: Deleting interface #59 ix3, 192.168.1.94#123, interface stats: received=0, sent=0, dropped=4, active_time=5 secs
2024-03-07T02:38:45.000-05:00	check_reload_status[460]:	check_reload_status[460]: rc.newwanip starting ix3
2024-03-07T02:38:45.000-05:00	charon[71782]:	charon[71782]: 10[KNL] 192.168.1.94 appeared on ix3
2024-03-07T02:38:45.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:45.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[25149]: /sbin/route add -host 192.168.1.254 -iface ix3
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[23218]: Adding new routes to interface: ix3
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[22332]: New Routers (ix3): 192.168.1.254
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[20657]: New Broadcast Address (ix3): 192.168.1.255
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[19212]: New Subnet Mask (ix3): 255.255.255.0
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[17977]: New IP Address (ix3): 192.168.1.94
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[15786]: ifconfig ix3 inet 192.168.1.94 netmask 255.255.255.0 broadcast 192.168.1.255
2024-03-07T02:38:45.000-05:00	GAFW	GAFW dhclient[8968]: DHCPREQUEST on ix3 to 255.255.255.255 port 67
2024-03-07T02:38:46.000-05:00	php-fpm[19319]:	php-fpm[19319]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.1.94) (interface: WAN[wan]) (real interface: ix3).
2024-03-07T02:38:46.000-05:00	php-fpm[19319]:	php-fpm[19319]: /rc.newwanip: rc.newwanip: Info: starting on ix3.
2024-03-07T02:38:48.000-05:00	ntpd[86868]:	ntpd[86868]: Listen normally on 60 ix3 192.168.1.94:123
2024-03-07T02:38:57.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com) There was an error trying to determine the public IP for interface - wan (ix3 ).
2024-03-07T02:38:57.000-05:00	php-fpm[93379]:	php-fpm[93379]: /rc.dyndns.update: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:38:58.000-05:00	php-fpm[44338]:	php-fpm[44338]: /rc.newwanip: Dynamic DNS (vpn.networkingtitan.com): running get_failover_interface for wan. found ix3
2024-03-07T02:39:00.000-05:00	GAFW	GAFW dhclient[8968]: DHCPREQUEST on ix3 to 192.168.1.254 port 67

                  based on some of that timestamping and behavior, ISP maintenance seems within the realm of possibilites, too (if you haven't already confirmed or not with them).

                  again, it's the outage that casues the cascade of restarts/retries/timeout-counters counting/etc., etc. disabling gateway monitoring action doesn't mean that other processes/services and packages won't still 'react' to a link going down.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Mmm, can you try a switch between the WAN and the modem so it never actually loses link?

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Ok try running this from the console when it stops responding:

                      dtrace -n 'fbt::_task_fn_rx:entry' -n 'fbt::_task_fn_tx:entry' -n 'fbt::_task_fn_iov:entry' -n 'fbt::_task_fn_admin:entry' -n 'fbt::_task_fn_rx_watchdog:entry'

                      One of those things is probably stuck and firing off numerous times.

                      M 1 Reply Last reply Reply Quote 1
                      • M
                        michmoor LAYER 8 Rebel Alliance @stephenw10
                        last edited by

                        @stephenw10
                        Got it. Lets hope it doesnt happen 😧

                        Firewall: NetGate,Palo Alto-VM,Juniper SRX
                        Routing: Juniper, Arista, Cisco
                        Switching: Juniper, Arista, Cisco
                        Wireless: Unifi, Aruba IAP
                        JNCIP,CCNP Enterprise

                        M 1 Reply Last reply Reply Quote 0
                        • M
                          michmoor LAYER 8 Rebel Alliance @michmoor
                          last edited by

                          @stephenw10
                          It happened again. Got the trace for you.

                           dtrace -n 'fbt::_task_fn_rx:entry' -n 'fbt::_task_fn_tx:entry' -n 'fbt::_task_fn_iov:entry' -n 'fbt::_task_fn_admin:entry' -n 'fbt::_task_fn_rx_watchdog:entry'
dtrace: description 'fbt::_task_fn_rx:entry' matched 1 probe
dtrace: description 'fbt::_task_fn_tx:entry' matched 1 probe
dtrace: description 'fbt::_task_fn_iov:entry' matched 1 probe
dtrace: description 'fbt::_task_fn_admin:entry' matched 1 probe
dtrace: description 'fbt::_task_fn_rx_watchdog:entry' matched 1 probe
CPU     ID                    FUNCTION:NAME
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  2  25033                _task_fn_tx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  2  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  2  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  3  25031                _task_fn_rx:entry
  3  25033                _task_fn_tx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  3  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
  1  25031                _task_fn_rx:entry
  1  25033                _task_fn_tx:entry
dtrace: processing aborted: Abort due to systemic unresponsiveness

                          Firewall: NetGate,Palo Alto-VM,Juniper SRX
                          Routing: Juniper, Arista, Cisco
                          Switching: Juniper, Arista, Cisco
                          Wireless: Unifi, Aruba IAP
                          JNCIP,CCNP Enterprise

                          M 1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Hmm, I'll point that at out devs but as I understand it that's the normal output. 🤔

                            1 Reply Last reply Reply Quote 0
                            • M
                              michmoor LAYER 8 Rebel Alliance @michmoor
                              last edited by michmoor

                              @stephenw10
                              Thanks. Seeing how the dtrace stopped no more than 15s into it due to the high load.."dtrace: processing aborted: Abort due to systemic unresponsiveness"

                              Let me know what you hear back and what further traces i can run.
                              Watch it be something so ridiculous..

                              edit: checked syslogs afterward. Nothing out of the ordinary was happening. Anything related to ix3 (WAN) previously did not show up here. I was not home at the time so this was just a spontaneous event without me logging in or performing any administrative action.

                              Firewall: NetGate,Palo Alto-VM,Juniper SRX
                              Routing: Juniper, Arista, Cisco
                              Switching: Juniper, Arista, Cisco
                              Wireless: Unifi, Aruba IAP
                              JNCIP,CCNP Enterprise

                              stephenw10S 1 Reply Last reply Reply Quote 0
                              • M
                                michmoor LAYER 8 Rebel Alliance
                                last edited by

                                @stephenw10
                                Happened again a few hours later.

                                Only difference is that kernel{if_io_tqg_3}] is now kernel{if_io_tqg_0}]

                                dtrace was the same and stopped abruptly.

                                Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                Routing: Juniper, Arista, Cisco
                                Switching: Juniper, Arista, Cisco
                                Wireless: Unifi, Aruba IAP
                                JNCIP,CCNP Enterprise

                                ? 1 Reply Last reply Reply Quote 0
                                • ?
                                  A Former User @michmoor
                                  last edited by

                                  @michmoor

                                  You may have an ISP line problem. Something like that happened to me about a year ago. It turned out that the old external coax line was really bad. pfSense did not handle those short and frequent outages too well. My old ASUS router was better at it. Initially I thought that my new NETGATE box was failing completely, but no. It was actually working but very, very slowly. It was taking 3-4 minutes to log into the Web UI and the Internet download speed was 2-3 Mb/s. I think I would be a good idea to check the internal Internet line first and, if that’s fine, ask the ISP to check the external line.

                                  1 Reply Last reply Reply Quote 2
                                  • stephenw10S
                                    stephenw10 Netgate Administrator @michmoor
                                    last edited by

                                    @michmoor said in DNS/DHCP stop working suddenly:

                                    Seeing how the dtrace stopped no more than 15s into it due to the high load.."dtrace: processing aborted: Abort due to systemic unresponsiveness"

                                    Ah sorry I completely missed that. That's obviously not normal!

                                    Seeing the load on a different task queue is probably just coincidence.

                                    M 1 Reply Last reply Reply Quote 0
                                    • M
                                      michmoor LAYER 8 Rebel Alliance @stephenw10
                                      last edited by

                                      @stephenw10
                                      Could we be looking at something corrupted in the firmware? Would a re-install be a pathway here to resolving this?

                                      Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                      Routing: Juniper, Arista, Cisco
                                      Switching: Juniper, Arista, Cisco
                                      Wireless: Unifi, Aruba IAP
                                      JNCIP,CCNP Enterprise

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        You could try that but it doesn't look like that to me. A damaged filesystem would almost certainly fail immediately.

                                        M 1 Reply Last reply Reply Quote 1
                                        • M
                                          michmoor LAYER 8 Rebel Alliance @stephenw10
                                          last edited by michmoor

                                          @stephenw10
                                          Incident just happened about an hour ago.
                                          Ive checked each interface and there was no sudden sharp increase in bandwidth so this doesn't seem to be traffic-related.

                                          564b9d93-454b-4c8f-84a1-9f89d9d8be55-image.png

                                          I think....I may have found something very interesting and to your point traffic related....
                                          Restricted vlan is time based. It is not available yet.... that's a lot of packets being processed? how?

                                          88cb4a25-a9d2-4c99-a597-647d6e121efc-image.png

                                          Assuming the RRD graphs are accurate in terms of counting, the Restricted VLAN is the only vlan with millions of packets being recorded even compared to the WAN which doesn't register that many packets.

                                          7d3186d9-54fb-403e-bd57-881787963d8b-image.png

                                          Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                          Routing: Juniper, Arista, Cisco
                                          Switching: Juniper, Arista, Cisco
                                          Wireless: Unifi, Aruba IAP
                                          JNCIP,CCNP Enterprise

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            Hmm, I think that's actually milli-packets per second not millions. Compared to the in-block value it's very low on the graph.

                                            M 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.