Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense WAN dhcp client exiting (error)

    General pfSense Questions
    4
    68
    6.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      Nosense 0
      last edited by

      In the between I uninstalled Suricata, but I also did nothing.

      1 Reply Last reply Reply Quote 0
      • GertjanG
        Gertjan @Nosense 0
        last edited by

        is this what I think it is :

        @Nosense-0 said in pfSense WAN dhcp client exiting (error):

        servicewatchdog_cron

        ?
        Probably not related, but if you're using "Service_Watchdog" : remove it.
        This package is useful if you do debugging and other bleeding edge situations.
        For the other 99,99999 % pfSense users, like you and me, it inflict stress, and incomprehension, and a bad connection.

        When I see a line like this

        @Nosense-0 said in pfSense WAN dhcp client exiting (error):

        Mar 29 11:02:07 php-cgi[2020]: servicewatchdog_cron.php: dpinger: timeout while retrieving status for gateway WAN_DHCP

        and all the other lines, like :
        Interfaces come up.
        VLANs are created
        WAN IP is set up.
        Routes are applied.
        Tests are executes to see if you have "IPV6".
        DynDNS checks are executed,
        Etc. All the usual stuff.
        But also , you have some really heavy weights on your system :
        pfBlockerNG DNSBL,
        vnstat,
        suriacata,
        bandwith,
        ...

        What happens is : the system needs more time to "settle in".
        But "servicewatchdog" said it "enough" : it detects that there is no WAN gateway, and it pulls the plug on the WAN, and the whole restart story is starts again ....

        My advise : lower the load on the system == the number of events that can happen when "WAN" is created.
        Start with a clean and lean pfSense, ditch all the gadgets.
        Now, you're fine.
        Then add a 'thing', one by one.
        As soon as the system shows issues, go one step back, and you'll be good.

        No "help me" PM's please. Use the forum, the community will thank you.
        Edit : and where are the logs ??

        1 Reply Last reply Reply Quote 0
        • N
          Nosense 0
          last edited by

          Since we have intermediary disorders from our ISP, we absolutely need the "Service_Watchdog", we cannot do without that.

          The Pfense has been running so stable in this configuration for 4 years (with up to 8 family members) and we are enthusiastic about it!

          So it is our first "problem" with the Pfense and therefore we won't change that (unless we have to), but I temporarily switch off the watchdog and look at what happens.

          Thanks for the support!

          GertjanG 1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by stephenw10

            What services do you have selected in the Watchdog?

            What happens if you don't have it enabled? What fails to start?

            With Suricata in non-blocking mode it's not using netgraph or reconnecting anything. That's not the issue.

            The only thing logged that looks even vaguely like it might affect the link state is:

            Mar 29 11:00:43 kernel: em0: promiscuous mode disabled
Mar 29 11:00:43 kernel: re0.10: promiscuous mode disabled
Mar 29 11:00:43 kernel: re0: promiscuous mode disabled
Mar 29 11:00:43 kernel: re0.20: promiscuous mode disabled
Mar 29 11:00:43 kernel: re0: promiscuous mode enabled
Mar 29 11:00:43 kernel: em0: promiscuous mode enabled
Mar 29 11:00:43 kernel: re0.10: promiscuous mode enabled
Mar 29 11:00:43 kernel: re0.20: promiscuous mode enabled

            Which is probably arpwatch starting. Can you test with arpwatch disabled?

            1 Reply Last reply Reply Quote 0
            • N
              Nosense 0
              last edited by

              -All services removed from the watchdog
              -Suricata uninstalled
              -Pfblockerng stopped
              -Bandwidthd stopped

              Pfsense reboot
              Switched to DHCP, error still exists, new log:

              Mar 29 16:16:29 php-fpm[394]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:16:27 vnstatd[97122]: Monitoring (9): re1 (1000 Mbit) re0.20 (1000 Mbit) re0.10 (1000 Mbit) re0 (1000 Mbit) pfsync0 (1000 Mbit) pflog0 (1000 Mbit) enc0 (1000 Mbit) em1 (1000 Mbit) em0 (100 Mbit)
              Mar 29 16:16:27 vnstatd[99103]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
              Mar 29 16:16:27 vnstatd[97122]: Data retention: 48 5MinuteHours, 4 HourlyDays, 62 DailyDays, 25 MonthlyMonths, -1 YearlyYears, 20 TopDayEntries
              Mar 29 16:16:27 vnstatd[97122]: vnStat daemon 2.11 (pid:97122 uid:0 gid:0, SQLite 3.43.1)
              Mar 29 16:16:27 lighttpd_pfb[98864]: [pfBlockerNG] DNSBL Webserver started
              Mar 29 16:16:27 lighttpd_pfb[97703]: [pfBlockerNG] DNSBL Webserver stopped
              Mar 29 16:16:27 vnstatd[21916]: SIGTERM received, exiting.
              Mar 29 16:16:26 php-fpm[18742]: /widgets/widgets/gateways.widget.php: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:16:25 php-fpm[394]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:16:25 php-fpm[7795]: /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
              Mar 29 16:16:25 php-fpm[7795]: /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:16:25 php-fpm[7795]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:16:21 php-fpm[7795]: /rc.newwanip: Removing static route for monitor xxx.xxx.xxx.xxx and adding a new route through 192.168.xxx.xxx
              Mar 29 16:16:21 php-fpm[7795]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.xxx.xxx) (interface: WAN[wan]) (real interface: em1).
              Mar 29 16:16:21 php-fpm[7795]: /rc.newwanip: rc.newwanip: Info: starting on em1.
              Mar 29 16:16:20 check_reload_status[433]: Restarting IPsec tunnels
              Mar 29 16:16:20 php-fpm[394]: /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:16:20 check_reload_status[433]: rc.newwanip starting em1
              Mar 29 16:16:19 arpwatch[58965]: listening on re0.20
              Mar 29 16:16:19 arpwatch[58879]: listening on re0.10
              Mar 29 16:16:19 arpwatch[58468]: listening on em0
              Mar 29 16:16:19 arpwatch[58215]: listening on re0
              Mar 29 16:16:19 arpwatch[57777]: listening on re1
              Mar 29 16:16:17 vnstatd[54280]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
              Mar 29 16:16:17 check_reload_status[433]: Linkup starting em1
              Mar 29 16:16:17 kernel: em1: link state changed to UP
              Mar 29 16:16:17 kernel: em1: promiscuous mode enabled
              Mar 29 16:16:17 kernel: re0.20: promiscuous mode enabled
              Mar 29 16:16:17 kernel: re0.10: promiscuous mode enabled
              Mar 29 16:16:17 kernel: em0: promiscuous mode enabled
              Mar 29 16:16:17 kernel: re0: promiscuous mode enabled
              Mar 29 16:16:17 kernel: re1: promiscuous mode enabled
              Mar 29 16:16:17 kernel: em1: promiscuous mode disabled
              Mar 29 16:16:17 kernel: re0.20: promiscuous mode disabled
              Mar 29 16:16:17 kernel: re0: promiscuous mode disabled
              Mar 29 16:16:17 kernel: re0.10: promiscuous mode disabled
              Mar 29 16:16:17 kernel: em0: promiscuous mode disabled
              Mar 29 16:16:17 kernel: re1: promiscuous mode disabled
              Mar 29 16:16:17 check_reload_status[433]: Syncing firewall
              Mar 29 16:16:17 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
              Mar 29 16:16:17 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
              Mar 29 16:16:17 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
              Mar 29 16:16:17 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
              Mar 29 16:16:16 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
              Mar 29 16:16:16 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
              Mar 29 16:16:16 check_reload_status[433]: Syncing firewall
              Mar 29 16:16:16 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
              Mar 29 16:16:16 php-fpm[5949]: /rc.start_packages: The command '/usr/local/etc/rc.d/bandwidthd.sh stop' returned exit code '1', the output was 'No matching processes were found'
              Mar 29 16:16:16 check_reload_status[433]: Reloading filter
              Mar 29 16:16:16 check_reload_status[433]: Syncing firewall
              Mar 29 16:16:16 php-fpm[5949]: /rc.start_packages: Configuration Change: (system): pfBlockerNG: saving DNSBL changes
              Mar 29 16:16:15 php-fpm[5949]: /rc.start_packages: Restarting/Starting all packages.
              Mar 29 16:16:14 kernel: em1: link state changed to DOWN
              Mar 29 16:16:14 check_reload_status[433]: Linkup starting em1
              Mar 29 16:16:14 check_reload_status[433]: Reloading filter
              Mar 29 16:16:14 check_reload_status[433]: Starting packages
              Mar 29 16:16:14 php-fpm[47031]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 192.168.xxx.xxx - Restarting packages.
              Mar 29 16:16:14 php-fpm[394]: /rc.linkup: HOTPLUG: Configuring interface wan
              Mar 29 16:16:14 php-fpm[394]: /rc.linkup: DEVD Ethernet attached event for wan
              Mar 29 16:16:14 php-fpm[394]: /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
              Mar 29 16:16:14 check_reload_status[433]: Reloading filter
              Mar 29 16:16:12 php-fpm[47031]: /rc.newwanip: Creating rrd update script
              Mar 29 16:16:12 php-fpm[47031]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
              Mar 29 16:16:11 php-fpm[393]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:16:10 php-fpm[68905]: /rc.filter_configure_sync: dpinger: No dpinger session running for gateway WAN_DHCP
              Mar 29 16:16:10 check_reload_status[433]: Reloading filter
              Mar 29 16:16:10 check_reload_status[433]: Restarting OpenVPN tunnels/interfaces
              Mar 29 16:16:10 check_reload_status[433]: Restarting IPsec tunnels
              Mar 29 16:16:10 check_reload_status[433]: updating dyndns WAN_DHCP
              Mar 29 16:16:10 rc.gateway_alarm[84752]: >>> Gateway alarm: WAN_DHCP (Addr:192.168.xxx.xxx Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
              Mar 29 16:16:09 php-fpm[7795]: /rc.linkup: DEVD Ethernet detached event for wan
              Mar 29 16:16:09 php-fpm[7795]: /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
              Mar 29 16:16:09 check_reload_status[433]: Reloading filter
              Mar 29 16:16:09 check_reload_status[433]: updating dyndns wan
              Mar 29 16:16:08 php-fpm[47031]: /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1711725368] unbound[57484:0] error: bind: address already in use [1711725368] unbound[57484:0] fatal error: could not open ports'
              Mar 29 16:16:05 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:16:04 php-fpm[18742]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:16:02 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:16:01 php-fpm[18742]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:59 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:58 vnstatd[21916]: Monitoring (9): re1 (1000 Mbit) re0.20 (1000 Mbit) re0.10 (1000 Mbit) re0 (1000 Mbit) pfsync0 (1000 Mbit) pflog0 (1000 Mbit) enc0 (1000 Mbit) em1 (1000 Mbit) em0 (100 Mbit)
              Mar 29 16:15:58 vnstatd[21916]: Data retention: 48 5MinuteHours, 4 HourlyDays, 62 DailyDays, 25 MonthlyMonths, -1 YearlyYears, 20 TopDayEntries
              Mar 29 16:15:58 vnstatd[21916]: vnStat daemon 2.11 (pid:21916 uid:0 gid:0, SQLite 3.43.1)
              Mar 29 16:15:58 vnstatd[29163]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
              Mar 29 16:15:58 php-fpm[18742]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:58 lighttpd_pfb[19472]: [pfBlockerNG] DNSBL Webserver started
              Mar 29 16:15:58 lighttpd_pfb[17749]: [pfBlockerNG] DNSBL Webserver stopped
              Mar 29 16:15:57 vnstatd[99232]: SIGTERM received, exiting.
              Mar 29 16:15:55 php-fpm[393]: /widgets/widgets/gateways.widget.php: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:54 php-fpm[18742]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:54 php-fpm[47031]: /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
              Mar 29 16:15:54 php-fpm[47031]: /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:15:54 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:51 php-fpm[47031]: /rc.newwanip: Removing static route for monitor xxx.xxx.xxx.xxx and adding a new route through 192.168.xxx.xxx
              Mar 29 16:15:50 php-fpm[47031]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.xxx.xxx) (interface: WAN[wan]) (real interface: em1).
              Mar 29 16:15:50 php-fpm[47031]: /rc.newwanip: rc.newwanip: Info: starting on em1.
              Mar 29 16:15:50 arpwatch[71913]: listening on re0.20
              Mar 29 16:15:50 arpwatch[71545]: listening on re0.10
              Mar 29 16:15:49 arpwatch[71233]: listening on em0
              Mar 29 16:15:49 arpwatch[71028]: listening on re0
              Mar 29 16:15:49 arpwatch[69563]: listening on re1
              Mar 29 16:15:49 check_reload_status[433]: Restarting IPsec tunnels
              Mar 29 16:15:49 php-fpm[18742]: /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:15:49 check_reload_status[433]: rc.newwanip starting em1
              Mar 29 16:15:48 kernel: em1: link state changed to UP
              Mar 29 16:15:48 check_reload_status[433]: Linkup starting em1
              Mar 29 16:15:47 vnstatd[40633]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
              Mar 29 16:15:47 kernel: em1: promiscuous mode enabled
              Mar 29 16:15:47 kernel: re0.20: promiscuous mode enabled
              Mar 29 16:15:47 kernel: re0.10: promiscuous mode enabled
              Mar 29 16:15:47 kernel: em0: promiscuous mode enabled
              Mar 29 16:15:47 kernel: re0: promiscuous mode enabled
              Mar 29 16:15:47 kernel: re1: promiscuous mode enabled
              Mar 29 16:15:47 kernel: em1: promiscuous mode disabled
              Mar 29 16:15:47 kernel: re0.20: promiscuous mode disabled
              Mar 29 16:15:47 kernel: re0: promiscuous mode disabled
              Mar 29 16:15:47 kernel: re0.10: promiscuous mode disabled
              Mar 29 16:15:47 kernel: em0: promiscuous mode disabled
              Mar 29 16:15:47 kernel: re1: promiscuous mode disabled
              Mar 29 16:15:47 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
              Mar 29 16:15:47 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
              Mar 29 16:15:47 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
              Mar 29 16:15:47 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
              Mar 29 16:15:47 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
              Mar 29 16:15:47 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
              Mar 29 16:15:47 check_reload_status[433]: Syncing firewall
              Mar 29 16:15:47 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
              Mar 29 16:15:46 php-fpm[68905]: /rc.start_packages: The command '/usr/local/etc/rc.d/bandwidthd.sh stop' returned exit code '1', the output was 'No matching processes were found'
              Mar 29 16:15:46 check_reload_status[433]: Reloading filter
              Mar 29 16:15:46 check_reload_status[433]: Syncing firewall
              Mar 29 16:15:46 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): pfBlockerNG: saving DNSBL changes
              Mar 29 16:15:46 php-fpm[68905]: /rc.start_packages: Restarting/Starting all packages.
              Mar 29 16:15:45 kernel: em1: link state changed to DOWN
              Mar 29 16:15:45 check_reload_status[433]: Linkup starting em1
              Mar 29 16:15:45 check_reload_status[433]: Reloading filter
              Mar 29 16:15:45 check_reload_status[433]: Starting packages
              Mar 29 16:15:45 php-fpm[47031]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 192.168.xxx.xxx - Restarting packages.
              Mar 29 16:15:45 php-fpm[18742]: /rc.linkup: HOTPLUG: Configuring interface wan
              Mar 29 16:15:45 php-fpm[18742]: /rc.linkup: DEVD Ethernet attached event for wan
              Mar 29 16:15:45 php-fpm[18742]: /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
              Mar 29 16:15:45 check_reload_status[433]: Reloading filter
              Mar 29 16:15:43 php-fpm[47031]: /rc.newwanip: Creating rrd update script
              Mar 29 16:15:43 php-fpm[47031]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
              Mar 29 16:15:42 php-fpm[68905]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:15:40 check_reload_status[433]: Reloading filter
              Mar 29 16:15:40 check_reload_status[433]: Restarting OpenVPN tunnels/interfaces
              Mar 29 16:15:40 check_reload_status[433]: Restarting IPsec tunnels
              Mar 29 16:15:40 check_reload_status[433]: updating dyndns WAN_DHCP
              Mar 29 16:15:40 rc.gateway_alarm[96758]: >>> Gateway alarm: WAN_DHCP (Addr:192.168.xxx.xxx Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
              Mar 29 16:15:39 php-fpm[5949]: /rc.linkup: DEVD Ethernet detached event for wan
              Mar 29 16:15:39 php-fpm[5949]: /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
              Mar 29 16:15:39 check_reload_status[433]: Reloading filter
              Mar 29 16:15:39 check_reload_status[433]: updating dyndns wan
              Mar 29 16:15:38 php-fpm[47031]: /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1711725338] unbound[61492:0] error: bind: address already in use [1711725338] unbound[61492:0] fatal error: could not open ports'
              Mar 29 16:15:36 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:35 php-fpm[41925]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:32 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:32 php-fpm[41925]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:30 php-fpm[393]: /widgets/widgets/gateways.widget.php: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:29 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:28 php-fpm[41925]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:25 php-fpm[41925]: /rc.linkup: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:25 php-fpm[47031]: /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
              Mar 29 16:15:25 php-fpm[47031]: /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:15:25 php-fpm[47031]: /rc.newwanip: dpinger: timeout while retrieving status for gateway WAN_DHCP
              Mar 29 16:15:23 vnstatd[99232]: Monitoring (9): re1 (1000 Mbit) re0.20 (1000 Mbit) re0.10 (1000 Mbit) re0 (1000 Mbit) pfsync0 (1000 Mbit) pflog0 (1000 Mbit) enc0 (1000 Mbit) em1 (1000 Mbit) em0 (100 Mbit)
              Mar 29 16:15:23 vnstatd[99232]: Data retention: 48 5MinuteHours, 4 HourlyDays, 62 DailyDays, 25 MonthlyMonths, -1 YearlyYears, 20 TopDayEntries
              Mar 29 16:15:23 vnstatd[99232]: vnStat daemon 2.11 (pid:99232 uid:0 gid:0, SQLite 3.43.1)
              Mar 29 16:15:23 vnstatd[99817]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
              Mar 29 16:15:23 lighttpd_pfb[99016]: [pfBlockerNG] DNSBL Webserver started
              Mar 29 16:15:23 lighttpd_pfb[97011]: [pfBlockerNG] DNSBL Webserver stopped
              Mar 29 16:15:23 vnstatd[49902]: SIGTERM received, exiting.
              Mar 29 16:15:21 php-fpm[47031]: /rc.newwanip: Removing static route for monitor xxx.xxx.xxx.xxx and adding a new route through 192.168.xxx.xxx
              Mar 29 16:15:21 php-fpm[47031]: /rc.newwanip: rc.newwanip: on (IP address: 192.168.xxx.xxx) (interface: WAN[wan]) (real interface: em1).
              Mar 29 16:15:21 php-fpm[47031]: /rc.newwanip: rc.newwanip: Info: starting on em1.
              Mar 29 16:15:20 check_reload_status[433]: Restarting IPsec tunnels
              Mar 29 16:15:20 php-fpm[41925]: /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:15:20 check_reload_status[433]: rc.newwanip starting em1
              Mar 29 16:15:16 kernel: em1: link state changed to UP
              Mar 29 16:15:16 check_reload_status[433]: Linkup starting em1
              Mar 29 16:15:15 arpwatch[69275]: listening on re0.20
              Mar 29 16:15:15 arpwatch[68805]: listening on re0.10
              Mar 29 16:15:15 arpwatch[68500]: listening on em0
              Mar 29 16:15:15 arpwatch[68200]: listening on re0
              Mar 29 16:15:15 arpwatch[68058]: listening on re1
              Mar 29 16:15:13 kernel: em1: promiscuous mode enabled
              Mar 29 16:15:13 vnstatd[50127]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
              Mar 29 16:15:13 kernel: re0.20: promiscuous mode enabled
              Mar 29 16:15:13 kernel: re0.10: promiscuous mode enabled
              Mar 29 16:15:13 kernel: em0: promiscuous mode enabled
              Mar 29 16:15:13 kernel: re0: promiscuous mode enabled
              Mar 29 16:15:13 kernel: re1: promiscuous mode enabled
              Mar 29 16:15:13 kernel: em1: promiscuous mode disabled
              Mar 29 16:15:13 kernel: re0.20: promiscuous mode disabled
              Mar 29 16:15:13 kernel: re0: promiscuous mode disabled
              Mar 29 16:15:13 kernel: re0.10: promiscuous mode disabled
              Mar 29 16:15:13 kernel: em0: promiscuous mode disabled
              Mar 29 16:15:13 kernel: re1: promiscuous mode disabled
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
              Mar 29 16:15:12 kernel: em1: link state changed to DOWN
              Mar 29 16:15:12 check_reload_status[433]: Linkup starting em1
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
              Mar 29 16:15:12 php-fpm[41925]: /rc.linkup: HOTPLUG: Configuring interface wan
              Mar 29 16:15:12 php-fpm[41925]: /rc.linkup: DEVD Ethernet attached event for wan
              Mar 29 16:15:12 php-fpm[41925]: /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
              Mar 29 16:15:12 check_reload_status[433]: Reloading filter
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
              Mar 29 16:15:12 check_reload_status[433]: Syncing firewall
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
              Mar 29 16:15:12 php-fpm[68905]: /rc.start_packages: The command '/usr/local/etc/rc.d/bandwidthd.sh stop' returned exit code '1', the output was 'No matching processes were found'
              Mar 29 16:15:12 check_reload_status[433]: Reloading filter
              Mar 29 16:15:11 check_reload_status[433]: Syncing firewall
              Mar 29 16:15:11 php-fpm[68905]: /rc.start_packages: Configuration Change: (system): pfBlockerNG: saving DNSBL changes
              Mar 29 16:15:11 php-fpm[68905]: /rc.start_packages: Restarting/Starting all packages.
              Mar 29 16:15:10 check_reload_status[433]: Reloading filter
              Mar 29 16:15:10 check_reload_status[433]: Starting packages
              Mar 29 16:15:10 php-fpm[5949]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 192.168.xxx.xxx - Restarting packages.
              Mar 29 16:15:09 php-fpm[47031]: /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
              Mar 29 16:15:08 check_reload_status[433]: Reloading filter
              Mar 29 16:15:08 check_reload_status[433]: Restarting OpenVPN tunnels/interfaces
              Mar 29 16:15:08 check_reload_status[433]: Restarting IPsec tunnels
              Mar 29 16:15:08 check_reload_status[433]: updating dyndns WAN_DHCP
              Mar 29 16:15:08 rc.gateway_alarm[24575]: >>> Gateway alarm: WAN_DHCP (Addr:192.168.xxx.xxx Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
              Mar 29 16:15:08 php-fpm[5949]: /rc.newwanip: Creating rrd update script
              Mar 29 16:15:08 php-fpm[5949]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
              Mar 29 16:15:07 php-fpm[7795]: /rc.linkup: DEVD Ethernet detached event for wan
              Mar 29 16:15:07 php-fpm[7795]: /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
              Mar 29 16:15:07 check_reload_status[433]: Reloading filter
              Mar 29 16:15:07 check_reload_status[433]: updating dyndns wan
              Mar 29 16:15:06 check_reload_status[433]: Syncing firewall
              Mar 29 16:15:06 php-fpm[393]: /pkg_edit.php: Configuration Change: admin@192.168.6.68 (Local Database): Removed cron job for /bin/pkill -HUP -F /var/run/bandwidthd.pid

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                What exactly are you doing to trigger this? Reboot pfSense and the modem at the same time?

                N 1 Reply Last reply Reply Quote 0
                • N
                  Nosense 0 @stephenw10
                  last edited by

                  There are many opportunities:

                  Deactivate and activate the WAN interface on the Pfense
                  pull out the network plug
                  restart the modem

                  Everything leads to a DHCP Loop, but in these cases the Pfense was not restarted.

                  Only when the Pfense itself is restarted does the error do not occur until there is a problem with the modem or ISP again.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Ok, interesting. This is not a DHCP issue though, that's just a symptom. The WAN interface is actually losing link and that triggers everything else to restart.

                    What is the WAN connected to? Can you set it to a fixed link speed? That could be a good test if it's a link negotiation issue. You would have to set it both ends though. Or putting a switch in between would prevent it.

                    Was this working in some earlier pfSense version?

                    1 Reply Last reply Reply Quote 0
                    • GertjanG
                      Gertjan @Nosense 0
                      last edited by

                      @Nosense-0 said in pfSense WAN dhcp client exiting (error):

                      Since we have intermediary disorders from our ISP, we absolutely need the "Service_Watchdog", we cannot do without that.

                      I don't see the relation between the uplink connection and the usage of "Service_Watchdog".

                      This is what "Service_Watchdog" is doing :
                      It wakes up every minute.
                      Checks all the process that you've listed.
                      If it finds one that's not running, it starts it.

                      Btw : "Service_Watchdog" is a Netgate pfSense open source tool. I use=d the code the describe what it does.
                      It isn't smart or something like that.

                      It can not and will not help you in getting a better ISP connection.

                      I'm actually quiet convinced that it only worse your uplink connection quality and worse, the entire pfSense stability.

                      Processes like Suricata, unbound, etc etc don't just suddenly die. They do get restarted (example) when an interface event arrives : they are told to stopped, the orderly, controlled way - and when stopped, they get started.
                      At that moment : "Service_Watchdog" is also doing it's thing, and when it wakes up it does the same thing : example "Suricata" is just about to get restarted so "Service_Watchdog"found it stopped : it starts it. But is was already in the starting phase.
                      The scenario is set for a lot of hard to debug race conditions.

                      And things are even better / worse (you pick) : you are using other heavy weight processes like bandwidth, vnstat, pfBlockerng, and you have two independent sources of process restarts.
                      Just think about the mess that this can ... no, will create.

                      Of course, I hope to be wrong. As I said above, its easy to test.

                      No "help me" PM's please. Use the forum, the community will thank you.
                      Edit : and where are the logs ??

                      1 Reply Last reply Reply Quote 0
                      • N
                        Nosense 0
                        last edited by

                        I have now read extensively here in the forum and it is not an isolated case.

                        As with the others, all proposed solutions did not bring a solution.

                        It seems as if it were a problem in a script. Therefore, I will now stay with a fixed IP for the time being, since everything works as it used to be.

                        When I have time I'll take a look at the scripts.

                        Until then, thank you for the help.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          It has to be a combination of things. The dhclient failing to pull a lease or just restarting cannot bounce the NIC link.

                          1 Reply Last reply Reply Quote 0
                          • N
                            Nosense 0
                            last edited by

                            I have now installed a router for testing between the ISP modem and the Pfsense and the router confirms that the Pfsense has received the IP address via DHCP.

                            Therefore, I am quite sure that something has to go wrong in the script from the Pfense.

                            Just have to find out and need a little luck...

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Yup since setting it fixed avoids the issue it must be dhclient triggering something. But dhclient alone cannot cause the link to flap so something else much be in play.

                              If you're able to I would test a default config and see if that still hit's it.

                              N 1 Reply Last reply Reply Quote 0
                              • N
                                Nosense 0 @stephenw10
                                last edited by

                                @stephenw10 said in pfSense WAN dhcp client exiting (error):

                                If you're able to I would test a default config and see if that still hit's it.

                                It would be roughly as if the hotline recommend to reinstall Windows, logically then the error will disappear.

                                With Linux, I may have a little chance to find the mistake, so a new installation would only be considered as the very last one for me.

                                Thanks anyway

                                N 1 Reply Last reply Reply Quote 0
                                • N
                                  Nosense 0 @Nosense 0
                                  last edited by

                                  Not really found, but error is gone.

                                  Probable cause Keyword: GATEWAY

                                  I had set up two additional gateways to monitor two of my websites BUT the "Disable Gateway Monitoring Action" was activated.

                                  In the script "/etc/rc.linkup" there was the following line for dynamic IP:

                                  else {
                                  log_error("Hotplug event detected for {$friendly}({$iface}) dynamic IP address {$staticv4}, ({$addrs})");
                                  switch ($action) {
                                  case "stop":
                                  log_error("DEVD Ethernet detached event for {$staticv4}, {$iface}");

                                  So there had to be an error in the Pfsense LOG and I kept having this entry.

                                  A little further down in the script it says:

                                  if (empty($gw4ip)) {
                                  $gw4ip = get_interface_gateway_last($iface, 'inet');

                                  Why “_last”?
                                  That seemed strange to me and so I deleted all gateway entries and a separate view under STATUS/MONITOR that used a gateway.

                                  Switched everything to DHCP again and the error was gone.

                                  Strange thing with the "_last"...

                                  N 1 Reply Last reply Reply Quote 0
                                  • N
                                    Nosense 0 @Nosense 0
                                    last edited by

                                    By the way, of course, the default gateway was set fixed (no automatic) to DHCP Gateway.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Can you test that with DHCP WAN?

                                      The purpose of testing with a default config (not reinstalled) is to determine if the issue is with the config or something with the local setup. That would have removed those additional gateways likely showing it is a config issue.

                                      That is interesting though, I would not expect that to make any difference there. It tries to use the 'last gateway' value there for a dynamic gateway if a new one cannot be pulled.

                                      1 Reply Last reply Reply Quote 0
                                      • N
                                        Nosense 0
                                        last edited by

                                        I tested this with WAN DHCP.

                                        However, I now have a new problem.

                                        I haven't changed the gateway configuration for years.

                                        When I pull the network plug now, the gateway suddenly disappears in the dashboard and also under system/routing/gateways

                                        Is that normal?

                                        Where can you get a default config?

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          If it's dynamic and you have never set anything custom on it (which seems unlikely here!) then it might. Resaving the gateway to create a config entry for it should allow to show as present but down.

                                          1 Reply Last reply Reply Quote 0
                                          • N
                                            Nosense 0
                                            last edited by

                                            What a mess!

                                            At a Thin Client, Pfense threw me out when I configured via WAN (RJ45) and a WLAN defined as a LAN before I could activate the WLAN (changed the WAN rule); However, did not work because the Pfense had not tolerated the WLAN stick :-(

                                            An old PC reactivated, but had to buy network cards and wanted to give it an SSD, but no clon program worked :-(

                                            Well on the PC now Pfense from newly installed and only the backup (XML file) recorded from the production machine and the error reappears and it loops without end :-)

                                            Okay the test machine is of course not identical to production, so I had to delete a lot on the log, but the error remains identical.

                                            There seems to be an automatic that automatically creates the gateway wan_dhcp and a script which classifies this gateway as invalid and deletes and then the automatism creates it again and so the whole thing turns in a loop.

                                            Is just wondering what that is?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post