Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    LAN interface toggling up/down

    Scheduled Pinned Locked Moved General pfSense Questions
    6 Posts 3 Posters 2.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      bobsuruncle
      last edited by

      In the last 2 weeks my network has gone down (internal hosts not accessible from outside) 3 times for about an hour.  This is unusual.  The most recent time I was home and noticed that none of the LAN devices could communicate with each other.  While troubleshooting I found that new or restarted devices couldn't get an IP address over DHCP from the pfsense router.  The only way I could access the router was via the WAN interface (using phone on cellular network) with vpn (openvpn daemon on pfsense).  In reviewing the pfsense logs, I could see the LAN interface going up and down every 3 minutes (log snippet below).  Devices on the network could not communicate at all (not just every 3 minutes).  Restarting pfsense did not resolve the issue.  I also tried moving the the cable connected from EM1 to the switch to a different port, but this also did not resolve the issue.  After an hour, the issue resolved itself.  Since this has presumably happened 3 times, I'd like to understand what's causing it and resolve it.

      I'd appreciate any tips to help me track down this issue.  Thanks in advance.

      
>grep "watchdog" /var/log/system.log
Dec 28 16:51:32 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 16:54:04 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 16:56:39 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 16:59:34 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:02:07 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:05:04 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:08:33 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:25:27 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:28:04 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:30:34 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:32:52 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:35:27 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:38:05 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:40:21 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:43:05 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:45:27 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:47:56 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 17:50:34 fw1 kernel: em1: Watchdog timeout -- resetting

      full log snippet of 1 of many cycles

      
Dec 28 16:51:32 fw1 check_reload_status: Linkup starting em1
Dec 28 16:51:32 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 16:51:32 fw1 kernel: em1: link state changed to DOWN
Dec 28 16:51:33 fw1 php-fpm[97430]: /rc.linkup: DEVD Ethernet detached event for lan
Dec 28 16:51:33 fw1 check_reload_status: Reloading filter
Dec 28 16:51:35 fw1 xinetd[17925]: Starting reconfiguration
Dec 28 16:51:35 fw1 xinetd[17925]: Swapping defaults
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 6969-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19000-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19000-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19001-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19002-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19002-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19003-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19003-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19004-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19004-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19005-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19005-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19006-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19006-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19007-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19007-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19008-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19008-udp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19009-tcp
Dec 28 16:51:35 fw1 xinetd[17925]: readjusting service 19009-udp
Dec 28 16:51:35 fw1 xinetd[17925]: Reconfigured: new=0 old=20 dropped=0 (services)
Dec 28 16:51:35 fw1 check_reload_status: Linkup starting em1
Dec 28 16:51:35 fw1 kernel: em1: link state changed to UP
Dec 28 16:51:36 fw1 php-fpm[54133]: /rc.linkup: DEVD Ethernet attached event for lan
Dec 28 16:51:36 fw1 php-fpm[54133]: /rc.linkup: HOTPLUG: Configuring interface lan
Dec 28 16:51:37 fw1 check_reload_status: Restarting ipsec tunnels
Dec 28 16:51:40 fw1 check_reload_status: updating dyndns lan
Dec 28 16:51:40 fw1 check_reload_status: Reloading filter
Dec 28 16:51:41 fw1 xinetd[17925]: Starting reconfiguration
Dec 28 16:51:41 fw1 xinetd[17925]: Swapping defaults
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 6969-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19000-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19000-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19001-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19002-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19002-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19003-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19003-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19004-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19004-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19005-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19005-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19006-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19006-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19007-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19007-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19008-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19008-udp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19009-tcp
Dec 28 16:51:41 fw1 xinetd[17925]: readjusting service 19009-udp
Dec 28 16:51:41 fw1 xinetd[17925]: Reconfigured: new=0 old=20 dropped=0 (services)
Dec 28 16:54:04 fw1 check_reload_status: Linkup starting em1
Dec 28 16:54:04 fw1 kernel: em1: Watchdog timeout -- resetting
Dec 28 16:54:04 fw1 kernel: em1: link state changed to DOWN
Dec 28 16:54:05 fw1 php-fpm[2278]: /rc.linkup: DEVD Ethernet detached event for lan
Dec 28 16:54:05 fw1 check_reload_status: Reloading filter
Dec 28 16:54:06 fw1 xinetd[17925]: Starting reconfiguration
Dec 28 16:54:06 fw1 xinetd[17925]: Swapping defaults
Dec 28 16:54:06 fw1 xinetd[17925]: readjusting service 6969-udp
Dec 28 16:54:06 fw1 xinetd[17925]: readjusting service 19000-tcp
      1 Reply Last reply Reply Quote 0
      • H
        heper
        last edited by

        nic about to die
        switch-port about to die
        cabling issue

        might be something else, but i'd start with verifying those

        1 Reply Last reply Reply Quote 0
        • B
          bobsuruncle
          last edited by

          thanks, cable has not changed or been disturbed but that's easy to replace.  As I said, I changed the switch port without resolution.  How would I go about pinpointing/eliminating the NIC as the problem?

          1 Reply Last reply Reply Quote 0
          • H
            heper
            last edited by

            you could swap lan ==> wan / wan===>lan. If the wan interface starts acting up, you know its the interface (if you've checked/verified the switch/cable)

            1 Reply Last reply Reply Quote 0
            • B
              bobsuruncle
              last edited by

              ah yes, good idea.  what's the best way to swap it without re-configuring everything?

              1 Reply Last reply Reply Quote 0
              • w0wW
                w0w
                last edited by

                @bobsuruncle:

                ah yes, good idea.  what's the best way to swap it without re-configuring everything?

                Download and edit manually your config.xml is the best way, I think. Always use saved copy for backup!
                Search for interfaces, ex em0 and em1 and swap them everywhere (match whole word only). For my setup it counts only 1 match for every interface assigned. After editing double check everything and restore your edited config on firewall using gui.
                The other way is just to reassign it via console menu or even gui, but I did not try this way and it could be complicated as you need to re-plug your cables on the fly may be.

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.