Problems with WAN Interface when ethernet cable is disconnected then reconnected



  • I am experiencing some problems when I disconnect and reconnect the ethernet cable from my WAN interface.

    The primary problem I am seeing is (what looks to me like) the WAN interface flapping when I reconnect the cable.  Other problems include the web GUI becoming very slow or not responding at all.

    To get the WAN interface working again I have to reboot the box.  I've tried a few of different things like disabling\enabling the WAN interface through the GUI but the only thing that works is a reboot.

    There is one error that repeats multiple times and probably tied to the root cause of the problem.

    "kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1" where 50.4.64.1 is the address same address for WAN Interface gateway before I pull the cable.

    I researched the "arpresolve: can't allocate llinfo" error but haven't found a solution yet.  A lot of the posts I've found said to add "192.168.100.1" to the Reject Lease From option on the WAN Interface page, but that was when that IP address was listed in the arpresolve error.  My cable modem management IP address is "192.168.100.1" so i thought I would try that just in case.  But that did not solve the problem.  I have since removed that adress from the Reject Lease option.

    Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:04 kernel em1: link state changed to UP
    Jun 8 18:12:04 check_reload_status Linkup starting em1
    Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:03 check_reload_status Reloading filter
    Jun 8 18:12:03 check_reload_status Restarting OpenVPN tunnels/interfaces
    Jun 8 18:12:03 check_reload_status Restarting ipsec tunnels
    Jun 8 18:12:03 check_reload_status updating dyndns WAN_DHCP
    Jun 8 18:12:03 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:03 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:03 php-fpm 20238 /rc.newwanip: Creating rrd update script
    Jun 8 18:12:03 php-fpm 20238 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
    Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:02 check_reload_status Reloading filter
    Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:12:02 kernel em1: link state changed to DOWN
    Jun 8 18:12:02 check_reload_status Linkup starting em1
    Jun 8 18:12:01 php-fpm 93063 /rc.linkup: HOTPLUG: Configuring interface wan
    Jun 8 18:12:01 php-fpm 93063 /rc.linkup: DEVD Ethernet attached event for wan
    Jun 8 18:12:01 check_reload_status Reloading filter
    Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
    Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
    Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
    Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: rc.newwanip: Info: starting on em1.
    Jun 8 18:12:00 php-fpm 66220 /rc.linkup: DEVD Ethernet detached event for wan
    Jun 8 18:12:00 check_reload_status Reloading filter
    Jun 8 18:12:00 check_reload_status updating dyndns wan
    Jun 8 18:12:00 check_reload_status Reloading filter
    Jun 8 18:12:00 php-fpm 18458 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
    Jun 8 18:12:00 php-fpm 18458 [pfBlockerNG] Starting cron process.
    Jun 8 18:11:59 php-fpm 18458 /rc.start_packages: Restarting/Starting all packages.
    Jun 8 18:11:58 check_reload_status Restarting ipsec tunnels
    Jun 8 18:11:58 php-fpm 20238 /rc.linkup: ROUTING: setting default route to 50.4.64.1
    Jun 8 18:11:58 check_reload_status rc.newwanip starting em1
    Jun 8 18:11:58 check_reload_status Starting packages
    Jun 8 18:11:58 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:58 php-fpm 76105 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 50.4.78.197 -> 50.4.78.197 - Restarting packages.
    Jun 8 18:11:58 php-fpm 93063 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
    Jun 8 18:11:58 kernel em1: link state changed to UP
    Jun 8 18:11:58 check_reload_status Linkup starting em1
    Jun 8 18:11:58 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:57 check_reload_status Reloading filter
    Jun 8 18:11:57 check_reload_status Restarting OpenVPN tunnels/interfaces
    Jun 8 18:11:57 check_reload_status Restarting ipsec tunnels
    Jun 8 18:11:57 check_reload_status updating dyndns WAN_DHCP
    Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:56 php-fpm 76105 /rc.newwanip: Creating rrd update script
    Jun 8 18:11:56 php-fpm 76105 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
    Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:55 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:55 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:55 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:55 kernel em1: link state changed to DOWN
    Jun 8 18:11:55 check_reload_status Linkup starting em1
    Jun 8 18:11:55 php-fpm 20238 /rc.linkup: HOTPLUG: Configuring interface wan
    Jun 8 18:11:55 php-fpm 20238 /rc.linkup: DEVD Ethernet attached event for wan
    Jun 8 18:11:55 check_reload_status Reloading filter
    Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
    Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
    Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
    Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: rc.newwanip: Info: starting on em1.
    Jun 8 18:11:54 php-fpm 66220 /rc.linkup: DEVD Ethernet detached event for wan
    Jun 8 18:11:54 check_reload_status Reloading filter
    Jun 8 18:11:54 check_reload_status updating dyndns wan
    Jun 8 18:11:53 check_reload_status Reloading filter
    Jun 8 18:11:53 php-fpm 48874 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
    Jun 8 18:11:52 php-fpm 48874 [pfBlockerNG] Starting cron process.
    Jun 8 18:11:52 php-fpm 48874 /rc.start_packages: Restarting/Starting all packages.
    Jun 8 18:11:52 php-fpm 48874 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
    Jun 8 18:11:51 check_reload_status Restarting ipsec tunnels
    Jun 8 18:11:51 php-fpm 76105 /rc.linkup: ROUTING: setting default route to 50.4.64.1
    Jun 8 18:11:51 check_reload_status rc.newwanip starting em1
    Jun 8 18:11:51 check_reload_status Starting packages
    Jun 8 18:11:51 php-fpm 48874 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 50.4.78.197 -> 50.4.78.197 - Restarting packages.
    Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:51 check_reload_status Reloading filter
    Jun 8 18:11:51 check_reload_status Restarting OpenVPN tunnels/interfaces
    Jun 8 18:11:51 check_reload_status Restarting ipsec tunnels
    Jun 8 18:11:51 check_reload_status updating dyndns WAN_DHCP
    Jun 8 18:11:51 kernel em1: link state changed to UP
    Jun 8 18:11:51 check_reload_status Linkup starting em1
    Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:50 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:50 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 php-fpm 48874 /rc.newwanip: Creating rrd update script
    Jun 8 18:11:49 php-fpm 48874 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:48 check_reload_status Linkup starting em1
    Jun 8 18:11:48 kernel em1: link state changed to DOWN
    Jun 8 18:11:48 php-fpm 76105 /rc.linkup: HOTPLUG: Configuring interface wan
    Jun 8 18:11:48 php-fpm 76105 /rc.linkup: DEVD Ethernet attached event for wan
    Jun 8 18:11:48 check_reload_status Reloading filter
    Jun 8 18:11:48 php-fpm 66220 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
    Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
    Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
    Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
    Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: rc.newwanip: Info: starting on em1.
    Jun 8 18:11:47 php-fpm 37612 /rc.linkup: DEVD Ethernet detached event for wan
    Jun 8 18:11:47 check_reload_status updating dyndns wan
    Jun 8 18:11:47 check_reload_status Reloading filter
    Jun 8 18:11:47 check_reload_status Restarting OpenVPN tunnels/interfaces
    Jun 8 18:11:47 check_reload_status Restarting ipsec tunnels
    Jun 8 18:11:47 check_reload_status updating dyndns WAN_DHCP
    Jun 8 18:11:45 check_reload_status Restarting ipsec tunnels
    Jun 8 18:11:45 php-fpm 48874 /rc.linkup: ROUTING: setting default route to 50.4.64.1
    Jun 8 18:11:45 check_reload_status rc.newwanip starting em1
    Jun 8 18:11:45 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:45 check_reload_status Reloading filter
    Jun 8 18:11:45 php-fpm 66220 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
    Jun 8 18:11:45 php-fpm 66220 [pfBlockerNG] Starting cron process.
    Jun 8 18:11:45 php-fpm 66220 /rc.start_packages: Restarting/Starting all packages.
    Jun 8 18:11:45 kernel em1: link state changed to UP
    Jun 8 18:11:45 check_reload_status Linkup starting em1
    Jun 8 18:11:45 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:45 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:44 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:44 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:44 check_reload_status Starting packages
    Jun 8 18:11:44 php-fpm 9542 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 50.4.78.197 -> 50.4.78.197 - Restarting packages.
    Jun 8 18:11:44 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:42 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:42 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:42 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
    Jun 8 18:11:42 kernel em1: link state changed to DOWN
    Jun 8 18:11:42 check_reload_status Linkup starting em1
    Jun 8 18:11:42 php-fpm 9542 /rc.newwanip: Creating rrd update script
    Jun 8 18:11:42 php-fpm 9542 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
    Jun 8 18:11:42 php-fpm 48874 /rc.linkup: HOTPLUG: Configuring interface wan
    Jun 8 18:11:42 php-fpm 48874 /rc.linkup: DEVD Ethernet attached event for wan
    Jun 8 18:11:42 check_reload_status Reloading filter
    Jun 8 18:11:41 php-fpm 37612 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
    Jun 8 18:11:41 php-fpm 37391 /rc.linkup: DEVD Ethernet detached event for wan
    Jun 8 18:11:41 check_reload_status Reloading filter
    Jun 8 18:11:41 check_reload_status updating dyndns wan
    Jun 8 18:11:41 php-fpm 9542 /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1496959901] unbound[21478:0] error: bind: address already in use [1496959901] unbound[21478:0] fatal error: could not open ports'
    Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
    Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
    Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
    Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: rc.newwanip: Info: starting on em1.
    Jun 8 18:11:39 check_reload_status Reloading filter
    Jun 8 18:11:39 php-fpm 48874 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
    Jun 8 18:11:39 php-fpm 48874 [pfBlockerNG] Starting cron process.
    Jun 8 18:11:38 php-fpm 48874 /rc.start_packages: Restarting/Starting all packages.
    Jun 8 18:11:38 check_reload_status Reloading filter



  • Here are some additional troubleshooting steps I've taken but have not worked for me.

    Changing the monitoring ip address on my wan gateway to 8.8.8.8

    Resetting all the firewall states.

    Disable and Enable the WAN interface.



  • More info….

    After I pull and reconnect the ethernet cable to the WAN port I ran a ping on the console and am seeing a very regular pattern.  I get 3-4(on average) replies followed by 3-4(on average) "ping: sendto: No route to host".  This continues until I reboot.



  • I've had this problem for over a year and didn't find a solution until now, when I did a dive into the code and put debugging around routing commands.

    I'm sharing my findings in case it helps you and others who Google this error.

    Symptoms:
    Reset a WAN interface and the routing dies.

    In the logs:

    Dec  6 06:46:35 fw kernel: arpresolve: can't allocate llinfo for 192.168.21.1 on lagg0_vlan21
    
    

    In the routing table:

    Destination        Gateway            Flags      Netif Expire
    192.168.21.0/24    link#14            U      lagg0_vl
    192.168.21.1       192.168.21.1       UGHS   lagg0_vl
    192.168.21.10      link#14            UHS         lo0
    
    

    Root cause:
    The DNS server and gateway are the same.

    Fix:
    Either change DNS servers, or patch the code (pfSense 2.2 and 2.3.5):

    /etc/inc/system.inc, near line 257:

    Change

                                mwexec("/sbin/route {$cmd} -host {$inet6}{$dnsserver} {$gatewayip}");
    
    ```To
    

    if( $dnsserver != $gatewayip )
                                mwexec("/sbin/route {$cmd} -host {$inet6}{$dnsserver} {$gatewayip}");


Log in to reply