Problems with WAN Interface when ethernet cable is disconnected then reconnected
-
I am experiencing some problems when I disconnect and reconnect the ethernet cable from my WAN interface.
The primary problem I am seeing is (what looks to me like) the WAN interface flapping when I reconnect the cable. Other problems include the web GUI becoming very slow or not responding at all.
To get the WAN interface working again I have to reboot the box. I've tried a few of different things like disabling\enabling the WAN interface through the GUI but the only thing that works is a reboot.
There is one error that repeats multiple times and probably tied to the root cause of the problem.
"kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1" where 50.4.64.1 is the address same address for WAN Interface gateway before I pull the cable.
I researched the "arpresolve: can't allocate llinfo" error but haven't found a solution yet. A lot of the posts I've found said to add "192.168.100.1" to the Reject Lease From option on the WAN Interface page, but that was when that IP address was listed in the arpresolve error. My cable modem management IP address is "192.168.100.1" so i thought I would try that just in case. But that did not solve the problem. I have since removed that adress from the Reject Lease option.
Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:04 kernel em1: link state changed to UP
Jun 8 18:12:04 check_reload_status Linkup starting em1
Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:04 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:03 check_reload_status Reloading filter
Jun 8 18:12:03 check_reload_status Restarting OpenVPN tunnels/interfaces
Jun 8 18:12:03 check_reload_status Restarting ipsec tunnels
Jun 8 18:12:03 check_reload_status updating dyndns WAN_DHCP
Jun 8 18:12:03 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:03 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:03 php-fpm 20238 /rc.newwanip: Creating rrd update script
Jun 8 18:12:03 php-fpm 20238 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:02 check_reload_status Reloading filter
Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:02 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:12:02 kernel em1: link state changed to DOWN
Jun 8 18:12:02 check_reload_status Linkup starting em1
Jun 8 18:12:01 php-fpm 93063 /rc.linkup: HOTPLUG: Configuring interface wan
Jun 8 18:12:01 php-fpm 93063 /rc.linkup: DEVD Ethernet attached event for wan
Jun 8 18:12:01 check_reload_status Reloading filter
Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
Jun 8 18:12:00 php-fpm 20238 /rc.newwanip: rc.newwanip: Info: starting on em1.
Jun 8 18:12:00 php-fpm 66220 /rc.linkup: DEVD Ethernet detached event for wan
Jun 8 18:12:00 check_reload_status Reloading filter
Jun 8 18:12:00 check_reload_status updating dyndns wan
Jun 8 18:12:00 check_reload_status Reloading filter
Jun 8 18:12:00 php-fpm 18458 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Jun 8 18:12:00 php-fpm 18458 [pfBlockerNG] Starting cron process.
Jun 8 18:11:59 php-fpm 18458 /rc.start_packages: Restarting/Starting all packages.
Jun 8 18:11:58 check_reload_status Restarting ipsec tunnels
Jun 8 18:11:58 php-fpm 20238 /rc.linkup: ROUTING: setting default route to 50.4.64.1
Jun 8 18:11:58 check_reload_status rc.newwanip starting em1
Jun 8 18:11:58 check_reload_status Starting packages
Jun 8 18:11:58 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:58 php-fpm 76105 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 50.4.78.197 -> 50.4.78.197 - Restarting packages.
Jun 8 18:11:58 php-fpm 93063 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
Jun 8 18:11:58 kernel em1: link state changed to UP
Jun 8 18:11:58 check_reload_status Linkup starting em1
Jun 8 18:11:58 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:57 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:57 check_reload_status Reloading filter
Jun 8 18:11:57 check_reload_status Restarting OpenVPN tunnels/interfaces
Jun 8 18:11:57 check_reload_status Restarting ipsec tunnels
Jun 8 18:11:57 check_reload_status updating dyndns WAN_DHCP
Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:56 php-fpm 76105 /rc.newwanip: Creating rrd update script
Jun 8 18:11:56 php-fpm 76105 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:56 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:55 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:55 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:55 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:55 kernel em1: link state changed to DOWN
Jun 8 18:11:55 check_reload_status Linkup starting em1
Jun 8 18:11:55 php-fpm 20238 /rc.linkup: HOTPLUG: Configuring interface wan
Jun 8 18:11:55 php-fpm 20238 /rc.linkup: DEVD Ethernet attached event for wan
Jun 8 18:11:55 check_reload_status Reloading filter
Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
Jun 8 18:11:54 php-fpm 76105 /rc.newwanip: rc.newwanip: Info: starting on em1.
Jun 8 18:11:54 php-fpm 66220 /rc.linkup: DEVD Ethernet detached event for wan
Jun 8 18:11:54 check_reload_status Reloading filter
Jun 8 18:11:54 check_reload_status updating dyndns wan
Jun 8 18:11:53 check_reload_status Reloading filter
Jun 8 18:11:53 php-fpm 48874 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Jun 8 18:11:52 php-fpm 48874 [pfBlockerNG] Starting cron process.
Jun 8 18:11:52 php-fpm 48874 /rc.start_packages: Restarting/Starting all packages.
Jun 8 18:11:52 php-fpm 48874 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
Jun 8 18:11:51 check_reload_status Restarting ipsec tunnels
Jun 8 18:11:51 php-fpm 76105 /rc.linkup: ROUTING: setting default route to 50.4.64.1
Jun 8 18:11:51 check_reload_status rc.newwanip starting em1
Jun 8 18:11:51 check_reload_status Starting packages
Jun 8 18:11:51 php-fpm 48874 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 50.4.78.197 -> 50.4.78.197 - Restarting packages.
Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:51 check_reload_status Reloading filter
Jun 8 18:11:51 check_reload_status Restarting OpenVPN tunnels/interfaces
Jun 8 18:11:51 check_reload_status Restarting ipsec tunnels
Jun 8 18:11:51 check_reload_status updating dyndns WAN_DHCP
Jun 8 18:11:51 kernel em1: link state changed to UP
Jun 8 18:11:51 check_reload_status Linkup starting em1
Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:51 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:50 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:50 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 php-fpm 48874 /rc.newwanip: Creating rrd update script
Jun 8 18:11:49 php-fpm 48874 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:49 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:48 check_reload_status Linkup starting em1
Jun 8 18:11:48 kernel em1: link state changed to DOWN
Jun 8 18:11:48 php-fpm 76105 /rc.linkup: HOTPLUG: Configuring interface wan
Jun 8 18:11:48 php-fpm 76105 /rc.linkup: DEVD Ethernet attached event for wan
Jun 8 18:11:48 check_reload_status Reloading filter
Jun 8 18:11:48 php-fpm 66220 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
Jun 8 18:11:47 php-fpm 48874 /rc.newwanip: rc.newwanip: Info: starting on em1.
Jun 8 18:11:47 php-fpm 37612 /rc.linkup: DEVD Ethernet detached event for wan
Jun 8 18:11:47 check_reload_status updating dyndns wan
Jun 8 18:11:47 check_reload_status Reloading filter
Jun 8 18:11:47 check_reload_status Restarting OpenVPN tunnels/interfaces
Jun 8 18:11:47 check_reload_status Restarting ipsec tunnels
Jun 8 18:11:47 check_reload_status updating dyndns WAN_DHCP
Jun 8 18:11:45 check_reload_status Restarting ipsec tunnels
Jun 8 18:11:45 php-fpm 48874 /rc.linkup: ROUTING: setting default route to 50.4.64.1
Jun 8 18:11:45 check_reload_status rc.newwanip starting em1
Jun 8 18:11:45 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:45 check_reload_status Reloading filter
Jun 8 18:11:45 php-fpm 66220 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Jun 8 18:11:45 php-fpm 66220 [pfBlockerNG] Starting cron process.
Jun 8 18:11:45 php-fpm 66220 /rc.start_packages: Restarting/Starting all packages.
Jun 8 18:11:45 kernel em1: link state changed to UP
Jun 8 18:11:45 check_reload_status Linkup starting em1
Jun 8 18:11:45 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:45 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:44 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:44 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:44 check_reload_status Starting packages
Jun 8 18:11:44 php-fpm 9542 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 50.4.78.197 -> 50.4.78.197 - Restarting packages.
Jun 8 18:11:44 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:43 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:42 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:42 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:42 kernel arpresolve: can't allocate llinfo for 50.4.64.1 on em1
Jun 8 18:11:42 kernel em1: link state changed to DOWN
Jun 8 18:11:42 check_reload_status Linkup starting em1
Jun 8 18:11:42 php-fpm 9542 /rc.newwanip: Creating rrd update script
Jun 8 18:11:42 php-fpm 9542 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jun 8 18:11:42 php-fpm 48874 /rc.linkup: HOTPLUG: Configuring interface wan
Jun 8 18:11:42 php-fpm 48874 /rc.linkup: DEVD Ethernet attached event for wan
Jun 8 18:11:42 check_reload_status Reloading filter
Jun 8 18:11:41 php-fpm 37612 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
Jun 8 18:11:41 php-fpm 37391 /rc.linkup: DEVD Ethernet detached event for wan
Jun 8 18:11:41 check_reload_status Reloading filter
Jun 8 18:11:41 check_reload_status updating dyndns wan
Jun 8 18:11:41 php-fpm 9542 /rc.newwanip: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1496959901] unbound[21478:0] error: bind: address already in use [1496959901] unbound[21478:0] fatal error: could not open ports'
Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 50.4.64.1
Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: ROUTING: setting default route to 50.4.64.1
Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: rc.newwanip: on (IP address: 50.4.78.197) (interface: WAN[wan]) (real interface: em1).
Jun 8 18:11:39 php-fpm 9542 /rc.newwanip: rc.newwanip: Info: starting on em1.
Jun 8 18:11:39 check_reload_status Reloading filter
Jun 8 18:11:39 php-fpm 48874 /rc.start_packages: The command '/sbin/ifconfig 'em0' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Jun 8 18:11:39 php-fpm 48874 [pfBlockerNG] Starting cron process.
Jun 8 18:11:38 php-fpm 48874 /rc.start_packages: Restarting/Starting all packages.
Jun 8 18:11:38 check_reload_status Reloading filter -
Here are some additional troubleshooting steps I've taken but have not worked for me.
Changing the monitoring ip address on my wan gateway to 8.8.8.8
Resetting all the firewall states.
Disable and Enable the WAN interface.
-
More info….
After I pull and reconnect the ethernet cable to the WAN port I ran a ping on the console and am seeing a very regular pattern. I get 3-4(on average) replies followed by 3-4(on average) "ping: sendto: No route to host". This continues until I reboot.
-
I've had this problem for over a year and didn't find a solution until now, when I did a dive into the code and put debugging around routing commands.
I'm sharing my findings in case it helps you and others who Google this error.
Symptoms:
Reset a WAN interface and the routing dies.In the logs:
Dec 6 06:46:35 fw kernel: arpresolve: can't allocate llinfo for 192.168.21.1 on lagg0_vlan21In the routing table:
Destination Gateway Flags Netif Expire 192.168.21.0/24 link#14 U lagg0_vl 192.168.21.1 192.168.21.1 UGHS lagg0_vl 192.168.21.10 link#14 UHS lo0Root cause:
The DNS server and gateway are the same.Fix:
Either change DNS servers, or patch the code (pfSense 2.2 and 2.3.5):/etc/inc/system.inc, near line 257:
Change
mwexec("/sbin/route {$cmd} -host {$inet6}{$dnsserver} {$gatewayip}"); ```Toif( $dnsserver != $gatewayip )
mwexec("/sbin/route {$cmd} -host {$inet6}{$dnsserver} {$gatewayip}");