WAN Loosing IP
-
I have been tracking this for a bit. This is about the fourth time this has happened. The symptom is internet traffic outbound becomes very slow / flacky.
I setup a baseline ping to router (LAN), router (WAN) and google.
WAN and google pings show same pattern of 10 or so packets tranfer then droping two then back.
Looking at logs: The one things that jumps out
Jun 1 23:31:08 rtsold 87757 <cap_rssend> sendmsg on igc0: Permission denied
Jun 1 23:31:24 php-fpm 67830 /rc.newwanip: The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_igc0.pid -M /var/etc/rtsold_igc0_script.sh -O /var/etc/rtsold_igc0_script.sh igc0' returned exit code '1',My initial guess is the ISP is trying to renew .. force renewal.. or visa versa the firewall is trying to refresh IP and their is some kind of script error. I googled around and got a few hits on this forum but nothing that matched.... or I did not find the correct posting :)
The unfortunate "fix" is reboot of the pfsense firewall. I at first assumed it was the upstream DOCSYS modem (Model: DPC3008-CC) which is just a bridge of cable to RJ45. I tried to create a baseline by rebooting that independantly (power cycle as well as use the ISP app to reset via diagnostic sequence) but no change. I don't have a very deep baseline , just two runs of that.. where the issue did NOT resolve with rebooting the DOCSIS modem. I will continue to work on this. On other thing I did not check is just force link state change on WAN interface (unplug cable) and see if that gets me a difference.
I would assume the next debug step is to run wire capture of the WAN during this state.. or maybe their is another suggestion.
Jun 1 23:31:07 check_reload_status 409 Linkup starting igc0 Jun 1 23:31:07 kernel igc0: link state changed to UP Jun 1 23:31:08 rtsold 87757 <cap_rssend> sendmsg on igc0: Permission denied Jun 1 23:31:08 check_reload_status 409 rc.newwanip starting igc0 Jun 1 23:31:08 php-fpm 370 /rc.linkup: calling interface_dhcpv6_configure. Jun 1 23:31:08 php-fpm 370 /rc.linkup: Accept router advertisements on interface igc0 Jun 1 23:31:08 php-fpm 370 /rc.linkup: Killing running rtsold process Jun 1 23:31:09 php-fpm 89909 /rc.newwanip: rc.newwanip: Info: starting on igc0. Jun 1 23:31:09 php-fpm 89909 /rc.newwanip: rc.newwanip: on (IP address: 56.14.73.161) (interface: WAN[wan]) (real interface: igc0). Jun 1 23:31:09 php-fpm 89909 /rc.newwanip: Accept router advertisements on interface igc0 Jun 1 23:31:09 php-fpm 89909 /rc.newwanip: Starting rtsold process Jun 1 23:31:10 php-fpm 370 /rc.linkup: Starting rtsold process Jun 1 23:31:11 check_reload_status 409 Reloading filter Jun 1 23:31:11 rtsold 4616 <cap_rssend> sendmsg on igc0: Permission denied Jun 1 23:31:12 php-fpm 370 /rc.linkup: The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_igc0.pid -M /var/etc/rtsold_igc0_script.sh -O /var/etc/rtsold_igc0_script.sh igc0' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Jun 1 23:31:12 php-fpm 370 /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Jun 1 23:31:12 check_reload_status 409 Restarting IPsec tunnels Jun 1 23:31:15 rtsold 4616 <cap_rssend> sendmsg on igc0: Permission denied Jun 1 23:31:16 check_reload_status 409 updating dyndns wan Jun 1 23:31:16 check_reload_status 409 Reloading filter Jun 1 23:31:16 php-fpm 4288 /rc.linkup: DEVD Ethernet detached event for wan Jun 1 23:31:17 check_reload_status 409 Reloading filter Jun 1 23:31:17 php-fpm 90936 /rc.linkup: DEVD Ethernet attached event for wan Jun 1 23:31:17 php-fpm 90936 /rc.linkup: HOTPLUG: Configuring interface wan Jun 1 23:31:17 check_reload_status 409 Linkup starting igc0 Jun 1 23:31:17 kernel igc0: link state changed to DOWN Jun 1 23:31:18 rc.gateway_alarm 63791 >>> Gateway alarm: WAN_DHCP (Addr:56.14.72.1 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) Jun 1 23:31:18 check_reload_status 409 updating dyndns WAN_DHCP Jun 1 23:31:18 check_reload_status 409 Restarting IPsec tunnels Jun 1 23:31:18 check_reload_status 409 Restarting OpenVPN tunnels/interfaces Jun 1 23:31:18 php-fpm 65848 /rc.dyndns.update: Dynamic DNS noip (acme.edu): IP address could not be extracted from Check IP Service Jun 1 23:31:18 php-fpm 65848 /rc.dyndns.update: Dynamic DNS (acme.edu) There was an error trying to determine the public IP for interface - wan (igc0 ). Jun 1 23:31:19 php-fpm 370 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP. Jun 1 23:31:19 php-fpm 65848 /rc.dyndns.update: Dynamic DNS noip (penguinpages.ddns.net): IP address could not be extracted from Check IP Service Jun 1 23:31:19 php-fpm 65848 /rc.dyndns.update: Dynamic DNS (penguinpages.ddns.net) There was an error trying to determine the public IP for interface - wan (igc0 ). Jun 1 23:31:20 rtsold 4616 <cap_rssend> sendmsg on igc0: Permission denied Jun 1 23:31:20 php-fpm 65848 /rc.dyndns.update: Dynamic DNS noip (www.acme.edu): IP address could not be extracted from Check IP Service Jun 1 23:31:20 php-fpm 65848 /rc.dyndns.update: Dynamic DNS (www.acme.edu) There was an error trying to determine the public IP for interface - wan (igc0 ). Jun 1 23:31:20 check_reload_status 409 Linkup starting igc0 Jun 1 23:31:20 kernel igc0: link state changed to UP Jun 1 23:31:21 check_reload_status 409 rc.newwanip starting igc0 Jun 1 23:31:21 php-fpm 90936 /rc.linkup: calling interface_dhcpv6_configure. Jun 1 23:31:21 php-fpm 90936 /rc.linkup: Accept router advertisements on interface igc0 Jun 1 23:31:21 php-fpm 90936 /rc.linkup: Starting rtsold process Jun 1 23:31:22 php-fpm 67830 /rc.newwanip: rc.newwanip: Info: starting on igc0. Jun 1 23:31:22 php-fpm 67830 /rc.newwanip: rc.newwanip: on (IP address: 56.14.73.161) (interface: WAN[wan]) (real interface: igc0). Jun 1 23:31:22 php-fpm 67830 /rc.newwanip: Accept router advertisements on interface igc0 Jun 1 23:31:22 php-fpm 67830 /rc.newwanip: Starting rtsold process Jun 1 23:31:23 php-fpm 90936 /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Jun 1 23:31:23 check_reload_status 409 Restarting IPsec tunnels Jun 1 23:31:24 rtsold 45932 <cap_rssend> sendmsg on igc0: Permission denied Jun 1 23:31:24 php-fpm 67830 /rc.newwanip: The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_igc0.pid -M /var/etc/rtsold_igc0_script.sh -O /var/etc/rtsold_igc0_script.sh igc0' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Jun 1 23:31:24 check_reload_status 409 Reloading filter Jun 1 23:31:27 check_reload_status 409 updating dyndns wan
-
possibly related: rtsold not running, IPv6 WAN (DHCP) keeps losing connectivity
-
Thanks for response. I think the messages are standard response sequence of DHCP lack of response on the WAN interface. Of course I do not have much of a baseline but turns out the ISP has had a LOT of issues over the last week in our area. I assumed it was me.... I think it is / was them. I will post as I get better baseline.