WAN Interface Losing Link

[Scallica]

I am running 2.4.5-p1 on a ProtectLi 4-port Vault box (FW4B).On random occasions, my WAN interface will not maintain a link. In the web interface, I can see the public IP address appearing then disappearing. On the physical device, I can see the link light on the WAN interface go on and off. I don't think the WAN cable is bad because a reboot of the pfSense box resolves the issue every time. However, the issue has happened numerous times. Is this is a possible hardware issue or a pfsense bug?

Any help is greatly appreciated.

Below is the contents of /var/log/system.log:

Aug 31 09:24:22 pfSense upsd[82472]: User local-monitor@::1 logged into UPS [BR700]
Aug 31 09:24:22 pfSense rc.gateway_alarm[35070]: >>> Gateway alarm: WAN_DHCP (Addr:XXX.XXX.XXX.1 Alarm:1 RTT:5.277ms RTTsd:2.972ms Loss:33%)
Aug 31 09:24:22 pfSense check_reload_status: updating dyndns WAN_DHCP
Aug 31 09:24:22 pfSense check_reload_status: Restarting ipsec tunnels
Aug 31 09:24:22 pfSense check_reload_status: Restarting OpenVPN tunnels/interfaces
Aug 31 09:24:23 pfSense php-fpm[3655]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Aug 31 09:24:23 pfSense php-fpm[3655]: /rc.newwanip: Creating rrd update script
Aug 31 09:24:23 pfSense kernel: arpresolve: can't allocate llinfo for XXX.XXX.XXX.1 on igb0
Aug 31 09:24:24 pfSense kernel: igb0: link state changed to UP
Aug 31 09:24:24 pfSense kernel: arpresolve: can't allocate llinfo for XXX.XXX.XXX.1 on igb0
Aug 31 09:24:24 pfSense check_reload_status: Linkup starting igb0
Aug 31 09:24:24 pfSense php-fpm[10245]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Aug 31 09:24:24 pfSense check_reload_status: rc.newwanip starting igb0
Aug 31 09:24:24 pfSense php-fpm[42329]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. ''
Aug 31 09:24:24 pfSense check_reload_status: Restarting ipsec tunnels
Aug 31 09:24:25 pfSense php-fpm[3655]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - XXX.XXX.XXX.126 ->  XXX.XXX.XXX.126 - Restarting packages.
Aug 31 09:24:25 pfSense check_reload_status: Starting packages
Aug 31 09:24:25 pfSense php-fpm[98630]: /rc.newwanip: rc.newwanip: Info: starting on igb0.
Aug 31 09:24:25 pfSense php-fpm[98630]: /rc.newwanip: rc.newwanip: on (IP address: XXX.XXX.XXX.126) (interface: WAN[wan]) (real interface: igb0).
Aug 31 09:24:26 pfSense php-fpm[10245]: /rc.start_packages: Restarting/Starting all packages.
Aug 31 09:24:26 pfSense php-fpm[10245]: /rc.start_packages: Stopping service nut
Aug 31 09:24:26 pfSense upsmon[27034]: Signal 15: exiting
Aug 31 09:24:26 pfSense upsd[82472]: User local-monitor@::1 logged out from UPS [BR700]
Aug 31 09:24:26 pfSense upsd[82472]: mainloop: Interrupted system call
Aug 31 09:24:26 pfSense upsd[82472]: Signal 15: exiting
Aug 31 09:24:26 pfSense usbhid-ups[42596]: Signal 15: exiting
Aug 31 09:24:26 pfSense php-fpm[10245]: /rc.start_packages: Starting service nut
Aug 31 09:24:26 pfSense upsmon[56999]: Startup successful
Aug 31 09:24:26 pfSense usbhid-ups[59558]: Startup successful
Aug 31 09:24:27 pfSense upsd[78344]: listening on ::1 port 3493
Aug 31 09:24:27 pfSense upsd[78344]: listening on 127.0.0.1 port 3493
Aug 31 09:24:27 pfSense upsd[78344]: Connected to UPS [BR700]: usbhid-ups-BR700
Aug 31 09:24:27 pfSense upsd[78525]: Startup successful
Aug 31 09:24:27 pfSense php-fpm[98630]: /rc.newwanip: Gateway, none 'available' for inet6, use the first one configured. ''
Aug 31 09:24:28 pfSense check_reload_status: updating dyndns wan
Aug 31 09:24:28 pfSense check_reload_status: Reloading filter
Aug 31 09:24:28 pfSense php-fpm[6725]: /rc.linkup: DEVD Ethernet detached event for wan
Aug 31 09:24:29 pfSense upsd[78525]: User local-monitor@::1 logged into UPS [BR700]
Aug 31 09:24:29 pfSense upsmon[93462]: Startup successful
Aug 31 09:24:29 pfSense check_reload_status: Reloading filter
Aug 31 09:24:29 pfSense php-fpm[2386]: /rc.linkup: DEVD Ethernet attached event for wan
Aug 31 09:24:29 pfSense php-fpm[2386]: /rc.linkup: HOTPLUG: Configuring interface wan
Aug 31 09:24:29 pfSense check_reload_status: Linkup starting igb0
Aug 31 09:24:29 pfSense kernel: igb0: link state changed to DOWN
Aug 31 09:24:29 pfSense usbhid-ups[22851]: Startup successful
Aug 31 09:24:30 pfSense kernel: arpresolve: can't allocate llinfo for XXX.XXX.XXX.1 on igb0
Aug 31 09:24:30 pfSense upsd[40061]: listening on ::1 port 3493
Aug 31 09:24:30 pfSense upsd[40061]: listening on 127.0.0.1 port 3493
Aug 31 09:24:30 pfSense upsd[40061]: Connected to UPS [BR700]: usbhid-ups-BR700
Aug 31 09:24:30 pfSense upsd[40088]: Startup successful
Aug 31 09:24:31 pfSense php-fpm[98630]: /rc.newwanip: Forcefully reloading IPsec
Aug 31 09:24:31 pfSense check_reload_status: Reloading filter
Aug 31 09:24:31 pfSense php-fpm[98630]: /rc.newwanip: WARNING: Setting i_dont_care_about_security_and_use_aggressive_mode_psk option because a phase 1 is configured using aggressive mode with pre-shared keys. This is not a secure configuration.
Aug 31 09:24:32 pfSense rc.gateway_alarm[51490]: >>> Gateway alarm: WAN_DHCP (Addr:XXX.XXX.XXX.1 Alarm:1 RTT:7.047ms RTTsd:2.679ms Loss:28%)
Aug 31 09:24:32 pfSense check_reload_status: updating dyndns WAN_DHCP
Aug 31 09:24:32 pfSense check_reload_status: Restarting ipsec tunnels
Aug 31 09:24:32 pfSense check_reload_status: Restarting OpenVPN tunnels/interfaces
Aug 31 09:24:32 pfSense kernel: arpresolve: can't allocate llinfo for XXX.XXX.XXX.1 on igb0
Aug 31 09:24:32 pfSense upsd[40088]: User local-monitor@::1 logged into UPS [BR700]
Aug 31 09:24:32 pfSense php-fpm[98630]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Aug 31 09:24:32 pfSense php-fpm[98630]: /rc.newwanip: Creating rrd update script
Aug 31 09:24:33 pfSense php-fpm[18250]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
Aug 31 09:24:33 pfSense kernel: arpresolve: can't allocate llinfo for XXX.XXX.XXX.1 on igb0
Aug 31 09:24:33 pfSense kernel: arpresolve: can't allocate llinfo for XXX.XXX.XXX.1 on igb0
Aug 31 09:24:33 pfSense kernel: igb0: link state changed to UP
Aug 31 09:24:33 pfSense check_reload_status: Linkup starting igb0
Aug 31 09:24:33 pfSense check_reload_status: rc.newwanip starting igb0
Aug 31 09:24:34 pfSense php-fpm[2386]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. ''

[akuma1x]

What I would do, to test if it is in fact a hardware issue, would be to, if possible, to switch to a different port for your WAN connection.

That box has 4 network ports. Do you have all of them in use?

Jeff

[Scallica]

@akuma1x Thanks for the quick reply. No, I am only using one port for WAN, and one port for LAN. I will try your suggestion.

[stephenw10]

Yup swap the ports, see if the fault follows the port.

Try putting a switch in between the WAN and modem, see if one side still looses link and which side it is.

Steve

[Scallica]

@stephenw10 Thanks, I will try the switch test. When the issue was happening, I removed the Ethernet cable from the WAN port, waited a few seconds, then plugged the cable back in. The issue persisted.

[Scallica]

The issue happened again today. The WAN link light was flashing on/off. I removed the WAN cable and plugged it into a gigabit switch. The link light remained on. I reconnected the WAN cable and then changed the WAN interface setting from autoselect to 1000base-T full duplex. The link light remained on.

[bingo600]

Watch out if you are "Only forcing one end of a link" , some (if not all) chipsets.
Will fallback to HDPX if the other end will not negotiate (is forced)

I always recommend to "force to the same in both ends" if possible.

/Bingo

[Gertjan]

@Scallica said in WAN Interface Losing Link:

Gateway alarm: WAN_DHCP .... Loss:28%)

and the threshold dpinger (the gateway reacability tester) is "pulling" the plug when ? 30 %

Your WAN goes down probably because the "Gateway alarm" (dpinger) is instructed to do so : to many ICMP packets are lost, the connection is considered bad. It resets your WAN.

Try checking this option :

Add to you do-check-list :
Why are there so many ICMP packets get lost ?
What happens if you instruct dpinger (gateway alarm tester) NOT to pull the plug : doing nothing when many packets are lost
Like this :

[Scallica]

I always recommend to "force to the same in both ends" if possible.

I agree, but in this case, the other end is a Verizon FiOS ONT. There have been no further issues in the last 30 days.

[Scallica]

@bingo600 As a test, I switched the Speed and Duplex setting back to autonegotiate. Immediately, the WAN interface started going up and down. The link lights would stay on for two seconds and off for one second. I think it is safe to say there is an negotiation issue between the WAN interface on the pfSense box and the Ethernet interface of the Verizon ONT.

[bingo600]

@Scallica

Hmm my bad (i need glasses)
I missed the 1000Mb .. Afaik 1000-TX can only run Fdpx

So setting 1000-Fdpx seems fine

/Bingo

[stephenw10]

Technically autonegotiation is mandatory for 1000base-T if it meets the spec for 802.3ab.
https://en.wikipedia.org/wiki/Gigabit_Ethernet#1000BASE-T
Though many devices allow it to be disabled which means you can link to equipment that does not comply with the spec.

Steve

[Ghost 0]

Hello, I'm currently experiencing similar issue. My WAN keeps dropping every few days. I couldn't figure it out until I read the gateway log that seems to suggest the WAN interface is the issue. Comcast is my ISP, and I own my own cable modem (Netgear CM700). My gateway {pfSense 2.5.2} is an old repurposed HP workstation with a 4-port intel NIC. I have been running pfSense for about two years. I have a fairly complex system with multiple LANs /VLans and a plethora of managed switches. It could be my cable modem that's on its last leg since it has a poor reputation (the controversial PUMA chipset) or my switches due to an improperly configured STP because sometimes the issue would be temporarily resolved after rebooting only the switches. However, the issue continues after properly configuring STP. Refreshing openvpn would also temporarily resolved the issue. Also quickly unplugging and plugging solely the WAN ethernet cable would temporarily resolved the problem. I'm currently at a lost for solutions. It wasn't until today when I finally really read the pfSense logs in details that pointed me to the potential culprit, the WAN interface. Before that, I tried all types of fruitless solutions. I serendipitously stumbled on your potential solution, changing the speed/duplex of the WAN interface from autoselect to 1000 base T full duplex. I have applied it and hope that it works; fingers crossed and will find out in a few days if this was the issue.