WAN dropouts aplenty! I'm almost out of ideas: kernel - re0: link state changed to DOWN

enricosanchez

I've been troubleshooting WAN dropouts for a couple of months now and have almost run out of ideas, the following event (re0: link state changed to DOWN) repeats randomly throughout the day, sometimes rarely, sometimes frequently. re0 is my WAN NIC:

Jul 18 21:05:02	php-fpm	362	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Jul 18 21:05:01	check_reload_status	400	Reloading filter
Jul 18 21:05:01	check_reload_status	400	updating dyndns wan
Jul 18 21:04:58	check_reload_status	400	Reloading filter
Jul 18 21:04:58	php-fpm	362	/rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: WAN[wan]) (real interface: re0).
Jul 18 21:04:58	php-fpm	362	/rc.newwanip: rc.newwanip: Info: starting on re0.
Jul 18 21:04:57	check_reload_status	400	Restarting IPsec tunnels
Jul 18 21:04:57	php-fpm	363	/rc.linkup: Gateway, none 'available' for inet6, use the first one configured. ''
Jul 18 21:04:57	check_reload_status	400	rc.newwanip starting re0
Jul 18 21:04:56	php-fpm	363	/rc.linkup: HOTPLUG: Configuring interface wan
Jul 18 21:04:56	php-fpm	363	/rc.linkup: DEVD Ethernet attached event for wan
Jul 18 21:04:55	kernel		re0: link state changed to UP
Jul 18 21:04:55	check_reload_status	400	Linkup starting re0
Jul 18 21:04:19	check_reload_status	400	Reloading filter
Jul 18 21:04:19	php-fpm	363	/rc.linkup: Shutting down Router Advertisment daemon cleanly
Jul 18 21:04:17	php-fpm	363	/rc.linkup: DEVD Ethernet detached event for wan
Jul 18 21:04:16	kernel		re0: link state changed to DOWN
Jul 18 21:04:16	check_reload_status	400	Linkup starting re0
Jul 18 20:50:00	sshguard	61968	Now monitoring attacks.
Jul 18 20:50:00	sshguard	27243	Exiting on signal.

I'm assuming the above log info is suggesting something between PFsense and the ISP is failing. I've changed the cables, changed the pfsense hardware, switched LAN/WAN NICs, updated realtek drivers (one of the NICs is realtek), and plenty more. Can anyone give me some sensible next step ideas?

Version 2.6.0-RELEASE (amd64)
built on Mon Jan 31 19:57:53 UTC 2022
FreeBSD 12.3-STABLE

CPU Type Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (inactive)
QAT Crypto: No

BIOS Vendor: Dell Inc.
Version: A23
Release Date: Mon Jun 25 2018

bmeeks

Realtek NICs and FreeBSD are notorious for poor reliability and performance. You can search here on the Netgate forums and find plenty of folks with issues caused by the Realtek driver in FreeBSD.

Some users have had good luck using a custom-compiled driver for FreeBSD directly from Realtek (instead of using the more generic one packaged with the FreeBSD operating system). pfSense is based on FreeBSD, so whatever driver is packaged with FreeBSD is what pfSense will be using.

If possible, I would change out the Realtek NIC for an Intel variety. If that is not possible, you can search around here on the forum for some threads about using a custom Realtek driver. Here is one of those: https://forum.netgate.com/topic/172869/realtek-driver-1-97-00.

enricosanchez

@bmeeks, thanks for the reply, I've already updated the realtek drivers, but I've installed the one you linked to, it seems to be a bit newer.

Regardless, if I switch the WAN/LAN NIC the problem persists, the LAN NIC is intel.

I'll update if the newer drivers make a difference.

bmeeks

@enricosanchez said in WAN dropouts aplenty! I'm almost out of ideas: kernel - re0: link state changed to DOWN:

@bmeeks, thanks for the reply, I've already updated the realtek drivers, but I've installed the one you linked to, it seems to be a bit newer.

Regardless, if I switch the WAN/LAN NIC the problem persists, the LAN NIC is intel.

I'll update if the newer drivers make a difference.

If the problem follows the interface (meaning the same issue occurs when you reverse LAN and WAN interfaces) and you have even swapped your pfSense hardware, then that really points to a hardware problem on the ISP's side. You say you have swapped cables, so that should eliminate cabling. What kind of ISP device do you have? Perhaps it has a hardware problem. Could be a dirty or corroded RJ45 connector, for example.

The error message is saying "hotplug" and "link down" events. Both of those mean the physical interface link disappeared. That's like losing power or unplugging the cable.

enricosanchez

@bmeeks said in WAN dropouts aplenty! I'm almost out of ideas: kernel - re0: link state changed to DOWN:

interface (me

I called the ISP this morning, they suggested it was on my end. There's a device plugged into the WAN which powers a VDSL unit out in the street which I can replace tomorrow (all cables have been replaced already). The driver you linked to before has kept the WAN up for the last 30 mins, so fingers crossed that fixed it.

EDIT: no it's still dropping with the newer drivers

enricosanchez

Just an update on this, I purchased a new intel NIC and the connection has been solid ever since, no dropped packets, no wan down. For anyone else reading, the updated realtek drivers helped a little, but it took a day before the dropouts slowed (no idea why). It also helped when I moved my realtek nic from WAN to LAN. The ultimate fix appears to be as @bmeeks suggested "change out the Realtek NIC for an Intel variety"