WAN Link Down causes pfSense to stop responding on LAN?

pst

@jhg said in WAN Link Down causes pfSense to stop responding on LAN?:

Feb 14 10:37:50 janus php-fpm[19256]: /rc.dyndns.update: Dynamic DNS () There was an error trying to determine the public IP for interface - wan (re1 ).

Hi,
I just wanted to say I have seen the same behaviour twice, but I have yet to report/debug it. I kept the logs though.

In my case a reboot (power cycle) appeared to fail, the pfSense never became accessible through SSH and no "boot complete" signal was heard. Logging in through the cosole I noticed the boot sequence was not completed, the dynamic DNS process that runs at bootup kept trying to obtain the WAN IP address continously, but failed.

So I think this problem is related to the DynDNS functionality that we both use. Anyone looking into fixing this in rc.dyndns.update should consider limit the number of attempts of obtaining the WAN address before giving up.

stephenw10

@jhg said in WAN Link Down causes pfSense to stop responding on LAN?:

Feb 14 10:48:03 janus kernel: re1: watchdog timeout
Feb 14 10:48:03 janus kernel: re1: link state changed to DOWN
Feb 14 10:48:03 janus check_reload_status[431]: Linkup starting re1
Feb 14 10:48:06 janus kernel: re1: link state changed to UP
Feb 14 10:48:06 janus check_reload_status[431]: Linkup starting re1
Feb 14 10:48:21 janus kernel: re1: watchdog timeout

That looks like a driver/hardware issue. Try using the alternative realtek-kmod driver.

Steve

jhg

@stephenw10 How are those watchdog timeouts different from the dozens before (are they)? Also, those are on re1 (WAN) not re0 (LAN).

stephenw10

They are not, I just grabbed those as examples. Anytime you see watchdog timeouts on Realtek hardware you should use the alt driver.

VioletDragon

Re0 is Realtek. Recommend replacing with an Intel Network Controller. Also looks like you are using a residential Internet Connection, if you are from UK, residential internet sucks and not maintained. I am with BT Business and Zen Business and don’t have problems.

jhg

@stephenw10 said in WAN Link Down causes pfSense to stop responding on LAN?:

That looks like a driver/hardware issue. Try using the alternative realtek-kmod driver.

OK, I installed the most recent kmod driver for FreeBSD 14 from this link (took a while to find it).

I checked that /boot/modules/if_re.ko was there, and created /boot/loader.conf.local as directed:

if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"

After a reboot (not reroot), kldstat still doesn't list the module:

]/root: kldstat
Id Refs Address                Size Name
 1   23 0xffffffff80200000  339ce08 kernel
 2    1 0xffffffff8359d000    1e2b0 opensolaris.ko
 3    1 0xffffffff835bc000     76f8 cryptodev.ko
 4    1 0xffffffff835c4000   5d7790 zfs.ko
 5    1 0xffffffff84420000     2220 cpuctl.ko
 6    1 0xffffffff84423000     3240 ichsmb.ko
 7    1 0xffffffff84427000     2178 smbus.ko

And the boot log messages still list the default driver:

/boot: dmesg | egrep 're[01]|rgephy'
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0x81400000-0x81400fff,0xa0100000-0xa0103fff irq 17 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x4c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 00:01:2e:xx:xx:xx
re0: netmap queues/slots: TX 1/256, RX 1/256
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0x81300000-0x81300fff,0xa0000000-0xa0003fff irq 18 at device 0.0 on pci2
re1: Using 1 MSI-X message
re1: Chip rev. 0x4c000000
re1: MAC rev. 0x00000000
miibus1: <MII bus> on re1
rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus1
rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re1: Using defaults for TSO: 65518/35/2048
re1: Ethernet address: 00:01:2e:xx:xx:xx
re1: netmap queues/slots: TX 1/256, RX 1/256
re0: link state changed to DOWN
re1: link state changed to DOWN
re0: link state changed to UP
re1: link state changed to UP

I saw that there's a directory /boot/loader.conf.d so I did

cp /boot/loader.conf.local /boot/loader.conf.d/realtek.conf

But that had no effect.

So as a last resort I added the loader config lines to /boot/loader.conf. After reboot the lines were moved but still present, but nothing changed in dmesg or kldstat.

I'm out of ideas. Suggestions?

I occasionally have ISP connection outages and experience what seems to be the usual pfSense issues. One is getting an IP address on WAN after the outage and the other, like you, losing the LAN access to pfSense. I do not think that’s a NIC driver issue. A year ago, I had a NETGATE SG-3100 that was having the same issues and actually I returned it because of that. In my experience the common Internet routes are better than pfSense in recovering from ISP outages.

I have decided to give pfSense a second chance on my own hardware this time and unfortunately I am struggling with the same issues again. Everything is fine when the ISP connection is stable, but those issues surface when it is not. One thing that seems to help with the LAN access issue, I’m still not sure, is to change the default gateway to WAN_DHCP from Automatic in System/Routing/Gateways. You may like to give it a try.

jhg

@kjk54 said in WAN Link Down causes pfSense to stop responding on LAN?:

One thing that seems to help with the LAN access issue, I’m still not sure, is to change the default gateway to WAN_DHCP from Automatic in System/Routing/Gateways.

I've already got it set to WAN_DHCP (defaulted that way at install). Thanks for the suggestion.

stephenw10

@jhg said in WAN Link Down causes pfSense to stop responding on LAN?:

OK, I installed the most recent kmod driver for FreeBSD 14

You have to use a module built against the actual kernel in pfSense. The realtek-kmod pkg is in our repo to provide that. So remove that pkg from FreeBSD and just 'pkg install' it from our repo.

johnpoz

@stephenw10 is there really an issue where if wan is down, be it dpinger or the actual interface down that you can not access the lan.. I have never seen such an issue on any device, be it my own, or vm or multiple flavors of netgate appliances - my old company, I got few 3100's into production for sites. And even one 2440..

I don't recall ever running into such an issue - yeah the gui can take a few to load, believe related to dns problems on the gui page, etc. But it would load after a delay.

But I don't recall ever not being able to access it from the lan side.. And its not like we didn't have internet outages.. Company would only allow me to use them for the local internet connection at some smaller sites.. So they were not on 5 9's sort of sla business connections..

I have had issues where internet down on my sg4860, and don't recall having any such issues.. Now I do have mine currently running through a switch, so even if internet is down the interface is up.. But before I set that up I had modem direct into pfsense interface, and don't ever recall having any issues..

I could for sure simulate this by either pulling ethernet at the modem itself or at pfsense.. Might do that when nobody streaming plex..

stephenw10

Nope, I've not seen that either.

The only time I've seen issues with that is if there's an incorrectly configured gateway on LAN or potentially a VPN. Such that when the WAN gateway goes down if the system default gateway is still set to auto the default becomes something invalid. That still shouldn't prevent access from LAN but if there are services trying to connect out they can use a lot of CPU cycles trying.

VioletDragon

@johnpoz I have only seen this activity when WAN is on the same LAN or LAGG interface with WAN in a VLAN but not seen this on a separate Interface I.e WAN Port and LAN port or LAGG then WAN on a Separate interface.

jhg

@stephenw10 said in WAN Link Down causes pfSense to stop responding on LAN?:

@jhg said in WAN Link Down causes pfSense to stop responding on LAN?:

OK, I installed the most recent kmod driver for FreeBSD 14

You have to use a module built against the actual kernel in pfSense. The realtek-kmod pkg is in our repo to provide that. So remove that pkg from FreeBSD and just 'pkg install' it from our repo.

Got it (finally :-) I should have realized pfSense would have its own repos in the list. kldstat now shows the module loaded. We'll see if the problem goes away.
Thanks