Cable Internet and pfSense - Interface keeps dropping
-
I have begun to run into this same issue. I was starting to suspect my Watchguard M400 was just beginning to show signs of its age but have started experimenting more as I believe that there is something else at play.
In my case I added a second backup WAN using a Netgear cell router and have found that this device can exhibit the same issues at times. However not during an IP refresh.
My primary ISP is Astound and I use a Motorola MB8611 modem.
I will have to compare my logs next time it happens to either interface.
-sometimes just hard reboot of the modem will cause this issue for me...
-
To find out who is doing this :
put a switch between your pfSense WAN and your ISP device.
get two new short ether cables to make the connection.
Now, wait and see.Btw : looking at the sub 10 ms up down up down up down wan events, this really doesn't' like a software event that can take the connection down for a moment, this looks like bad contact, bad cable bad NIC etc.
-
@Gertjan Hi there, I would agree, however the equipment was replaced a few times, that includes different cables, NICs, modems - and pfsense always had this same behaviour, where if the wan dropped for a moment, it would never re-connect and would just keep connecting and dropping.
I will put a switch in between the modem and pfsense and see if anything changes.
-
@bartkus05 said in Cable Internet and pfSense - Interface keeps dropping:
where if the wan dropped for a moment
If the pfSense WAN goes down, this event is used to remove the WAN IP, as there is no interface anymore, and internal routes are also removed.
From then on, no more connection.
Then, normally, the WAN comes back, UP.
This fires a new system event, and one of them is that the system starts the DHCP client process, as the DHCP client is assigned to this WAN interface with this option :
The DHCP client uses the interface, it doesn't (can't) take it down or up. It uses the connection, a serial connection the the NIC on the other side, to send out 'bits' and waits for an answer from the NIC on the other side of the cable.
Your issue might be the pfSense WAN NIC - so, an easy test : swap (re assign) it with another interface.
If the issue stays on WAN : this mens its not the pfSense WAN interfaceb but the ISP device that took the connection down : modems can do this to signal down stream (signal to pfSense) that their uplink went 'bad'. The down up signal is used to re activate the DHCP client, as a new IP connection has to be established. -
@Gertjan Here's what I've done, all produce the same result:
- Replaced cables with brand new ones I had lying around, still unpacked...
- Swapped around IX0 and IX1 as LAN/WAN, same problem. Only WAN drops out as shown on attached Video. No matter what I do with LAN, cannot replicate this behaviour
1.1. Disabled plugins (pfBlockerNG, Suricata, Arpwatch, Darkstat, LLDP) - Used a different NICs, Intel I350-T2 V2 and QNAP QXG-2G2T-I225, Same behaviour
- Used different ports on Modem, Same behaviour
- Connect through a TP Link Smart Switch, without and with Modem attached, Same behaviour
- Grabbed my spare Intel Z87 system, loaded the config up on this one (Only tried the X550-T2 Card), Same behaviour
I think I pretty much ruled out hardware as being the issue, including the Modem itself, as the same happens if the WAN is plugged into the switch alone.
What I see in Interfaces Dashboard is, WAN shows up correct, 1000base-T <Full-duplex> XXX.XXX.XXX.XXX, then immediately switched to autoselect, then back to showing the IP. And this loops forever.
With the switch attached, it was getting an IP from the switch of 192.168.0.10 (it has DHCP), and same behaviour. -
@bartkus05 said in Cable Internet and pfSense - Interface keeps dropping:
Connect through a TP Link Smart Switch, without and with Modem attached, Same behaviour
Hummm. so, it seems that to me to pfSense it-self pulling down its own WAN interface.
But this, ihmo, isn't done by 'software' :
which isn't this :
@bartkus05 said in Cable Internet and pfSense - Interface keeps dropping:
The interface keeps cycling, so connects, gets IP, disconnects and on and on it goes.
as obtaining a lease from an upstream DHCP server can't be done in "less then 10 ms" or so.
This interface flapping is way to fast.
Maybe (I'm not sure) the system (pfSense) can follow the event sequence and launches as much "start DHCP (WAN) client - and it bails out because no more interface - start DHCP (WAN) client - and it bails out because no more interface - start DHCP (WAN) client - and it bails out because no more interface - etcAfaik, only dpinger can reset the interface if pings send out don't come back anymore.
But this doesn't look like a dpinger action neither. It's disabled anyway.Your WAN uses 10 Gbit/sec ?
pfSense version ? Packages used ? -
First, I also want to say Thank You very much for trying to help me with this, it is much appreciated!
Here's a full Spec of Hardware, version of software, packages and settings, just to paint the full picture:
Hardware (I know bit overkill, its what I had lying around unused):
- AMD Ryzen 5 7600
- AsRock B650M-HDV
- 32GB Corsair Non-ECC 5200
- Intel X550T2BLK (Genuine, bought from Scan.co.uk) - Fan attached for active cooling
Modem / Router: Hitron Chita (Virgin Media Business) - Configured in Modem Mode
10Gbit LAN, 1Gbit WAN. I have a couple of NASes, Servers etc, hence why the 10Gbit LAN.
I have also tried setting the WAN to 1000base-T instead of autoselect, but its the same behaviour.
I have also tried with and without the loader.conf.local values set
Another log extract from today
Feb 10 14:12:16 syslogd kernel boot file is /boot/kernel/kernel
Feb 10 14:11:16 syslogd exiting on signal 15
Feb 10 14:11:16 check_reload_status 441 Reloading filter
Feb 10 14:11:16 check_reload_status 441 Starting packages
Feb 10 14:11:16 php-fpm 73153 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> xxx.xxx.xxx.xxx - Restarting packages.
Feb 10 14:11:14 php-fpm 398 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
Feb 10 14:11:13 php-fpm 73153 /rc.newwanip: Creating rrd update script
Feb 10 14:11:13 php-fpm 73153 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Feb 10 14:11:13 php-fpm 73153 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Feb 10 14:11:13 php-fpm 73153 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
Feb 10 14:11:13 check_reload_status 441 Linkup starting ix0
Feb 10 14:11:13 kernel ix0: link state changed to DOWN
Feb 10 14:11:13 check_reload_status 441 Restarting OpenVPN tunnels/interfaces
Feb 10 14:11:13 check_reload_status 441 Restarting IPsec tunnels
Feb 10 14:11:13 check_reload_status 441 updating dyndns vm
Feb 10 14:11:13 rc.gateway_alarm 78336 >>> Gateway alarm: vm (Addr:xxx.xxx.xxx.xxx Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
Feb 10 14:11:13 php-fpm 397 /rc.linkup: HOTPLUG: Configuring interface wan
Feb 10 14:11:13 php-fpm 397 /rc.linkup: DEVD Ethernet attached event for wan
Feb 10 14:11:13 php-fpm 397 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Feb 10 14:11:13 check_reload_status 441 Reloading filter
Feb 10 14:11:13 php-fpm 73153 /rc.newwanip: rc.newwanip: on (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: ix0).
Feb 10 14:11:13 php-fpm 73153 /rc.newwanip: rc.newwanip: Info: starting on ix0.
Feb 10 14:11:12 php-fpm 398 /rc.linkup: DEVD Ethernet detached event for wan
Feb 10 14:11:12 php-fpm 398 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Feb 10 14:11:12 check_reload_status 441 Reloading filter
Feb 10 14:11:12 check_reload_status 441 updating dyndns wan
Feb 10 14:11:12 check_reload_status 441 Restarting IPsec tunnels
Feb 10 14:11:12 php-fpm 73153 /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
Feb 10 14:11:12 check_reload_status 441 rc.newwanip starting ix0
Feb 10 14:11:11 check_reload_status 441 Linkup starting ix0
Feb 10 14:11:11 kernel ix0: link state changed to UP
Feb 10 14:11:08 php-fpm 397 /rc.interfaces_wan_configure: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var/run/dhclient.ix0.pid ix0 > /tmp/ix0_output 2> /tmp/ix0_error_output' returned exit code '1', the output was ''
Feb 10 14:11:07 php-fpm 398 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
Feb 10 14:11:07 kernel ix0: link state changed to DOWN
Feb 10 14:11:07 check_reload_status 441 Linkup starting ix0
Feb 10 14:11:06 check_reload_status 441 Restarting OpenVPN tunnels/interfaces
Feb 10 14:11:06 check_reload_status 441 Restarting IPsec tunnels
Feb 10 14:11:06 check_reload_status 441 updating dyndns vm
Feb 10 14:11:06 rc.gateway_alarm 1906 >>> Gateway alarm: vm (Addr:xxx.xxx.xxx.xxx Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
Feb 10 14:11:06 check_reload_status 441 Configuring interface wan
Feb 10 14:11:06 php-fpm 398 /rc.newwanip: rc.newwanip: Failed to update wan IP, restarting...
Feb 10 14:11:06 php-fpm 398 /rc.newwanip: rc.newwanip: on (IP address: ) (interface: WAN[wan]) (real interface: ix0).
Feb 10 14:11:06 php-fpm 398 /rc.newwanip: rc.newwanip: Info: starting on ix0.
Feb 10 14:11:06 php-fpm 73153 /rc.linkup: HOTPLUG: Configuring interface wan
Feb 10 14:11:06 php-fpm 73153 /rc.linkup: DEVD Ethernet attached event for wan
Feb 10 14:11:06 php-fpm 73153 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Feb 10 14:11:06 check_reload_status 441 Reloading filter
Feb 10 14:11:05 php-fpm 397 /rc.linkup: DEVD Ethernet detached event for wan
Feb 10 14:11:05 php-fpm 397 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Feb 10 14:11:05 check_reload_status 441 Reloading filter
Feb 10 14:11:05 check_reload_status 441 updating dyndns wan
Feb 10 14:11:05 check_reload_status 441 Restarting IPsec tunnels
Feb 10 14:11:05 php-fpm 398 /rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
Feb 10 14:11:05 check_reload_status 441 rc.newwanip starting ix0
Feb 10 14:11:05 kernel ix0: link state changed to UP
Feb 10 14:11:05 check_reload_status 441 Linkup starting ix0
Feb 10 14:11:01 kernel ix0: link state changed to DOWN
Feb 10 14:11:01 check_reload_status 441 Linkup starting ix0
Feb 10 14:11:01 php-fpm 398 /rc.linkup: HOTPLUG: Configuring interface wan
Feb 10 14:11:01 php-fpm 398 /rc.linkup: DEVD Ethernet attached event for wan
Feb 10 14:11:01 php-fpm 398 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Feb 10 14:11:00 kernel ix0: link state changed to UP
Feb 10 14:11:00 check_reload_status 441 Linkup starting ix0
Feb 10 14:10:57 php-fpm 397 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was '' -
Ok, I can see, while reading/following the log from bottom (old) to top (more recent) that some sort of "set up WAN, WAN hets pulled down, cycle is happening.
Somewhat strange to to IPv6 happening :
Feb 10 14:11:13 php-fpm 73153 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
while IPv6 is de activated.
You have none of these :
Feb 10 14:11:13 check_reload_status 441 Restarting OpenVPN tunnels/interfaces
Feb 10 14:11:13 check_reload_status 441 Restarting IPsec tunnels
Feb 10 14:11:13 check_reload_status 441 updating dyndns vm(no OpenVPN server or client, IPsec tunnels or dyndns) right ?
What - and now I'm really saying something I've nnever used before myself :
Suricata, as it installs its spy-hooks between the firewall; and the driver.
Is Suricata active (listens) on the WAN ?I don't know what lldpd is - not sure if I need it.
But the other package do
Other packages have any network activity, or they are, imho, harmless (as I use them also).All current system_patches proposed by Netgate are applied ?
Btw : X550T2BLK : should work ...
This :
you that never seen before scenario ?
Not related :
You don't trusted Internet's root server ?
(1111 and 1001 use Internet's root servers for you
)
What you can try - takes 10 minutes or so :
Make config backup.
Go console, and reset to default.
Whne it reboots, assign a DHCP WAN (is the default) and the default 192.168.1.1/24 LAN witn an active DHCP server - this is all being done at the console.
You are of course allowed to change the default pfSense password.
Nothing else. Don't add stuff - change the password and deatch the keyboard. Nothing else is needed anyway.
Question : you see the same behaviour ?When your tests are done, import the backed up config, reboot, and you're back at square one.
-
For IPv6, I had to disable it - it caused a lot of issues for some reason, with most devices not getting an internet connection. No idea why this was happening, as some worked and some didn't, especially mobile phones had an issue. Very bizarre.
Nope, no VPN servers, no dyndns etc
Suricata is only active on LAN. But I have only installed it only recently, last month as I have one sketchy IOT device and my switches don't support VLANs, so I cannot isolate it. Put through Suricata in Legacy mode to combat any "sketchy" action it might want to take.
Yes, all patches applied from Netgate, no custom patches added. lldpd I use on Omada to create a very pretty network map
Non-local gateway was one of the options I have tried to try and fix this (desperation lol)
My ISPs default DNS is absolutely shockingly bad. I found 1.1.1.1 to work the best hehe - if that's what you meant
Okay, I will give the scenario a try once everyone stops using the internet in the late evening.
Thanks.
-
@bartkus05 said in Cable Internet and pfSense - Interface keeps dropping:
My ISPs default DNS is absolutely shockingly bad. I found 1.1.1.1 to work the best hehe - if that's what you meant
No- He is saying don't add any DNS servers. Just use resolver, do not forward.
Like this-
so you see this-
-
Okay, so after a very long night and few coffees this morning, its now fully working without an issue, I can unplug the WAN cable and reconnect it, it jumps back up straight away. Rebooting the modem, the same, it waits for the link - once its there it reconnects and remains connected.
I first did a straight forward Factory Defaults reset, that didn't work. Next, did a fresh pfSense CE install, and that also has not helped, the same behaviour was repeating itself. I also tried the "forbidden fruit" fork, that didn't work either - same behaviour.
I still had the option to install pfSense plus from the days when I had the home lab license, so I installed that and the issue disappeared.... Rock solid. I set everything back up, all the packages I use and it still is working as expected. I have scanned through the changelogs / tickets on Redmine but couldn't see anything specific relating to this issue so my only conclusion is that something must have changed within FreeBSD itself.
How the logs look now, I have highlighted the event that never showed up before.
Feb 11 01:36:20 php-fpm 540 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Feb 11 01:36:20 php-fpm 540 /rc.linkup: DEVD Ethernet attached event for wan
Feb 11 01:36:20 php-fpm 540 /rc.linkup: HOTPLUG: Configuring interface wan
Feb 11 01:36:20 check_reload_status 653 rc.newwanip starting ix0
Feb 11 01:36:20 php-fpm 540 /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Feb 11 01:36:20 php-fpm 540 /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Feb 11 01:36:20 check_reload_status 653 Restarting IPsec tunnels
Feb 11 01:36:21 php-fpm 20277 /rc.start_packages: Restarting/Starting all packages.
Feb 11 01:36:21 php-fpm 20277 /rc.newwanip: rc.newwanip: Info: starting on ix0.
Feb 11 01:36:21 php-fpm 20277 /rc.newwanip: rc.newwanip: on (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: ix0).
Feb 11 01:36:22 php-fpm 63053 /rc.newwanip: Resyncing OpenVPN instances for interface LAN.
Feb 11 01:36:22 php-fpm 63053 /rc.newwanip: Creating rrd update script
Feb 11 01:36:23 php-fpm 20277 /rc.newwanip: Gateway, NONE AVAILABLE
Feb 11 01:36:23 php-fpm 20277 /rc.newwanip: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Feb 11 01:36:23 php-fpm 20277 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
Feb 11 01:36:24 php-fpm 63053 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 192.168.2.1 -> 192.168.2.1 - Restarting packages.
Feb 11 01:36:24 check_reload_status 653 Starting packages
Feb 11 01:36:24 check_reload_status 653 Reloading filter
Feb 11 01:36:24 check_reload_status 653 Reloading filter
Feb 11 01:36:25 php-fpm 3134 /rc.start_packages: Restarting/Starting all packages.
Feb 11 01:36:25 check_reload_status 653 updating dyndns wan
Feb 11 01:36:25 check_reload_status 653 Reloading filterI have also followed your suggestions and removed remote DNS servers, only using Resolver now.
Thank you for your help!
