Seemingly random ethernet link drops, usually at DHCP lease T1
-
Hi, I have something I think may be software related, but not 100% sure.
We have been getting random internet dropouts which I have now narrowed down to the link between my firewall and the ISP device.
Setup:
Home Internet: ISP provides wireless dish and a box inside my house with ethernet connection, no authentication needed as it uses IPOE. ISP also provides CGNAT.
Firewall: HP t620 thin client with Intel i350 4 port NIC running PfSense+.Problem:
Usually very close (within seconds) to DHCP lease renewal T1, the link between the ISP and firewall drops for a couple of seconds. This has also happened at other random times, but usually close to T1. I have confirmed this with camera surveillance footage showing the link light go out when the link down message appears in the log.What I have tried:
- Switching to a new ethernet cable
- Using an ethernet cable I haven't had issues with
- Swapping from igb0 to igb3
Further testing:
Next I will try plug a switch in between and monitor for dropouts between firewall/switch and ISP/switch to see if the problem still remains.Logs:
Attached is the DHCP log from PfSense.
Anyone have any further ideas?
Regards,
Andy -
Compare side by side the DHCP log and the system log at the same moment.
In the system log, are there any 'dpinger' (WAN monitoring) packet loss messages ?@Andy142 said in Seemingly random ethernet link drops, usually at DHCP lease T1:
howing the link light go out when the link down message appears in the log.
Your mission is, if you accept it, to determine which side of the cable, which interface = which device, pulls down the connection.
Put a switch between the pfSense WAN interface and the upstream ISP device and you have your answer. -
In the system logs it's just as if the cable was unplugged and plugged back in.
I currently have a switch in the middle waiting for another drop out, then I can review the camera footage and see which one it was. I have adjusted the gateway monitoring times to detect short dropouts now since I may not have a link change notification if its the ISP box.
-
@Andy142 said in Seemingly random ethernet link drops, usually at DHCP lease T1:
I can review the camera footage and see which one it was
The system log, pfSense side, will mention that event, even if it happened during one millisecond.
If there were no dpinger messages to signal packet loss then it would not activate it's action :
the dpinger action is : pulling down and up the WAN interface so it get re init.
You could even, for a while, check this "Disable Gateway Monitoring Action" so dpinger would only measure the ICMP latency, and nothing else.
Btw : the DHCP client on WAN gets an CGNAT IP. The one starting with "100."
Is this normal ? -
@Gertjan Thanks for the info, so from what I understand you're saying it could be better to turn off monitoring so any interruption wouldn't pull the interface down to re-init on the pfSense side?
The 100. IP is normal from my understanding. It's not PPPoE, its IPoE so maybe thats why it looks a little different?
-
So I have some results back.
Since adding a dumb switch in between the ISP box and pfSense I haven't had any link drops. My partner has also noted the internet has been much better compared to how it use to be.
However..... I am still getting drops on the gateway at the same times as I was seeing the link drop previously, just now the link stays active through the switch. There is also no alignment with the DHCP lease renewal atm so I think I can rule that out. This is really puzzling me. Looking at the switch lights during these events now shows they stay on.
Gateway Log:
System Log:
Gateway Monitoring settings:
-
@Andy142 said in Seemingly random ethernet link drops, usually at DHCP lease T1:
Since adding a dumb switch in between the ISP box and pfSense I haven't had any link drops.
Which means it was the ISP box pulling it's LAN port, the one connected to the pfSense WAN, down.
Power issue ?
Is this a modem type device, for example cable modems tend to do this to signal a 'bad uplink'.
If its a router type, I would consider that behavior as 'not normal'.Like a clock : every 30 minutes sharp you the connection drops packets (== the monitor pings ), to re establish 10 a 20 seconds later.
Welll... what to say ? "Not all ISPs are equal ^^" ?! -
@Gertjan I'm not sure if it's power related.... I have a security camera setup on the switch, pfSense NIC and ISP box. Before adding the switch int he middle the link lights would go out at the same time as the lag spike. Now with the switch in the middle the link lights for both stay active on all 3 devices. I even changed it out to a smart switch to see if that replicated the issue but the lights stayed active.
I've sent an email to the ISP with the logs for them to investigate.
-
ISP asked me to take some logs using pingPlotter. You can see a significant lag spike at hop 2.
-
Something missing : your avaible bandwidth.
After all, what happens with ICMP packets when the upstream or downstream "pipe" is full ? They get discard. And that shows up as a rising latency, or even packet loss, and it looks like the connection went 'bad'.
But its none of all this : it just queuing = delays. -
@Gertjan Is there a way I can get this? I wasn't using any internet at the time.
-
@Andy142 said in Seemingly random ethernet link drops, usually at DHCP lease T1:
I wasn't using any internet at the time
Like you as a person ? Maybe.
And your devices ? When a PC decides to upgrade to the last 2H24, or your phone has the newest OS version avaible, it won't ask you for permission, it just starts downloading.pfSense can show you what happened when :
If no traffic goes ever the WAN at the moment latency started to rise, then ... well, be ready to "never have the answer".
As ISPs normally do not reserve your 1 Gbit symmetrical (if that's what you have) just for you.
They will rent out the same bandwidth to many of their clients and then they hope you guys won't use their bandwidth all at the same time, because if that happens, while you doing nothing, cellmate will spike.
An ISP normally never admits that this happens ^^
Read the contract : somewhere you'll find written : 'connection speed is best effort'. -
@Gertjan Unfortunately no gigabit symmetric connection here. I'm lucky to get 120/8. I'll put a isolated fresh VM with updates disabled overnight to see what happens. Given the every 30 minutes nature of the problem I can't see this being a bandwidth issue.
Will see what the ISP comes back with, I have some good data so far.
Still interested to see why the ethernet link drops. I disabled gateway monitoring actions and tried again without the switch, still issues.
-
@Andy142 said in Seemingly random ethernet link drops, usually at DHCP lease T1:
Still interested to see why the ethernet link drops. I disabled gateway monitoring actions and tried again without the switch, still issues.
Test with the switch in the "WAN line" and "monitoring action" disabled.
If then still issues, stop looking : it's the ISP device or ISP connection. -
@Gertjan said in Seemingly random ethernet link drops, usually at DHCP lease T1:
n the "WAN line" and "monitoring
With or without monitoring actions enabled it's stable when the switch is in the middle.
-
Pretty solid proof then that the ISP device, connected to the pfSense WAN port took down the interface.
Afaik : reasons can be : if its a modem type device : they do this to signal down stream a data carrier loss.
Bad power.
Bad NIC.Most often, these ISP devices have also a GUI. It's time to have a look at, maybe there are details about the loss available.
