DEVD Ethernet detached event for wan

digitaladdictions · Aug 20, 2021, 9:58 PM

I replaced my Ubiquiti ERPoe-5 with a Qotom-Q555G6-S05 running PFSense 2.5.2-RELEASE(amd64) last week. The Qotom has an Intel Core i5 process with 16GB of DDR4 RAM. I have installed the following packages:

darkstat 3.1.3_5
nmap 1.4.4_2
ntopng 0.8.13_10
pfBlockerNG-devel 3.0.0_16

Multiple times a day my WAN port goes down briefly disconnecting me from VPN and conference calls I am on. It always automatically fixes itself within seconds but it's a huge pain when on conference calls for work. Here is a snippet of the log. You can see the first message is an innocent enough arp update and then 12-3min later there is a check_reload_status igb5 and (WAN) goes down.

It just so happened that between the time I ordered the Qotom and the time it arrived we had some bad storms that caused an outage where I had to have Spectrum come out. At the same time they came out for the repair I upgraded my service with them from 100Mbps to 200Mbps which required a new modem. So there is that variable.

Aug 20 15:36:08 pfSense kernel: arp: 10.0.1.105 moved from dc:a6:32:36:ed:03 to dc:a6:32:36:ed:04 on igb1
Aug 20 15:48:54 pfSense check_reload_status[386]: Linkup starting igb5
Aug 20 15:48:54 pfSense kernel: igb5: link state changed to DOWN
Aug 20 15:48:55 pfSense php-fpm[347]: /rc.linkup: DEVD Ethernet detached event for wan
Aug 20 15:48:56 pfSense check_reload_status[386]: Reloading filter
Aug 20 15:48:57 pfSense check_reload_status[386]: Linkup starting igb5
Aug 20 15:48:57 pfSense kernel: igb5: link state changed to UP
Aug 20 15:48:58 pfSense php-fpm[46505]: /rc.linkup: DEVD Ethernet attached event for wan
Aug 20 15:48:58 pfSense php-fpm[46505]: /rc.linkup: HOTPLUG: Configuring interface wan
Aug 20 15:48:59 pfSense check_reload_status[386]: rc.newwanip starting igb5
Aug 20 15:48:59 pfSense php-fpm[46505]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. ''
Aug 20 15:48:59 pfSense check_reload_status[386]: Restarting ipsec tunnels
Aug 20 15:49:00 pfSense php-fpm[6707]: /rc.newwanip: rc.newwanip: Info: starting on igb5.
Aug 20 15:49:00 pfSense php-fpm[6707]: /rc.newwanip: rc.newwanip: on (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: igb5).
Aug 20 15:49:00 pfSense check_reload_status[386]: Reloading filter
Aug 20 15:49:03 pfSense check_reload_status[386]: updating dyndns wan
Aug 20 15:49:12 pfSense check_reload_status[386]: Reloading filter

I have tried adding the following system tuneables to no avail.

kern.ipc.nmbclusters = 1000000 	
dev.igb.0.eee_disabled = 1 	
dev.igb.1.eee_disabled = 1 	
dev.igb.2.eee_disabled = 1 	
dev.igb.3.eee_disabled = 1 	
dev.igb.4.eee_disabled = 1 	
dev.igb.5.eee_disabled = 1 	
hw.igb.0.fc = 0 	
hw.igb.1.fc = 0 	
hw.igb.2.fc = 0 	
hw.igb.3.fc = 0 	
hw.igb.4.fc = 0 	
hw.igb.5.fc = 0

sysctl net.inet.ip.intr_queue_drops shows 0 queue drops

[2.5.2-RELEASE][admin@pfSense.home.arpa]/root: sysctl net.inet.ip.intr_queue_drops
net.inet.ip.intr_queue_drops: 0

I tried changing the physical interface that was used for WAN. When I first set it up igb0 was WAN and I moved WAN to igb5. It only ever happens on the WAN interface never the LAN interface. I also swapped ethernet cables.

I tried switching back to the Ubiquiti for an evening and did not have any drops however I discovered the ERPoe-5 configured with 3 VLANs cant keep up with a 200Mbps internet connection which seems kind of suspicious. The CPU pegs at 100% around 150Mbps and can't get faster than that. It is frustrating since I can't reproduce the issue on demand so I don't know for sure it's not happening on the Ubiquiti but it does not seem to be. The Qotom running PFSense has no problems getting full speed at 220Mbps+.

digitaladdictions · Aug 21, 2021, 12:09 AM

I tried changing the gateway monitoring address to something further upstream 8.8.8.8. That did not help.

For the moment I am trying to disable the gateway monitoring action. I suspect restarting the WAN interface may be more disruptive than the brief packet loss half a dozen times a day for a second or two.

Gertjan · Aug 21, 2021, 8:57 AM

@digitaladdictions said in DEVD Ethernet detached event for wan:

I tried changing the physical interface that was used for WAN. When I first set it up igb0 was WAN and I moved WAN to igb5. It only ever happens on the WAN interface never the LAN interface. I also swapped ethernet cables.

Good plan.
You swapped interfaces and cables, so changes are great it's not the port, neither the cable.
As an even DOWN and UP is what I call a "carrier loss" (the signal on the ethernet wire) something ws detcetd as "went away", and "came back".
This could be the device - the NIC - that you have in front of your pfSense not playing well with the NIC pfSense uses : the Spectrum NIC ?

Btw : the WAN wire connection is auto negotiated to 1 Gbs, right ?
What happens when you force it to "1 Gbs" ?
Same thing for forcing it to "100 Mbits/sec".

stephenw10 · Aug 21, 2021, 1:43 PM

@digitaladdictions said in DEVD Ethernet detached event for wan:

Aug 20 15:48:54 pfSense kernel: igb5: link state changed to DOWN

Log entries like that are pretty much an indicator that it really did go down.

Try putting a switch in between the WAN and modem. That will eliminate negotiation issues as a test.

Steve

digitaladdictions · Aug 21, 2021, 7:13 PM

I was finally able to reproduce the issue on the old Ubiquiti router pinging my gateway overnight. So I guess it's time to give spectrum another call.

23428 packets transmitted, 23403 packets received, 0.1% packet loss
round-trip min/avg/max/stddev = 5.296/15.057/272.363/9.528 ms

When they came out 2 weeks ago and put the new modem in they replaced the amp I had with an attenuator. I tried taking the attenuator out just for another data point but did not seem to make much difference. I'll put that back in and put a switch inline between the modem and router to see if that helps reduce the impact in the interim. I don't have a ton of faith that they will be able to fix this and not just measure the signal strength and tell me everything looks fine from their end.

Gertjan · Aug 22, 2021, 6:44 AM

@digitaladdictions

Use this icon.
Enter "spectrum.

You'll find many (far to many) forum threads mentioning this spectrum modem.
Maybe you find something useful.
Or the reason to change ISP.

digitaladdictions · Aug 23, 2021, 10:36 PM

Got the answer I suspected from Spectrum. All the signal strengths check out fine and they can't replicate the issue in the 10min I was on the phone with them. But if it's of use to anyone searching on Qotom putting a dumb unmanaged switch between the spectrum modem and Qotom running PFSense seemed to completely resolve the issue.

Granted it's only been about 24hrs but I have not had a single packet drop since putting the switch in. I can live with the extra switch, lord knows I have enough of them floating around. Switching ISP is not an option as there is no serious competition. Maybe once AT&T fiber gets here.

y2raza · Oct 21, 2023, 12:25 AM

@digitaladdictions Hello, I am facing the same problem. Did putting a switch between the ONT and Pf appliance fix the packet loss problem?