DEVD Ethernet detached event for wan

digitaladdictions

I replaced my Ubiquiti ERPoe-5 with a Qotom-Q555G6-S05 running PFSense 2.5.2-RELEASE(amd64) last week. The Qotom has an Intel Core i5 process with 16GB of DDR4 RAM. I have installed the following packages:

darkstat 3.1.3_5
nmap 1.4.4_2
ntopng 0.8.13_10
pfBlockerNG-devel 3.0.0_16

Multiple times a day my WAN port goes down briefly disconnecting me from VPN and conference calls I am on. It always automatically fixes itself within seconds but it's a huge pain when on conference calls for work. Here is a snippet of the log. You can see the first message is an innocent enough arp update and then 12-3min later there is a check_reload_status igb5 and (WAN) goes down.

It just so happened that between the time I ordered the Qotom and the time it arrived we had some bad storms that caused an outage where I had to have Spectrum come out. At the same time they came out for the repair I upgraded my service with them from 100Mbps to 200Mbps which required a new modem. So there is that variable.

Aug 20 15:36:08 pfSense kernel: arp: 10.0.1.105 moved from dc:a6:32:36:ed:03 to dc:a6:32:36:ed:04 on igb1
Aug 20 15:48:54 pfSense check_reload_status[386]: Linkup starting igb5
Aug 20 15:48:54 pfSense kernel: igb5: link state changed to DOWN
Aug 20 15:48:55 pfSense php-fpm[347]: /rc.linkup: DEVD Ethernet detached event for wan
Aug 20 15:48:56 pfSense check_reload_status[386]: Reloading filter
Aug 20 15:48:57 pfSense check_reload_status[386]: Linkup starting igb5
Aug 20 15:48:57 pfSense kernel: igb5: link state changed to UP
Aug 20 15:48:58 pfSense php-fpm[46505]: /rc.linkup: DEVD Ethernet attached event for wan
Aug 20 15:48:58 pfSense php-fpm[46505]: /rc.linkup: HOTPLUG: Configuring interface wan
Aug 20 15:48:59 pfSense check_reload_status[386]: rc.newwanip starting igb5
Aug 20 15:48:59 pfSense php-fpm[46505]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. ''
Aug 20 15:48:59 pfSense check_reload_status[386]: Restarting ipsec tunnels
Aug 20 15:49:00 pfSense php-fpm[6707]: /rc.newwanip: rc.newwanip: Info: starting on igb5.
Aug 20 15:49:00 pfSense php-fpm[6707]: /rc.newwanip: rc.newwanip: on (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: igb5).
Aug 20 15:49:00 pfSense check_reload_status[386]: Reloading filter
Aug 20 15:49:03 pfSense check_reload_status[386]: updating dyndns wan
Aug 20 15:49:12 pfSense check_reload_status[386]: Reloading filter

I have tried adding the following system tuneables to no avail.

kern.ipc.nmbclusters = 1000000 	
dev.igb.0.eee_disabled = 1 	
dev.igb.1.eee_disabled = 1 	
dev.igb.2.eee_disabled = 1 	
dev.igb.3.eee_disabled = 1 	
dev.igb.4.eee_disabled = 1 	
dev.igb.5.eee_disabled = 1 	
hw.igb.0.fc = 0 	
hw.igb.1.fc = 0 	
hw.igb.2.fc = 0 	
hw.igb.3.fc = 0 	
hw.igb.4.fc = 0 	
hw.igb.5.fc = 0

sysctl net.inet.ip.intr_queue_drops shows 0 queue drops

[2.5.2-RELEASE][admin@pfSense.home.arpa]/root: sysctl net.inet.ip.intr_queue_drops
net.inet.ip.intr_queue_drops: 0

I tried changing the physical interface that was used for WAN. When I first set it up igb0 was WAN and I moved WAN to igb5. It only ever happens on the WAN interface never the LAN interface. I also swapped ethernet cables.

I tried switching back to the Ubiquiti for an evening and did not have any drops however I discovered the ERPoe-5 configured with 3 VLANs cant keep up with a 200Mbps internet connection which seems kind of suspicious. The CPU pegs at 100% around 150Mbps and can't get faster than that. It is frustrating since I can't reproduce the issue on demand so I don't know for sure it's not happening on the Ubiquiti but it does not seem to be. The Qotom running PFSense has no problems getting full speed at 220Mbps+.

digitaladdictions

I tried changing the gateway monitoring address to something further upstream 8.8.8.8. That did not help.

For the moment I am trying to disable the gateway monitoring action. I suspect restarting the WAN interface may be more disruptive than the brief packet loss half a dozen times a day for a second or two.

Gertjan

@digitaladdictions said in DEVD Ethernet detached event for wan:

I tried changing the physical interface that was used for WAN. When I first set it up igb0 was WAN and I moved WAN to igb5. It only ever happens on the WAN interface never the LAN interface. I also swapped ethernet cables.

Good plan.
You swapped interfaces and cables, so changes are great it's not the port, neither the cable.
As an even DOWN and UP is what I call a "carrier loss" (the signal on the ethernet wire) something ws detcetd as "went away", and "came back".
This could be the device - the NIC - that you have in front of your pfSense not playing well with the NIC pfSense uses : the Spectrum NIC ?

Btw : the WAN wire connection is auto negotiated to 1 Gbs, right ?
What happens when you force it to "1 Gbs" ?
Same thing for forcing it to "100 Mbits/sec".

stephenw10

@digitaladdictions said in DEVD Ethernet detached event for wan:

Aug 20 15:48:54 pfSense kernel: igb5: link state changed to DOWN

Log entries like that are pretty much an indicator that it really did go down.

Try putting a switch in between the WAN and modem. That will eliminate negotiation issues as a test.

Steve

digitaladdictions

I was finally able to reproduce the issue on the old Ubiquiti router pinging my gateway overnight. So I guess it's time to give spectrum another call.

23428 packets transmitted, 23403 packets received, 0.1% packet loss
round-trip min/avg/max/stddev = 5.296/15.057/272.363/9.528 ms

When they came out 2 weeks ago and put the new modem in they replaced the amp I had with an attenuator. I tried taking the attenuator out just for another data point but did not seem to make much difference. I'll put that back in and put a switch inline between the modem and router to see if that helps reduce the impact in the interim. I don't have a ton of faith that they will be able to fix this and not just measure the signal strength and tell me everything looks fine from their end.

Gertjan

@digitaladdictions

Use this icon.
Enter "spectrum.

You'll find many (far to many) forum threads mentioning this spectrum modem.
Maybe you find something useful.
Or the reason to change ISP.

digitaladdictions

Got the answer I suspected from Spectrum. All the signal strengths check out fine and they can't replicate the issue in the 10min I was on the phone with them. But if it's of use to anyone searching on Qotom putting a dumb unmanaged switch between the spectrum modem and Qotom running PFSense seemed to completely resolve the issue.

Granted it's only been about 24hrs but I have not had a single packet drop since putting the switch in. I can live with the extra switch, lord knows I have enough of them floating around. Switching ISP is not an option as there is no serious competition. Maybe once AT&T fiber gets here.

y2raza

@digitaladdictions Hello, I am facing the same problem. Did putting a switch between the ONT and Pf appliance fix the packet loss problem?