How To Disable/Enable Energy Efficient Ethernet (EEE)?
-
Hey @uplink - I think I'm experiencing the same issue: a cheap 4 port Inter 226i unit from Aliexpress works almost flawlessly, except for sporadic cutouts that last precisely 14 seconds almost every time. This has a nack of happening when on voip calls (slack/zoom/whatsapp/etc), although it can happen anytime. Here is a screenshot from Ping Doctor: the ping to the router seems fine, but the internet part is down. I also get dpinger issues:
send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 100.64.123.253 bind_addr 100.67.44.144 identifier "FIBER_PPPOE "were you ever able to resolve this issue?
-
Does it actually lose link? Anything logged in pfSense?
-
Hey @lazyt
Just real quick, the back story on this thread is that I was trying to figure out if EEE was possibly causing my random disconnects/drops which manifest in the pfSense logs as "hot plug" events. I went through a lot of troubleshooting with the members on this forum (many thanks), but to no avail. However, I would highly recommend that you read my post below and try out some of the troubleshooting there, maybe something will work for you.
My journey with the disconnects:
https://forum.netgate.com/topic/182384/help-random-hot-plug-events/7?_=1709917518893So, to answer your question, I did not discover the root of the disconnects, but I did come up with a workaround. I ended up connecting my AliExpress router's 2.5gbe port to a port on my switch via a RJ45 SFP+ transceiver module that was capable of auto negotiating down to 2.5gbe. I've been running this configuration for nearly 6 months now without drops. It's not ideal since it occupies one of my precious SFP+ ports on my switch, but it works. Its funny you revived this old thread, because just last week I was thinking of trying to get this to work again.
Can I ask, are you seeing "hot plug" events in your pfSense logs when you experience these disconnects? What switch are you using?
-
@uplink said in How To Disable/Enable Energy Efficient Ethernet (EEE)?:
are you seeing "hot plug" events in your pfSense logs
I don't think so - in which log did you see them?
-
@stephenw10 I usually
dpingererrors for all wan connections simultaneously (I have dual wan and an OpenVPN):send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr xxx bind_addr xxx identifier "ADSL_PPPOE " -
You can find the "hot plug" events in the system logs by navigating to: Status > System Logs > System > General
Below is an example of what you are looking for. You will see the link state change to "down" and a hot plug event. Keep in mind these logs only show you the last 500 events. So, usually you only have a day or two worth of logs. It's best to check right away when you experience a disconnect.
Aug 23 08:02:31 check_reload_status 436 Linkup starting igc1.40 Aug 23 08:02:31 check_reload_status 436 Linkup starting igc1.127 Aug 23 08:02:31 check_reload_status 436 Linkup starting igc1.20 Aug 23 08:02:31 check_reload_status 436 Linkup starting igc1.100 Aug 23 08:02:31 kernel igc1.30: link state changed to UP Aug 23 08:02:31 kernel igc1.40: link state changed to UP Aug 23 08:02:31 kernel igc1.127: link state changed to UP Aug 23 08:02:31 kernel igc1.20: link state changed to UP Aug 23 08:02:31 kernel igc1.100: link state changed to UP Aug 23 08:02:31 kernel igc1: link state changed to UP Aug 23 08:02:31 check_reload_status 436 Linkup starting igc1 Aug 23 08:02:28 php-fpm 21392 /rc.linkup: DEVD Ethernet detached event for opt5 Aug 23 08:02:28 php-fpm 21392 /rc.linkup: Hotplug event detected for GUEST(opt5) static IP address (4: 192.168.40.1) Aug 23 08:02:28 php-fpm 10095 /rc.linkup: DEVD Ethernet detached event for opt7 Aug 23 08:02:28 php-fpm 10095 /rc.linkup: Hotplug event detected for VPN(opt7) static IP address (4: 192.168.127.1) Aug 23 08:02:28 php-fpm 80452 /rc.linkup: DEVD Ethernet detached event for lan Aug 23 08:02:28 php-fpm 80452 /rc.linkup: Hotplug event detected for LAN(lan) dynamic IP address (4: 192.168.10.1, 6: track6) Aug 23 08:02:28 php-fpm 38979 /rc.linkup: DEVD Ethernet detached event for opt3 Aug 23 08:02:28 php-fpm 38979 /rc.linkup: Hotplug event detected for IOT(opt3) static IP address (4: 192.168.20.1) Aug 23 08:02:28 php-fpm 70223 /rc.linkup: DEVD Ethernet detached event for opt6 Aug 23 08:02:28 php-fpm 29521 /rc.linkup: DEVD Ethernet detached event for opt4 Aug 23 08:02:28 php-fpm 29521 /rc.linkup: Hotplug event detected for WORK(opt4) static IP address (4: 192.168.30.1) Aug 23 08:02:28 check_reload_status 436 Reloading filter Aug 23 08:02:28 check_reload_status 436 Reloading filter Aug 23 08:02:28 php-fpm 70223 /rc.linkup: Hotplug event detected for MGMT(opt6) static IP address (4: 192.168.100.1) Aug 23 08:02:27 check_reload_status 436 Linkup starting igc1.30 Aug 23 08:02:27 check_reload_status 436 Linkup starting igc1.40 Aug 23 08:02:27 check_reload_status 436 Linkup starting igc1.127 Aug 23 08:02:27 check_reload_status 436 Linkup starting igc1.20 Aug 23 08:02:27 kernel igc1.30: link state changed to DOWN Aug 23 08:02:27 check_reload_status 436 Linkup starting igc1.100 Aug 23 08:02:27 kernel igc1.40: link state changed to DOWN Aug 23 08:02:27 kernel igc1.127: link state changed to DOWN Aug 23 08:02:27 kernel igc1.20: link state changed to DOWN Aug 23 08:02:27 kernel igc1.100: link state changed to DOWN Aug 23 08:02:27 kernel igc1: link state changed to DOWN -
@uplink no, not seeing hot plug events
-
Interesting. OK, so at this point, maybe we should get a better understanding of what's going on. It sounds like you have a different issue than the one I was having. I was losing all network connectivity between my AliExpress router and my QNAP switch.
Since I'm not familiar with Ping Doctor, could you explain how you are testing your internet connection with it. It sounds like you are maybe running Ping Doctor on a PC and that pings your router's IP address? But, randomly the internet will go down for 14 second, even though you can still ping the router? Is that correct?
The 14 second duration seems suspect to me. Reminds me when I experienced something similar where an application running in pfSense would bring down my WAN interface while updating. However, this was easy to track down since it was happening at the same time everyday. You can read about it below:
https://forum.netgate.com/topic/177485/suricata-rules-update-drops-internet-connection-briefly/12?_=1710097873521
I understand that duration is 14 seconds, but is the frequency (when they occur) random or repetitive? Sometimes if can identify a pattern, it can give us a clue. If you feel comfortable, you might post a chunk of your system log (same location I mentioned previously) from the exact timeframe you had a disconnect. We might be able to spot something you missed.
-
@uplink PingDoctor is an app that runs continuously, runs traceroute, and graphs the results. This allows me to visualize the point and duration of the outage.
Unfortunately, the outages don't seem to have a pattern, although they seem to happen more often during VoIP calls (Whatsapp/Zoom/Slack/etc.).
Here are the logs from the time of an outage:
Mar 11 06:50:14 rc.gateway_alarm 49054 >>> Gateway alarm: VPN_NAME_REMOVED-OPENVPN_VPNV4 (Addr:100.78.0.1 Alarm:1 RTT:5.134ms RTTsd:.201ms Loss:22%) Mar 11 06:50:14 check_reload_status 439 updating dyndns VPN_NAME_REMOVED-OPENVPN_VPNV4 Mar 11 06:50:14 check_reload_status 439 Restarting IPsec tunnels Mar 11 06:50:14 check_reload_status 439 Restarting OpenVPN tunnels/interfaces Mar 11 06:50:14 check_reload_status 439 Reloading filter Mar 11 06:50:14 rc.gateway_alarm 52480 >>> Gateway alarm: FIBER_PPPOE (Addr:FIBER_IP_ADDR_REMOVED Alarm:1 RTT:3.991ms RTTsd:4.251ms Loss:22%) Mar 11 06:50:14 check_reload_status 439 updating dyndns FIBER_PPPOE Mar 11 06:50:14 check_reload_status 439 Restarting IPsec tunnels Mar 11 06:50:14 check_reload_status 439 Restarting OpenVPN tunnels/interfaces Mar 11 06:50:14 check_reload_status 439 Reloading filter Mar 11 06:50:15 php-fpm 78891 /rc.openvpn: MONITOR: FIBER_PPPOE has packet loss, omitting from routing group Failover Mar 11 06:50:15 php-fpm 78891 FIBER_IP_ADDR_REMOVED|FIBER_GW_ADDR_REMOVED|FIBER_PPPOE|3.903ms|4.181ms|23%|down|highloss Mar 11 06:50:15 php-fpm 78891 /rc.openvpn: Gateway, switch to: ADSL_PPPOE Mar 11 06:50:15 php-fpm 78891 /rc.openvpn: Default gateway setting Interface ADSL_PPPOE Gateway as default. Mar 11 06:50:15 php-fpm 14462 /rc.openvpn: Gateway, switch to: ADSL_PPPOE Mar 11 06:50:15 php-fpm 14462 /rc.openvpn: Default gateway setting Interface ADSL_PPPOE Gateway as default. Mar 11 06:50:15 php-fpm 62663 /rc.filter_configure_sync: Gateway, switch to: ADSL_PPPOE Mar 11 06:50:15 php-fpm 78891 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Mar 11 06:50:15 php-fpm 78891 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPN_NAME_REMOVED-OPENVPN_VPNV4. Mar 11 06:50:15 php-fpm 14462 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Mar 11 06:50:15 php-fpm 14462 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use FIBER_PPPOE. Mar 11 06:50:15 php-fpm 14462 /rc.openvpn: OpenVPN: Resync client1 VPN_NAME_REMOVED- Mar 11 06:50:15 php-fpm 14462 OpenVPN terminate old pid: 76667 Mar 11 06:50:15 kernel ovpnc1: link state changed to DOWN Mar 11 06:50:15 check_reload_status 439 Reloading filter Mar 11 06:50:16 php-fpm 62663 /rc.filter_configure_sync: GW States: Gateway is down but its IP address cannot be located. Skipping state kill.: VPN_NAME_REMOVED-OPENVPN_VPNV4 Mar 11 06:50:16 php-fpm 14462 OpenVPN PID written: 8169 Mar 11 06:50:17 kernel ovpnc1: link state changed to UP Mar 11 06:50:17 check_reload_status 439 rc.newwanip starting ovpnc1 Mar 11 06:50:18 php-fpm 30069 /rc.newwanip: rc.newwanip: Info: starting on ovpnc1. Mar 11 06:50:18 php-fpm 30069 /rc.newwanip: rc.newwanip: on (IP address: 100.78.0.61) (interface: VPN_NAME_REMOVED-OPENVPN[opt2]) (real interface: ovpnc1). Mar 11 06:50:20 php-fpm 30069 /rc.newwanip: MONITOR: FIBER_PPPOE is available now, adding to routing group Failover Mar 11 06:50:20 php-fpm 30069 FIBER_IP_ADDR_REMOVED|FIBER_GW_ADDR_REMOVED|FIBER_PPPOE|5.186ms|2.301ms|0.0%|online|none Mar 11 06:50:20 php-fpm 30069 /rc.newwanip: Gateway, switch to: FIBER_PPPOE Mar 11 06:50:20 php-fpm 30069 /rc.newwanip: Default gateway setting Interface FIBER_PPPOE Gateway as default. Mar 11 06:50:20 php-fpm 30069 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Mar 11 06:50:20 php-fpm 30069 /rc.newwanip: IP Address has changed, killing states on former IP Address 100.78.7.90. Mar 11 06:50:20 php-fpm 30069 /rc.newwanip: Creating rrd update script Mar 11 06:50:22 php-fpm 30069 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 100.78.7.90 -> 100.78.0.61 - Restarting packages. Mar 11 06:50:22 check_reload_status 439 Starting packages Mar 11 06:50:22 check_reload_status 439 Reloading filter zzz -
That looks like it just starts dropping packets on the fibre WAN and switches to the DSL WAN. That is the expected behaviour in that situation.
The first thing I would do there is change the monitor IP on the Fiber WAN to something external. Using the default WAN gateway IP can produce bad data. The gateway may not respond to pings when loaded. Or at all for that matter, though here it clearly does most of the time.
-
@stephenw10 I had it on something external for a long time. Will switch it back and try again. Also, I disabled monitoring totally for a bit - also didn't help.
-
With monitoring disabled then did the PPPoE session fail entirely and then reconnect? That isn't what's shown in the above log.
-
@stephenw10 No - I currently do have a monitor in place. I was just commenting that in the past I had tried removing the monitor and I still experienced the same issue
-
Ok well we'd need to see what is logged in that situation then since without monitoring it would not throw a gateway alarm and that's what's causing the issues you're seeing now. Assuming there is no loss of link logged before that which is omitted from the logs you posted.
-
@stephenw10 there is a small sinppet here, happy to share anything else that can help
-
Those logs show the first thing logged is an alarm from the gateway monitoring. So clearly at that time it was running the gateway monitor.
From what we can see there the NIC did not lose link. The WAN just started dropping packets. Unless there were entries before that showing it did lose link?
So it could simply be that the WAN is lossy under load and the gateway monitoring values should be adjusted to match that so alarms are not triggered. Or just disabled.
-
@stephenw10 said in How To Disable/Enable Energy Efficient Ethernet (EEE)?:
the WAN is lossy under load
a. I didn't notice any load (CPU or mem)
b. what would cause both WAN's to drop at exactly the same time?!@stephenw10 said in How To Disable/Enable Energy Efficient Ethernet (EEE)?:
gateway monitoring values should be adjusted
I'm happy to try - what would you suggest?
-
The WANs share the same NIC? Using VLANs? Same ISP? Something upstream maybe?
Try setting packet loss at 50%. Anything hitting that is almost certainly broken.
-
@stephenw10 said in How To Disable/Enable Energy Efficient Ethernet (EEE)?:
Try setting packet loss at 50%. Anything hitting that is almost certainly broken.
I did, same intermittent dropouts
-
OK and still no link losses logged? Just gateway alarms?
What about the interface stats in Status > Interfaces? Does it show errors, dropped packets? How about in the output of
netstat -idirectly?
