How To Disable/Enable Energy Efficient Ethernet (EEE)?

uplink · Jul 23, 2023, 5:48 PM

Just to be 100% clear, you're saying there is a typo in the description for the hw.igc.eee_setting? So, it should read "Disable Energy Efficient Ethernet" instead of "Enable Energy Efficient Ethernet"? If that's true then, yes, my EEE configuration should be all disabled then.

Should I bug in a bug ticket, or is there already on made for this?

Thanks!

stephenw10 · Jul 24, 2023, 4:27 PM

Yes I believe so:
https://github.com/freebsd/freebsd-src/blob/main/sys/dev/igc/if_igc.c#L266

It should probably be an upstream bug too. It doesn't look to have been fixed there yet.

Either way it's disabled by default in pfSense so isn't causing whatever you're seeing there.

Steve

lazyt · Mar 8, 2024, 5:31 PM

Hey @uplink - I think I'm experiencing the same issue: a cheap 4 port Inter 226i unit from Aliexpress works almost flawlessly, except for sporadic cutouts that last precisely 14 seconds almost every time. This has a nack of happening when on voip calls (slack/zoom/whatsapp/etc), although it can happen anytime. Here is a screenshot from Ping Doctor: the ping to the router seems fine, but the internet part is down. I also get dpinger issues:

send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 100.64.123.253 bind_addr 100.67.44.144 identifier "FIBER_PPPOE "

were you ever able to resolve this issue?

stephenw10 · Mar 8, 2024, 3:18 PM

Does it actually lose link? Anything logged in pfSense?

uplink · Mar 8, 2024, 5:31 PM

Hey @lazyt

Just real quick, the back story on this thread is that I was trying to figure out if EEE was possibly causing my random disconnects/drops which manifest in the pfSense logs as "hot plug" events. I went through a lot of troubleshooting with the members on this forum (many thanks), but to no avail. However, I would highly recommend that you read my post below and try out some of the troubleshooting there, maybe something will work for you.

My journey with the disconnects:
https://forum.netgate.com/topic/182384/help-random-hot-plug-events/7?_=1709917518893

So, to answer your question, I did not discover the root of the disconnects, but I did come up with a workaround. I ended up connecting my AliExpress router's 2.5gbe port to a port on my switch via a RJ45 SFP+ transceiver module that was capable of auto negotiating down to 2.5gbe. I've been running this configuration for nearly 6 months now without drops. It's not ideal since it occupies one of my precious SFP+ ports on my switch, but it works. Its funny you revived this old thread, because just last week I was thinking of trying to get this to work again.

Can I ask, are you seeing "hot plug" events in your pfSense logs when you experience these disconnects? What switch are you using?

lazyt · Mar 10, 2024, 9:19 AM

@uplink said in How To Disable/Enable Energy Efficient Ethernet (EEE)?:

are you seeing "hot plug" events in your pfSense logs

I don't think so - in which log did you see them?

lazyt · Mar 10, 2024, 9:22 AM

@stephenw10 I usually dpinger errors for all wan connections simultaneously (I have dual wan and an OpenVPN):

send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr xxx bind_addr xxx identifier "ADSL_PPPOE "

uplink · Mar 10, 2024, 7:01 PM

@lazyt

You can find the "hot plug" events in the system logs by navigating to: Status > System Logs > System > General

Below is an example of what you are looking for. You will see the link state change to "down" and a hot plug event. Keep in mind these logs only show you the last 500 events. So, usually you only have a day or two worth of logs. It's best to check right away when you experience a disconnect.

Aug 23 08:02:31	check_reload_status	436	Linkup starting igc1.40
Aug 23 08:02:31	check_reload_status	436	Linkup starting igc1.127
Aug 23 08:02:31	check_reload_status	436	Linkup starting igc1.20
Aug 23 08:02:31	check_reload_status	436	Linkup starting igc1.100
Aug 23 08:02:31	kernel		igc1.30: link state changed to UP
Aug 23 08:02:31	kernel		igc1.40: link state changed to UP
Aug 23 08:02:31	kernel		igc1.127: link state changed to UP
Aug 23 08:02:31	kernel		igc1.20: link state changed to UP
Aug 23 08:02:31	kernel		igc1.100: link state changed to UP
Aug 23 08:02:31	kernel		igc1: link state changed to UP
Aug 23 08:02:31	check_reload_status	436	Linkup starting igc1
Aug 23 08:02:28	php-fpm	21392	/rc.linkup: DEVD Ethernet detached event for opt5
Aug 23 08:02:28	php-fpm	21392	/rc.linkup: Hotplug event detected for GUEST(opt5) static IP address (4: 192.168.40.1)
Aug 23 08:02:28	php-fpm	10095	/rc.linkup: DEVD Ethernet detached event for opt7
Aug 23 08:02:28	php-fpm	10095	/rc.linkup: Hotplug event detected for VPN(opt7) static IP address (4: 192.168.127.1)
Aug 23 08:02:28	php-fpm	80452	/rc.linkup: DEVD Ethernet detached event for lan
Aug 23 08:02:28	php-fpm	80452	/rc.linkup: Hotplug event detected for LAN(lan) dynamic IP address (4: 192.168.10.1, 6: track6)
Aug 23 08:02:28	php-fpm	38979	/rc.linkup: DEVD Ethernet detached event for opt3
Aug 23 08:02:28	php-fpm	38979	/rc.linkup: Hotplug event detected for IOT(opt3) static IP address (4: 192.168.20.1)
Aug 23 08:02:28	php-fpm	70223	/rc.linkup: DEVD Ethernet detached event for opt6
Aug 23 08:02:28	php-fpm	29521	/rc.linkup: DEVD Ethernet detached event for opt4
Aug 23 08:02:28	php-fpm	29521	/rc.linkup: Hotplug event detected for WORK(opt4) static IP address (4: 192.168.30.1)
Aug 23 08:02:28	check_reload_status	436	Reloading filter
Aug 23 08:02:28	check_reload_status	436	Reloading filter
Aug 23 08:02:28	php-fpm	70223	/rc.linkup: Hotplug event detected for MGMT(opt6) static IP address (4: 192.168.100.1)
Aug 23 08:02:27	check_reload_status	436	Linkup starting igc1.30
Aug 23 08:02:27	check_reload_status	436	Linkup starting igc1.40
Aug 23 08:02:27	check_reload_status	436	Linkup starting igc1.127
Aug 23 08:02:27	check_reload_status	436	Linkup starting igc1.20
Aug 23 08:02:27	kernel		igc1.30: link state changed to DOWN
Aug 23 08:02:27	check_reload_status	436	Linkup starting igc1.100
Aug 23 08:02:27	kernel		igc1.40: link state changed to DOWN
Aug 23 08:02:27	kernel		igc1.127: link state changed to DOWN
Aug 23 08:02:27	kernel		igc1.20: link state changed to DOWN
Aug 23 08:02:27	kernel		igc1.100: link state changed to DOWN
Aug 23 08:02:27	kernel		igc1: link state changed to DOWN

lazyt · Mar 10, 2024, 7:06 PM

@uplink no, not seeing hot plug events

uplink · Mar 10, 2024, 7:33 PM

@lazyt

Interesting. OK, so at this point, maybe we should get a better understanding of what's going on. It sounds like you have a different issue than the one I was having. I was losing all network connectivity between my AliExpress router and my QNAP switch.

Since I'm not familiar with Ping Doctor, could you explain how you are testing your internet connection with it. It sounds like you are maybe running Ping Doctor on a PC and that pings your router's IP address? But, randomly the internet will go down for 14 second, even though you can still ping the router? Is that correct?

The 14 second duration seems suspect to me. Reminds me when I experienced something similar where an application running in pfSense would bring down my WAN interface while updating. However, this was easy to track down since it was happening at the same time everyday. You can read about it below:

https://forum.netgate.com/topic/177485/suricata-rules-update-drops-internet-connection-briefly/12?_=1710097873521

I understand that duration is 14 seconds, but is the frequency (when they occur) random or repetitive? Sometimes if can identify a pattern, it can give us a clue. If you feel comfortable, you might post a chunk of your system log (same location I mentioned previously) from the exact timeframe you had a disconnect. We might be able to spot something you missed.

lazyt · Mar 11, 2024, 8:43 AM

@uplink PingDoctor is an app that runs continuously, runs traceroute, and graphs the results. This allows me to visualize the point and duration of the outage.

Unfortunately, the outages don't seem to have a pattern, although they seem to happen more often during VoIP calls (Whatsapp/Zoom/Slack/etc.).

Here are the logs from the time of an outage:

Mar 11 06:50:14	rc.gateway_alarm	49054	>>> Gateway alarm: VPN_NAME_REMOVED-OPENVPN_VPNV4 (Addr:100.78.0.1 Alarm:1 RTT:5.134ms RTTsd:.201ms Loss:22%)
Mar 11 06:50:14	check_reload_status	439	updating dyndns VPN_NAME_REMOVED-OPENVPN_VPNV4
Mar 11 06:50:14	check_reload_status	439	Restarting IPsec tunnels
Mar 11 06:50:14	check_reload_status	439	Restarting OpenVPN tunnels/interfaces
Mar 11 06:50:14	check_reload_status	439	Reloading filter
Mar 11 06:50:14	rc.gateway_alarm	52480	>>> Gateway alarm: FIBER_PPPOE (Addr:FIBER_IP_ADDR_REMOVED Alarm:1 RTT:3.991ms RTTsd:4.251ms Loss:22%)
Mar 11 06:50:14	check_reload_status	439	updating dyndns FIBER_PPPOE
Mar 11 06:50:14	check_reload_status	439	Restarting IPsec tunnels
Mar 11 06:50:14	check_reload_status	439	Restarting OpenVPN tunnels/interfaces
Mar 11 06:50:14	check_reload_status	439	Reloading filter
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: MONITOR: FIBER_PPPOE has packet loss, omitting from routing group Failover
Mar 11 06:50:15	php-fpm	78891	FIBER_IP_ADDR_REMOVED|FIBER_GW_ADDR_REMOVED|FIBER_PPPOE|3.903ms|4.181ms|23%|down|highloss
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: Gateway, switch to: ADSL_PPPOE
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: Default gateway setting Interface ADSL_PPPOE Gateway as default.
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: Gateway, switch to: ADSL_PPPOE
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: Default gateway setting Interface ADSL_PPPOE Gateway as default.
Mar 11 06:50:15	php-fpm	62663	/rc.filter_configure_sync: Gateway, switch to: ADSL_PPPOE
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPN_NAME_REMOVED-OPENVPN_VPNV4.
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use FIBER_PPPOE.
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: OpenVPN: Resync client1 VPN_NAME_REMOVED-
Mar 11 06:50:15	php-fpm	14462	OpenVPN terminate old pid: 76667
Mar 11 06:50:15	kernel		ovpnc1: link state changed to DOWN
Mar 11 06:50:15	check_reload_status	439	Reloading filter
Mar 11 06:50:16	php-fpm	62663	/rc.filter_configure_sync: GW States: Gateway is down but its IP address cannot be located. Skipping state kill.: VPN_NAME_REMOVED-OPENVPN_VPNV4
Mar 11 06:50:16	php-fpm	14462	OpenVPN PID written: 8169
Mar 11 06:50:17	kernel		ovpnc1: link state changed to UP
Mar 11 06:50:17	check_reload_status	439	rc.newwanip starting ovpnc1
Mar 11 06:50:18	php-fpm	30069	/rc.newwanip: rc.newwanip: Info: starting on ovpnc1.
Mar 11 06:50:18	php-fpm	30069	/rc.newwanip: rc.newwanip: on (IP address: 100.78.0.61) (interface: VPN_NAME_REMOVED-OPENVPN[opt2]) (real interface: ovpnc1).
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: MONITOR: FIBER_PPPOE is available now, adding to routing group Failover
Mar 11 06:50:20	php-fpm	30069	FIBER_IP_ADDR_REMOVED|FIBER_GW_ADDR_REMOVED|FIBER_PPPOE|5.186ms|2.301ms|0.0%|online|none
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: Gateway, switch to: FIBER_PPPOE
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: Default gateway setting Interface FIBER_PPPOE Gateway as default.
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: IP Address has changed, killing states on former IP Address 100.78.7.90.
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: Creating rrd update script
Mar 11 06:50:22	php-fpm	30069	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 100.78.7.90 -> 100.78.0.61 - Restarting packages.
Mar 11 06:50:22	check_reload_status	439	Starting packages
Mar 11 06:50:22	check_reload_status	439	Reloading filter
zzz

stephenw10 · Mar 11, 2024, 1:53 PM

That looks like it just starts dropping packets on the fibre WAN and switches to the DSL WAN. That is the expected behaviour in that situation.

The first thing I would do there is change the monitor IP on the Fiber WAN to something external. Using the default WAN gateway IP can produce bad data. The gateway may not respond to pings when loaded. Or at all for that matter, though here it clearly does most of the time.

lazyt · Mar 11, 2024, 2:05 PM

@stephenw10 I had it on something external for a long time. Will switch it back and try again. Also, I disabled monitoring totally for a bit - also didn't help.

stephenw10 · Mar 11, 2024, 2:18 PM

With monitoring disabled then did the PPPoE session fail entirely and then reconnect? That isn't what's shown in the above log.

lazyt · Mar 11, 2024, 2:30 PM

@stephenw10 No - I currently do have a monitor in place. I was just commenting that in the past I had tried removing the monitor and I still experienced the same issue

stephenw10 · Mar 11, 2024, 3:09 PM

Ok well we'd need to see what is logged in that situation then since without monitoring it would not throw a gateway alarm and that's what's causing the issues you're seeing now. Assuming there is no loss of link logged before that which is omitted from the logs you posted.

lazyt · Mar 12, 2024, 10:31 AM

@stephenw10 there is a small sinppet here, happy to share anything else that can help

stephenw10 · Mar 12, 2024, 3:00 PM

Those logs show the first thing logged is an alarm from the gateway monitoring. So clearly at that time it was running the gateway monitor.

From what we can see there the NIC did not lose link. The WAN just started dropping packets. Unless there were entries before that showing it did lose link?

So it could simply be that the WAN is lossy under load and the gateway monitoring values should be adjusted to match that so alarms are not triggered. Or just disabled.

lazyt · Mar 12, 2024, 5:10 PM

@stephenw10

@stephenw10 said in How To Disable/Enable Energy Efficient Ethernet (EEE)?:

the WAN is lossy under load

a. I didn't notice any load (CPU or mem)
b. what would cause both WAN's to drop at exactly the same time?!

@stephenw10 said in How To Disable/Enable Energy Efficient Ethernet (EEE)?:

gateway monitoring values should be adjusted

I'm happy to try - what would you suggest?

stephenw10 · Mar 12, 2024, 5:22 PM

The WANs share the same NIC? Using VLANs? Same ISP? Something upstream maybe?

Try setting packet loss at 50%. Anything hitting that is almost certainly broken.