Aliexpress "Findarling" I226-V sporadic drops (~15 seconds ea.)

lazyt

I have this mini pc from aliexpress, which is working basically just fine except for sporadic disconnects. sysctl dev.igc.0 | grep %desc says this device is an Intel(R) Ethernet Controller I226-V

Whats interesting is:

disconnects are almost always exactly 15 seconds
igc0 always continues working - only igc1 & igc2, my two wan connections, seem to drop.
Seems to happen more often on voip calls (slack/whatsapp/zoom/etc), which is understandably quite annoying

Here is a visual from PingDoctor showing the connection to router as fine, but the internet is down:

Screenshot 2024-03-08 at 1.18.37 PM.png

The logs show this for both wan connections

send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 100.64.123.253 bind_addr 100.67.44.144 identifier "FIBER_PPPOE "

I've disabled:

Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading

I've tried tunning as listed here, but that didn't help.

Is there a solution for this hardware or do I need to replace it?

stephenw10

Does it actually log a link loss or just the gateway not responding?

What does pciconf -lv show the hardware revision as?

Steve

TheNarc

@lazyt @stephenw10 I've got a very similar (though not identical) box that's been up for a few weeks now and have not observed this problem. So in case it serves as a useful point of comparison for you, here's my pciconf -lv output (only included for one of the 4 interfaces since they're identical):

igc0@pci0:1:0:0:	class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I226-V'
    class      = network
    subclass   = ethernet

Edit to add: I'm on a 500/20 cable modem connection and my LAN is gigabit, so none of my I-226s are linking at 2.5Gbps. To my recollection, the grief that these and some versions of the I-225 chipsets have caused mostly if not completely had to do with linking at 2.5.

stephenw10

Yeah, I've not seen any such issues on our own boxes using i225 or i226 NICs. It feels like there is more to the reported disconnection issues than the NIC chip alone.

lazyt

@stephenw10 said in Aliexpress "Findarling" I226-V sporadic drops (~15 seconds ea.):

What does pciconf -lv show the hardware revision as?

igc0@pci0:2:0:0:	class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I226-V'
    class      = network
    subclass   = ethernet

lazyt

@stephenw10

Happy to provide more logs, if you can point out which would be helpful. In the interim, I'll start with this: I have dual wan and a VPN connections, and they always drop at exactly the same time - leading me to belive that this isn't an isp specific issue

Mar 8 22:10:00	dpinger	60499	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr x.x.x.x bind_addr x.x.x.x identifier "ADSL_PPPOE "
Mar 8 22:10:00	dpinger	60590	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr x.x.x.x bind_addr x.x.x.x identifier "NETFREEOPENVPN_VPNV4 "
Mar 8 22:10:00	dpinger	61123	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr x.x.x.x bind_addr x.x.x.x identifier "FIBER_PPPOE "

lazyt

@TheNarc said in Aliexpress "Findarling" I226-V sporadic drops (~15 seconds ea.):

mostly if not completely had to do with linking at 2.5.

Is there some way to force the speed to be lower? Being a PPPoE connection, the interface stage doesn't offer the option to set the ethernet speed.

stephenw10

Ok so that points to some common cause. That looks like the gateway log. What is shown in the main system logs at that time?

You can assign the parent interface separately and set a link speed there.

lazyt

@stephenw10 said in Aliexpress "Findarling" I226-V sporadic drops (~15 seconds ea.):

You can assign the parent interface separately and set a link speed there.

How? I'm not sure what you mean

stephenw10

I mean if you have an interface pppoe0 that configured as WAN and that running on igc0 you can also assign and enable igc0 dircetly as a different interface. The link speed/duplex can be set there.
However be aware that the igc NICs can only use autonegotiate. If you set 1G there it just limits the available negotiation options to 1G only. So it will fail to link to a device that is actually set to a fixed speed and will not negotiate. It should work for you here though.

lazyt

@stephenw10 oh, cool - TIL! I lowered it to 1gb, will let it run a bit to see how that affects things.

lazyt

@lazyt said in Aliexpress "Findarling" I226-V sporadic drops (~15 seconds ea.):

@stephenw10

@stephenw10 - setting it to 1gb did not help (might have made it worse)

stephenw10

Ok so the WAN still drops out? Both WANs at the same time?

What is shown in the system log when that happens?

lazyt

@stephenw10 here's a sample:

Mar 11 06:50:14	rc.gateway_alarm	49054	>>> Gateway alarm: VPN_NAME_REMOVED-OPENVPN_VPNV4 (Addr:100.78.0.1 Alarm:1 RTT:5.134ms RTTsd:.201ms Loss:22%)
Mar 11 06:50:14	check_reload_status	439	updating dyndns VPN_NAME_REMOVED-OPENVPN_VPNV4
Mar 11 06:50:14	check_reload_status	439	Restarting IPsec tunnels
Mar 11 06:50:14	check_reload_status	439	Restarting OpenVPN tunnels/interfaces
Mar 11 06:50:14	check_reload_status	439	Reloading filter
Mar 11 06:50:14	rc.gateway_alarm	52480	>>> Gateway alarm: FIBER_PPPOE (Addr:FIBER_IP_ADDR_REMOVED Alarm:1 RTT:3.991ms RTTsd:4.251ms Loss:22%)
Mar 11 06:50:14	check_reload_status	439	updating dyndns FIBER_PPPOE
Mar 11 06:50:14	check_reload_status	439	Restarting IPsec tunnels
Mar 11 06:50:14	check_reload_status	439	Restarting OpenVPN tunnels/interfaces
Mar 11 06:50:14	check_reload_status	439	Reloading filter
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: MONITOR: FIBER_PPPOE has packet loss, omitting from routing group Failover
Mar 11 06:50:15	php-fpm	78891	FIBER_IP_ADDR_REMOVED|FIBER_GW_ADDR_REMOVED|FIBER_PPPOE|3.903ms|4.181ms|23%|down|highloss
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: Gateway, switch to: ADSL_PPPOE
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: Default gateway setting Interface ADSL_PPPOE Gateway as default.
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: Gateway, switch to: ADSL_PPPOE
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: Default gateway setting Interface ADSL_PPPOE Gateway as default.
Mar 11 06:50:15	php-fpm	62663	/rc.filter_configure_sync: Gateway, switch to: ADSL_PPPOE
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 11 06:50:15	php-fpm	78891	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use VPN_NAME_REMOVED-OPENVPN_VPNV4.
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use FIBER_PPPOE.
Mar 11 06:50:15	php-fpm	14462	/rc.openvpn: OpenVPN: Resync client1 VPN_NAME_REMOVED-
Mar 11 06:50:15	php-fpm	14462	OpenVPN terminate old pid: 76667
Mar 11 06:50:15	kernel		ovpnc1: link state changed to DOWN
Mar 11 06:50:15	check_reload_status	439	Reloading filter
Mar 11 06:50:16	php-fpm	62663	/rc.filter_configure_sync: GW States: Gateway is down but its IP address cannot be located. Skipping state kill.: VPN_NAME_REMOVED-OPENVPN_VPNV4
Mar 11 06:50:16	php-fpm	14462	OpenVPN PID written: 8169
Mar 11 06:50:17	kernel		ovpnc1: link state changed to UP
Mar 11 06:50:17	check_reload_status	439	rc.newwanip starting ovpnc1
Mar 11 06:50:18	php-fpm	30069	/rc.newwanip: rc.newwanip: Info: starting on ovpnc1.
Mar 11 06:50:18	php-fpm	30069	/rc.newwanip: rc.newwanip: on (IP address: 100.78.0.61) (interface: VPN_NAME_REMOVED-OPENVPN[opt2]) (real interface: ovpnc1).
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: MONITOR: FIBER_PPPOE is available now, adding to routing group Failover
Mar 11 06:50:20	php-fpm	30069	FIBER_IP_ADDR_REMOVED|FIBER_GW_ADDR_REMOVED|FIBER_PPPOE|5.186ms|2.301ms|0.0%|online|none
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: Gateway, switch to: FIBER_PPPOE
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: Default gateway setting Interface FIBER_PPPOE Gateway as default.
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: IP Address has changed, killing states on former IP Address 100.78.7.90.
Mar 11 06:50:20	php-fpm	30069	/rc.newwanip: Creating rrd update script
Mar 11 06:50:22	php-fpm	30069	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 100.78.7.90 -> 100.78.0.61 - Restarting packages.
Mar 11 06:50:22	check_reload_status	439	Starting packages
Mar 11 06:50:22	check_reload_status	439	Reloading filter

stephenw10

So the first thing that's logged is the gateway alarm?

If this was the 'typical' i225/226 reported issue it would lose link first and that would be logged before the gateway alarm triggers.

lazyt

thanks all for trying to help! I'm giving up and looking for a new box.