Chinese I226-V on 23.05.1, problems

stephenw10 · Jul 23, 2023, 4:25 PM

What's it actually connected to? Can you put a switch between as a test?

w0w · Jul 23, 2023, 6:29 PM

@stephenw10
Tried 3 different switches. Currently connected to 2.5G dumb zyxel switch, tried 1G some tp-link and 2.5G tp-link. No difference at all.

w0w · Jul 24, 2023, 4:10 PM

@stephenw10
Hmm....
After six days, something similar happened now on the ix0 interface on the main unit

Jul 24 02:10:11 	kernel 		ix0: link state changed to UP
Jul 24 02:10:11 	check_reload_status 	480 	Linkup starting ix0
Jul 24 02:10:11 	check_reload_status 	480 	Reloading filter
Jul 24 02:10:11 	php-fpm 	15509 	/rc.linkup: Removing static route for monitor 8.8.8.8 and adding a new route through 199.0.100.1
Jul 24 02:10:11 	php-fpm 	15509 	/rc.linkup: Shutting down Router Advertisement daemon cleanly
Jul 24 02:10:11 	ppp 	39147 	[wan] IFACE: Set description "WAN"
Jul 24 02:10:11 	ppp 	39147 	[wan] IFACE: Rename interface pppoe0 to pppoe0
Jul 24 02:10:11 	ppp 	39147 	[wan] IFACE: Down event
Jul 24 02:10:11 	check_reload_status 	480 	Rewriting resolv.conf
Jul 24 02:10:09 	php-cgi 	1355 	rc.kill_states: rc.kill_states: Removing states for interface pppoe0
Jul 24 02:10:09 	php-cgi 	1355 	rc.kill_states: rc.kill_states: Removing states for IP fe80::a236:9fff:fec3:4a2c%pppoe0/32
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: LayerDown
Jul 24 02:10:09 	ppp 	39147 	[wan] error writing len 8 frame to b0: Network is down
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: SendTerminateReq #2
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: state change Opened --> Closing
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: Close event
Jul 24 02:10:09 	ppp 	39147 	[wan] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
Jul 24 02:10:09 	check_reload_status 	480 	Rewriting resolv.conf
Jul 24 02:10:09 	php-cgi 	98827 	rc.kill_states: rc.kill_states: Removing states for interface pppoe0
Jul 24 02:10:08 	php-cgi 	98827 	rc.kill_states: rc.kill_states: Removing states for IP xx.yy.21.204/32
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: LayerDown
Jul 24 02:10:08 	ppp 	39147 	[wan] error writing len 8 frame to b0: Network is down
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: SendTerminateReq #4
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: state change Opened --> Closing
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: Close event
Jul 24 02:10:08 	ppp 	39147 	[wan] IFACE: Close event
Jul 24 02:10:08 	ppp 	39147 	caught fatal signal TERM
Jul 24 02:10:07 	php-fpm 	15509 	/rc.linkup: DEVD Ethernet detached event for opt1
Jul 24 02:10:07 	php-fpm 	15509 	/rc.linkup: Hotplug event detected for ISP_LAN(opt1) dynamic IP address (4: dhcp)
Jul 24 02:10:05 	kernel 		ix0: link state changed to DOWN
Jul 24 02:10:05 	check_reload_status 	480 	Linkup starting ix0 
Jul 24 01:01:46 	php-cgi 	74734 	notify_monitor.php: Message sent to -@gmail.com OK

Those two lines in question…

Jul 24 02:10:05 	kernel 		ix0: link state changed to DOWN
Jul 24 02:10:05 	check_reload_status 	480 	Linkup starting ix0

Since timestamp is the same…
check_reload_status I believe that happened later because on ix0 down, is not it?

stephenw10 · Jul 24, 2023, 5:33 PM

Do the Suricata logs show it restarting when that happens? In inline mode (netmap) it will bounce the link if the netmap interface is recreated. But that should be on the LAN side....

w0w · Jul 24, 2023, 6:10 PM

@stephenw10
I will clarify and draw your attention to the fact that this is my main or primary unit where igc replaced for test with ix (x550-t2) card. There is nothing in suricata logs. I do not think it's suricata or netmap. It looks more like card or driver or some kernel part failure… or i don't know what else it can be.

Is there something in the FreeBSD that ix and igc can use at some level?

Now testing secondary unit igc0 under iperf 100Mbit load, port speed is 2500, switch is placed between test server and igc0 port.

stephenw10 · Jul 25, 2023, 2:19 AM

Well the logs show the link bouncing. So either it really did bounce in which case I'd be trying to confirm that from logs in the switch. Or it was a virtual interface of some sort like netmap. The only time I have seen the link bounced by something in software (other than a NIC config change) is when using Snort or Suricata in in-line mode.

w0w · Jul 26, 2023, 8:59 AM

@stephenw10
Well, how can I disable netmap completely?

stephenw10 · Jul 26, 2023, 4:21 PM

Put Suricata in legacy mode. It uses netmap for in-line mode.

w0w · Jul 26, 2023, 4:21 PM

@stephenw10
I've removed suricata completely, but I still see
ix1: netmap queues/slots: TX 4/2048, RX 4/2048 in dmesg output for all the cards, is this normal?

w0w · Jul 26, 2023, 4:53 PM

I don’t want to jump to conclusions, but at the moment there is a suspicion that there is some kind of dependence between these connection breaks and the netmap, which is built into the kernel, as I understand it, and PPPoE. I can’t imagine what kind of dependence, but if the port is used on 226 as a normal DHCP through a similar connection, as in the case of PPPoE, the link is stable.
At the moment, I have replaced both cards on both testlab firewalls with the original Intel X550-T2 running latest firmware.
Also I removed the suricata, and we will see if the situation repeats itself next month.

stephenw10 · Jul 26, 2023, 5:22 PM

@w0w said in Chinese I226-V on 23.05.1, problems:

is this normal?

Yes. The driver shows that as available queues when it attaches:

ix0: <Intel(R) X553 N (SFP+)> mem 0x80400000-0x805fffff,0x80604000-0x80607fff at device 0.0 on pci9
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 00:08:a2:12:17:7e
ix0: eTrack 0x8000084b PHY FW V65535
ix0: netmap queues/slots: TX 4/2048, RX 4/2048

That doesn't mean that netmap itself is in use.

w0w · Aug 13, 2023, 4:58 AM

Preliminary information on one of the firewalls — X550-T2 works without problems. If the connection is interrupted, then only from the provider, every known amount of days. Link never going down as it did with igc.

stephenw10 · Aug 13, 2023, 2:05 PM

Hmm, so the physical link stays up and only the PPPoE is restarted?

w0w · Aug 14, 2023, 8:22 AM

@stephenw10
Yes, exactly.

stephenw10 · Aug 14, 2023, 2:57 PM

Hmm, I have no idea what would cause that on igc but not ix. Unless it's actually the igc link dropping causing PPPoE to reset. I've never seen it on any of our igc NICs but that is the reported symptom from those early igc NICs in Linux or Windows.

w0w · Aug 14, 2023, 2:57 PM

@stephenw10
Everything is possible. It can be fake i225-v3 or even 226, just something reflashed early 225. Also it can be this buggy pcie hub/switch that is present on 2 port version cards.
But, what is really strange is that when i used it just as DHCP WAN it was also rock stable, the problem remains when PPPoE is used. I don't know. Maybe some hidden bug in the driver

Unoptanio · Aug 31, 2023, 9:50 AM

@w0w
Hello,

what are the 2.5G NICs that definitely work fine with pfsense 2.7.0?

stephenw10 · Aug 31, 2023, 11:55 AM

As far as I know the i225-V rev3 NICs work fine. That's what we have in the 6100 and 4100.

w0w · Sep 1, 2023, 7:48 AM

@Unoptanio
There is i226-V is on the market already, that is claimed as “fixed” version of 225.
But…
https://wccftech.com/intel-releases-new-driver-to-mitigate-i226-i225-ethernet-controller-issues/
This somehow still trial and error…
You can try it anyway, at your own risk. There is no good solution for that amount of money, only if you're going with next level, like 10Gbps Intel cards that have issues as well, but support is a bit better.

stephenw10 · Sep 1, 2023, 8:50 AM

We use that in the 8200 and haven't seen any issues there either. The i226 is significantly more efficient. Uses about half the power IIRC.