Chinese I226-V on 23.05.1, problems

w0w

@stephenw10
In logs, I see igc0 down event, then a bunch of other events that happening right after, like PPPoE disconnection and so on, nothing unusual, like you just pulled the cable out for a second, maybe, I don't know. The port down event is somewhere between all this mess, I believe a bit later then down even. Currently, I have no logs anymore, sorry, but I remember, it was in the same minute at least, if not the same second.

stephenw10

What's it actually connected to? Can you put a switch between as a test?

w0w

@stephenw10
Tried 3 different switches. Currently connected to 2.5G dumb zyxel switch, tried 1G some tp-link and 2.5G tp-link. No difference at all.

w0w

@stephenw10
Hmm....
After six days, something similar happened now on the ix0 interface on the main unit

Jul 24 02:10:11 	kernel 		ix0: link state changed to UP
Jul 24 02:10:11 	check_reload_status 	480 	Linkup starting ix0
Jul 24 02:10:11 	check_reload_status 	480 	Reloading filter
Jul 24 02:10:11 	php-fpm 	15509 	/rc.linkup: Removing static route for monitor 8.8.8.8 and adding a new route through 199.0.100.1
Jul 24 02:10:11 	php-fpm 	15509 	/rc.linkup: Shutting down Router Advertisement daemon cleanly
Jul 24 02:10:11 	ppp 	39147 	[wan] IFACE: Set description "WAN"
Jul 24 02:10:11 	ppp 	39147 	[wan] IFACE: Rename interface pppoe0 to pppoe0
Jul 24 02:10:11 	ppp 	39147 	[wan] IFACE: Down event
Jul 24 02:10:11 	check_reload_status 	480 	Rewriting resolv.conf
Jul 24 02:10:09 	php-cgi 	1355 	rc.kill_states: rc.kill_states: Removing states for interface pppoe0
Jul 24 02:10:09 	php-cgi 	1355 	rc.kill_states: rc.kill_states: Removing states for IP fe80::a236:9fff:fec3:4a2c%pppoe0/32
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: LayerDown
Jul 24 02:10:09 	ppp 	39147 	[wan] error writing len 8 frame to b0: Network is down
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: SendTerminateReq #2
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: state change Opened --> Closing
Jul 24 02:10:09 	ppp 	39147 	[wan] IPV6CP: Close event
Jul 24 02:10:09 	ppp 	39147 	[wan] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address
Jul 24 02:10:09 	check_reload_status 	480 	Rewriting resolv.conf
Jul 24 02:10:09 	php-cgi 	98827 	rc.kill_states: rc.kill_states: Removing states for interface pppoe0
Jul 24 02:10:08 	php-cgi 	98827 	rc.kill_states: rc.kill_states: Removing states for IP xx.yy.21.204/32
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: LayerDown
Jul 24 02:10:08 	ppp 	39147 	[wan] error writing len 8 frame to b0: Network is down
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: SendTerminateReq #4
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: state change Opened --> Closing
Jul 24 02:10:08 	ppp 	39147 	[wan] IPCP: Close event
Jul 24 02:10:08 	ppp 	39147 	[wan] IFACE: Close event
Jul 24 02:10:08 	ppp 	39147 	caught fatal signal TERM
Jul 24 02:10:07 	php-fpm 	15509 	/rc.linkup: DEVD Ethernet detached event for opt1
Jul 24 02:10:07 	php-fpm 	15509 	/rc.linkup: Hotplug event detected for ISP_LAN(opt1) dynamic IP address (4: dhcp)
Jul 24 02:10:05 	kernel 		ix0: link state changed to DOWN
Jul 24 02:10:05 	check_reload_status 	480 	Linkup starting ix0 
Jul 24 01:01:46 	php-cgi 	74734 	notify_monitor.php: Message sent to -@gmail.com OK

Those two lines in question…

Jul 24 02:10:05 	kernel 		ix0: link state changed to DOWN
Jul 24 02:10:05 	check_reload_status 	480 	Linkup starting ix0

Since timestamp is the same…
check_reload_status I believe that happened later because on ix0 down, is not it?

stephenw10

Do the Suricata logs show it restarting when that happens? In inline mode (netmap) it will bounce the link if the netmap interface is recreated. But that should be on the LAN side....

w0w

@stephenw10
I will clarify and draw your attention to the fact that this is my main or primary unit where igc replaced for test with ix (x550-t2) card. There is nothing in suricata logs. I do not think it's suricata or netmap. It looks more like card or driver or some kernel part failure… or i don't know what else it can be.

Is there something in the FreeBSD that ix and igc can use at some level?

Now testing secondary unit igc0 under iperf 100Mbit load, port speed is 2500, switch is placed between test server and igc0 port.

stephenw10

Well the logs show the link bouncing. So either it really did bounce in which case I'd be trying to confirm that from logs in the switch. Or it was a virtual interface of some sort like netmap. The only time I have seen the link bounced by something in software (other than a NIC config change) is when using Snort or Suricata in in-line mode.

w0w

@stephenw10
Well, how can I disable netmap completely?

stephenw10

Put Suricata in legacy mode. It uses netmap for in-line mode.

w0w

@stephenw10
I've removed suricata completely, but I still see
ix1: netmap queues/slots: TX 4/2048, RX 4/2048 in dmesg output for all the cards, is this normal?

w0w

I don’t want to jump to conclusions, but at the moment there is a suspicion that there is some kind of dependence between these connection breaks and the netmap, which is built into the kernel, as I understand it, and PPPoE. I can’t imagine what kind of dependence, but if the port is used on 226 as a normal DHCP through a similar connection, as in the case of PPPoE, the link is stable.
At the moment, I have replaced both cards on both testlab firewalls with the original Intel X550-T2 running latest firmware.
Also I removed the suricata, and we will see if the situation repeats itself next month.

stephenw10

@w0w said in Chinese I226-V on 23.05.1, problems:

is this normal?

Yes. The driver shows that as available queues when it attaches:

ix0: <Intel(R) X553 N (SFP+)> mem 0x80400000-0x805fffff,0x80604000-0x80607fff at device 0.0 on pci9
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: 00:08:a2:12:17:7e
ix0: eTrack 0x8000084b PHY FW V65535
ix0: netmap queues/slots: TX 4/2048, RX 4/2048

That doesn't mean that netmap itself is in use.

w0w

Preliminary information on one of the firewalls — X550-T2 works without problems. If the connection is interrupted, then only from the provider, every known amount of days. Link never going down as it did with igc.

stephenw10

Hmm, so the physical link stays up and only the PPPoE is restarted?

w0w

@stephenw10
Yes, exactly.

stephenw10

Hmm, I have no idea what would cause that on igc but not ix. Unless it's actually the igc link dropping causing PPPoE to reset. I've never seen it on any of our igc NICs but that is the reported symptom from those early igc NICs in Linux or Windows.

w0w

@stephenw10
Everything is possible. It can be fake i225-v3 or even 226, just something reflashed early 225. Also it can be this buggy pcie hub/switch that is present on 2 port version cards.
But, what is really strange is that when i used it just as DHCP WAN it was also rock stable, the problem remains when PPPoE is used. I don't know. Maybe some hidden bug in the driver

Unoptanio

@w0w
Hello,

what are the 2.5G NICs that definitely work fine with pfsense 2.7.0?

stephenw10

As far as I know the i225-V rev3 NICs work fine. That's what we have in the 6100 and 4100.

w0w

@Unoptanio
There is i226-V is on the market already, that is claimed as “fixed” version of 225.
But…
https://wccftech.com/intel-releases-new-driver-to-mitigate-i226-i225-ethernet-controller-issues/
This somehow still trial and error…
You can try it anyway, at your own risk. There is no good solution for that amount of money, only if you're going with next level, like 10Gbps Intel cards that have issues as well, but support is a bit better.