Port Speed issue on 25.07.1 and Netgate 4200-S

scruzuser

Been running pfsense+ 25.07.1-RELEASE on Netgate 4200-S for months Connected to new AT&T CGW452 which has 2.5Gbps ports. AT&T service is 1Gbps fiber. CGW452 is in "IP Passthrough" mode. Just recently realized, by looking at the logs, the "service" isn't stable - explains why my VPN drops occasionally (that's what triggered the review of the logs).

When WAN interface "Speed and Duplex" setting is set to either "Default (no-preference, typically auto-select)" or "2500BaseT" the link connection resets randomly and I lose my VPN connections:

Jan 18 17:23:36 pfSense-tsv check_reload_status[637]: Linkup starting igc3
Jan 18 17:23:36 pfSense-tsv kernel: igc3: link state changed to DOWN
Jan 18 17:23:40 pfSense-tsv check_reload_status[637]: Linkup starting igc3
Jan 18 17:23:40 pfSense-tsv kernel: igc3: link state changed to UP
Jan 18 17:23:41 pfSense-tsv php-fpm[59733]: /rc.linkup: HOTPLUG: Triggering address refresh on wan (igc3)
Jan 18 17:23:41 pfSense-tsv check_reload_status[637]: rc.newwanip starting igc3
Jan 18 17:23:42 pfSense-tsv php-fpm[59733]: /rc.newwanip: rc.newwanip: Info: starting on igc3.
Jan 18 17:23:42 pfSense-tsv php-fpm[59733]: /rc.newwanip: rc.newwanip: on (IP address: XXX.XXX.XXX.XXX) (interface: WAN[wan]) (real interface: igc3).

If I set the WAN link connection to "1000baseT full-duplex" connection remains stable, no restarts of the igc3 link. My igc2 link, which is LAN, is rock solid. It is connected to a 1Gbps Unifii switch.

All interfaces on the AT&T CGW452 are set to auto-select.

I have swapped ports on the CGW452 for the WAN connection. I have purchased new Cat6 cables and installed.

igc firmware is at v2.17-0, which I think is a bit behind, lots of info out there about V2.17-0 being problematic (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265714)

igc3: <Intel(R) Ethernet Controller I226-V> mem 0x67100000-0x671fffff,0x67200000-0x67203fff at device 0.0 on pci6
igc3: EEPROM V2.17-0 eTrack 0x80000303
igc3: Using 1024 TX descriptors and 1024 RX descriptors
igc3: Using 1 RX queues 1 TX queues
igc3: Using MSI-X interrupts with 2 vectors
igc3: Ethernet address: 90:ec:77:92:40:da
igc3: netmap queues/slots: TX 1/1024, RX 1/1024

I'm hopeful an upgrade to 25.11 will update the igc driver and resolve this.
Any thoughts on this?

patient0

@scruzuser said in Port Speed issue on 25.07.1 and Netgate 4200-S:

on Netgate 4200-S

Is 4200-S a special model of the 4200?

If I set the WAN link connection to "1000baseT full-duplex" connection remains stable

See here: https://forum.netgate.com/post/1181117 and https://forum.netgate.com/post/1223314

It seems the driver on the 4200 is best set to default, nothing else. Changing it back to default seems to need a reboot sometimes to really stick.

Can set the port speed on the modem to a fixed 2.5GbE and see if then the speed negotiation works better?

SteveITS

@scruzuser see the blue note at https://docs.netgate.com/pfsense/en/latest/solutions/netgate-4200/io-ports.html#networking-ports

Maybe try a small switch between the devices?

Of what gain is a faster port speed with 1Gbps Internet?

scruzuser

@patient0 The 4200-S is the 4200 with an M.2 SSD instead of the eMMC memory.

I changed from "Default (no-preference, typically auto-select)" to "1000BaseT full-duplex" and restarted the interface and it locked at 1000Mbps. After a few days, I changed it to the 2500BaseT and it locked at 2500Mbps. As shown by ifconfig. No need to reboot. If I go back to "Default (no-preference, typically auto-select)" I could see I might need to reboot.

Speed negotiation is fine, I just suspect something is wrong with the firmware at 2500BaseT (auto-negotiated or "set" - even though it is "negotiated" as per the docs that @SteveITS) shows below.

scruzuser

Things are working fine at 1000BaseT full-duplex and it is no problem with my 1Gbps service.

I'm just reporting a problem in case someone else has it set to "Default (no-preference, typically auto-select)" with a CGW452 and wonders why things are going DOWN/UP.

As per the doc you reference, me setting the WAN speed to 1000BaseT full-duplex just " limiting the values offered during autonegotiation to the speed/duplex value selected in the GUI." And things are happy.

Try to use 2500BaseT and things aren't happy. I'm a business customer in a shared building in California and AT&T won't sell me 2.5Gbps as they say they might run out of bandwidth for the building. But if I were at 2.5Gbps, I'd be bummed.

I'll report back when I try 25.11 which says it uses FreeBSD 16-CURRENT and from what I can tell that upgrades the igc firmware to v2.23-0 or beyond...

stephenw10

The new driver does not update the NIC firmware. More likely it's compatible with it.

scruzuser

@stephenw10 Darn, more digging agrees with you.....

Is it possible to update the firmware? There are just lots of posts out there that suggest there are instabilities with 2.5Gbps links and you need to be at newer versions, i.e. 25 or 27 for it to be stable.

Is upgrading the firmware fraught with peril? Is this something that Netgate would release?
Edit: Further reading suggests we would have to wait for Netgate to release new firmware.

Right now, auto-select and running at 2.5Gbps is just buggy and doesn't work with the CGW452. I'd like to help the community out by solving this, so any suggestions are appreciated.
Edit: I'll report back once I upgrade to 25.11

stephenw10

I've never tried it. I've not seen any link issue like that here.

Do you see anything in the MAC stats in the sysctls that might indicate it losing link for some reason?

Can you test a 2.5G switch in between?

scruzuser

Upgraded to 25.11 - link still flaps when set to 2500BaseT.
Really need Netgate to provide firmware upgrade if the 4200 is to work with AT&T CGW452 at speed.
I'm locked at 1000BaseT full-duplex and that continues to be stable.
I don't have a 2.5G switch right now. Will try to acquire and test with switch in between.

stephenw10

With igc NICs you pretty much always want them set to 'default' not either 2.5G or autoselect. Setting 'default' does not apply any link rate setting and for some connections it will flap when set. That also means you might have to power cycle the unit after setting it in order to get the default values on the NIC and not set anything else. Try that if you have not already.

scruzuser

@stephenw10 I had it set to default - that's what started me down this journey. The link was flapping down/up... I can try again now that I upgraded to 25.11. An experiment for the weekend as this is a production system - have to find a "quiet" time.

If you search around the internet though, you'll see many improvements to i226 firmware for 2.5G performance. I think Netgate really needs to look at providing us a firmware upgrade.

stephenw10

I'll see what we can do but I'm not sure it's possible to do it from FreeBSD.

scruzuser

I would expect a firmware upgrade will have to be done from the BIOS interface somehow. Something along the lines of what's documented here:
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-4200/fw-update-sbl.html

stephenw10

It would need to include something additionally there. The NVM firmware is not part of the BIOS.

It does look like there is a utility that will run in FreeBSD that can update it though.

workaholiker

@stephenw10
i'm also suffering from port flapping on my 4200.

igc2: link state changed to DOWN
igc2: link state changed to UP
igc3: link state changed to DOWN
igc3: link state changed to UP
igc2: link state changed to DOWN
igc3: link state changed to DOWN
igc2: link state changed to UP
igc3: link state changed to UP
igc3: link state changed to DOWN
igc2: link state changed to DOWN
igc3: link state changed to UP

WAN is connected to ISP Modem, LAN to a switch. Both ports are loosing connections periodically. The only solution is to set the ports to fixed speed.

I followed this thread https://forum.opnsense.org/index.php?topic=48695.120 and managed to upgrade NVM, but the problem is still there.

[26.03-RELEASE][admin@xxx]/root: dmesg | grep -e 'igc.*EEPROM'
OLD:
igc0: EEPROM V2.17-0 eTrack 0x80000303
NEW:
igc0: EEPROM V2.32-0 eTrack 0x80000422

stephenw10

Hmm, setting it to fixed usually creates the flapping rather than solves it.

Are both WAN and LAN are linking at 2.5G?

If they link and are solid at 1G that implies auto-negotiation is working and it could just be a cabling issue.

workaholiker

@stephenw10

Cables were the first thing i've checked and even replaced with brand new cat6a (1 meter long)
wan is connected to 2.5G Port and can sync with 2.5 if set to autoneg
lan is connected to a 1gbit switch, also connects with 1Gbit with autoneg.

auto-negotioation is working, but flapping. only after setting speed and duplex to 2500Base-T or 1000Base-T full duplex issues are solved.

stephenw10

Hmm, interesting. The igc NICs in the 4200 can in fact only connect using auto negotiate. When you set them to a fixed speed it actually just only negotiates to that one speed. If you try to link them to something that's actually fixed with negotiation disabled they will fail to link.
So that implies they can link and be stable as long as the negotiated speed is appropriate.

scruzuser

@stephenw10 My experience (posted above), with the older firmware, showed that even if I set 2500BaseT and other side was auto-negotiate I still had flapping. I still have 1000BaseT set now and it has been stable for months. This is on my WAN side, igc3.

LAN side, igc2, is set to auto-negotiate and other side is 1000BaseT and it has been stable since day one.

Looks like @workaholiker has proven that with new firmware setting speed to 2500Base-T stops flapping.

I also had purchased new 1m Cat6a cables to rule them out.