Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1

C80SGEEK

Hello,

I have been using pfSense for over a decade, so I have some experience with this fantastic product, but today I am facing an issue for which I need your help.

I am encountering a problem with the latest version of pfSense CE.
Since upgrading to version 2.8.0, the speed of our WAN port has been reduced to half-duplex, which greatly impacts the upload speed, stagnating at 1.2 MB/s, whereas normally we have an upload speed of 90 MB/s.

I downgraded to version 2.7.2 on the same hardware, and the WAN port negotiates full-duplex again, reaching the expected upload/download speed of 90 MB/s.

I also tried upgrading directly from version 2.7.2 to the recently released version 2.8.1, but the issue persists: the WAN port is still negotiated in half-duplex, and the upload speeds remain reduced.

My research on the internet mostly led me to issues related to Realtek cards, where this phenomenon can occur.

However, the WAN network card in our hardware is an Intel NIC I210.

To be thorough, I tested replacing the cables, but the problem persists.
On some forums, it is suggested to disable IPv6, but this did not help.
Another option discussed was to disable hardware checksum offload, but even after a reboot, the link remains half-duplex.

Since these problems are usually related to Realtek drivers, I don’t know where to look regarding my Intel cards.

Thank you in advance for any help, which I would greatly appreciate.

C80SGEEK

Here is a little up,

As the big difference between pfsense 2.7.2 and 2.8.0 is FreeBSD 15. i checked the hardware compatibility :

My Intel NIC I210 is listed as compatible :

https://www.freebsd.org/releases/15.0R/hardware/#ethernet

The em driver supports Gigabit Ethernet adapters based on the Intel 82540, 82541ER, 82541PI, 82542, 82543, 82544, 82545, 82546, 82546EB, 82546GB, 82547, 82571, 82572, 82573, 82574, 82575, 82576, and 82580 controller chips:

Intel Gigabit ET Dual Port Server Adapter (82576)

Intel Gigabit VT Quad Port Server Adapter (82575)

Intel Single, Dual and Quad Gigabit Ethernet Controller (82580)

Intel i210 and i211 Gigabit Ethernet Controller

checking hardware info i have the same result on both devices:

sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'

dev.igb.7.%desc: Intel(R) I210 (Copper)
dev.igb.6.%desc: Intel(R) I210 (Copper)
dev.igb.5.%desc: Intel(R) I210 (Copper)
dev.igb.4.%desc: Intel(R) I210 (Copper)
dev.igb.3.%desc: Intel(R) I210 (Copper)
dev.igb.2.%desc: Intel(R) I210 (Copper)
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)
dev.ix.1.%desc: Intel(R) X552 (SFP+)
dev.ix.0.%desc: Intel(R) X552 (SFP+)

I connected a switch between the firewall wan and the router.
Port negociation becomes 100MB FullDuplex, but speed test still having bad results. like 1,2 MB/s Upload where on pfsens 2.7.2 i have all working as expected on 90 MB/S.

I really dont know where to search for this issue, all help will be most apreciated.

Kind regards

patient0

@C80SGEEK what is the actual link speed that is negotiated between firewall <-> switch and switch <-> router?

randman76

I am experiencing the same problem. Pfsense CE 2.7.2 everything was great with download speeds of 2300mpbs. After upgrading to 2.8.0 or 2.8.1 I get around 700mpbs down. I even tried pfsense + and getting the same results. For the time being I went back to 2.7.2

patient0

@randman76 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

I am experiencing the same problem.

You got issues with the WAN interface speed negotiation, too? To what speed did it negotiate before and to what does it negotiate now? And what are the NICs in pfSense and to what is it connected?

C80SGEEK

@patient0 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

@C80SGEEK what is the actual link speed that is negotiated between firewall <-> switch and switch <-> router?

Hello,

Please note that normally there is no switch between Pfsense and router, I set this up to look closer into port negiciations :

On Pfsense 2.8.1

Wan is configured as 100BasteTXFullduplex.

Our internet provider sais that the router is configured to 100FDX

On the switch we can see that ports are not negociated automatically like they should :
The Router is automaic negiciated to 100HDx half duplex

                       Intrusion                               MDI   Flow *
  Port         Type       Alert    Enabled  Status     Mode     Mode  Ctrl *

------------ ---------- --------- ------- ------ ---------- ---- ---- *
1 100/1000T No Yes Up 100FDx MDI off
2 100/1000T No Yes Up 100HDx MDI off

Port 1 is Pfsense
Port 2 is router

After forcing the switch to configure Port 2 AS 100FDx

Everything start working like it should having up and down load speed at 90Mb/s

  Port         Type       Alert    Enabled  Status     Mode     Mode  Ctrl *

------------ ---------- --------- ------- ------ ---------- ---- ---- *
1 100/1000T No Yes Up 100FDx MDI off
2 100/1000T No Yes Up 100FDx MDI off

Our internet provider said that manual port setting are needed to make the router work normally on its lan section where pfsense is connected to. This worked since years and on PFsense 2.7.2 the wan port is forced to 100BasteTXFullduplex and working well with the router.

Which makes me think that the Web GUI configuration "Speed and Duplex" in pfSense 2.8.1 is not being applied correctly to the network card.

here is ifconfig for wan from Pfsense 2.8.1 :

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

here is ifconfig for wan on Pfsense 2.7.2 :

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:70:a5:ba
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Kind Regards

C80SGEEK

I have made some researches about speed negociation :

Our internet provider said that manual port setting are needed to make the router work normally on its lan section where pfsense is connected to so i think this is hard coded.

Pfsense documentation said : 

[https://docs.netgate.com/pfsense/en/latest/troubleshooting/low-throughput.html](link url)

If the CPE is hard-coded, but the firewall is not, it would show as using 100Mbit/s half-duplex on Status > Interfaces. The duplex mismatch will lead to interface errors, collisions, and low throughput.

We have set "Speed and Duplex" to 100BaseTXFullDuplex on Pfsense 2.7.2 and 2.8.1.This worked well since years and on PFsense 2.7.2 and previous versions with the router.

stephenw10

It's absurd to find ISPs still supplying devices with fixed rate ports like it's 1998! But.... ISPs what can you do.

So, yes, if the ISP modem/router is set to fixed speed and duplex you will need to configured whatever is connected to it to match that. So in pfSense or in the switch if you have that in between.

What media options are you offered in ifconfig -vvm igb0?

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

ifconfig -vvm igb0

So, yes, if the ISP modem/router is set to fixed speed and duplex you will need to configured whatever is connected to it to match that

We do not have a switch between ISP modem and Pfsense. we always had configured igb0 through web Gui Interfaces --> igb0 --> Speed and Duplex = 100baseTX full-duplex.

This normally configure igb0 to force speed and douplex matching the ISP Router. Note that this is working well on Pfsense 2.7.2.

This is why i thing that there must be a problem with the Gui configurations not well applied on Pfsense 2.8.1.

ifconfig -vvm igb0

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: igb0

So we can see that the NIC is 100baseTXfull-duplex capable. (like it worked on pfsense 2.7.2)

Testing to set it from command line with no luck :
I launched the commands through ssh to be sure to have the right feedback and the command completes normally.

ifconfig igb0 media 100baseTX mediaopt full-duplex
ifconfig -vvm igb0

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: igb0

stephenw10

Yes, that doesn't look like a gui issue. You can see the NIC is set to 100M Full-duplex but is still linking at half for some reason:
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)

With the switch in between it should pass traffic correctly as long as the switch port connected to the ISP router is set to 100M-Full. Have you demonstrated that?

It looks like this is a change in the igb driver somehow but I'm not aware of anything there.

stephenw10

Aha, this looks a likely suspect: https://github.com/pfsense/FreeBSD-src/commit/3ff0231c87f360afa4521e635b46f6c711dc4ee3

stephenw10

What happens if you set the media to 100M without setting the mediaopt value so it still tries to negotiate that?

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

Yes, that doesn't look like a gui issue. You can see the NIC is set to 100M Full-duplex but is still linking at half for some reason:
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)

With the switch in between it should pass traffic correctly as long as the switch port connected to the ISP router is set to 100M-Full. Have you demonstrated that?

It looks like this is a change in the igb driver somehow but I'm not aware of anything there.


With the switch in between it should pass traffic correctly as long as the switch port connected to the ISP router is set to 100M-Full. Have you demonstrated that?

Yes, this test was already done 

After forcing the switch to configure Port 2 AS 100FDx

Everything start working like it should having up and down load speed at 90Mb/s

  Port         Type       Alert    Enabled  Status     Mode     Mode  Ctrl *

------------ ---------- --------- ------- ------ ---------- ---- ---- *
1 100/1000T No Yes Up 100FDx MDI off
2 100/1000T No Yes Up 100FDx MDI off

stephenw10

OK good.

It certainly looks like that reverted patch I linked to above is the culprit here. Just trying to see if it can be worked around with the existing code.

Otherwise it should be fix in 2.9 dev snapshots when they are available.

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

What happens if you set the media to 100M without setting the mediaopt value so it still tries to negotiate that?

I made the test by not specifying mediaportopt : from GUI and SSH

ifconfig igb0 media 100baseTX

ifconfig -vvm igb0

gb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX (100baseTX <half-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: igb0

No luck here proble still remains the same

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

Just trying to see if it can be worked around with the existing code.

This is an embarrassing situation, considering that this is a stable version of pfSense and that my hardware is in production. I also note that we had to wait a very long time between pfSense 2.7.2 and pfSense 2.8.

I am not sure I fully understand what the source code indicates and what it represents in the development branch: https://github.com/pfsense/FreeBSD-src/commit/3ff0231c87f360afa4521e635b46f6c711dc4ee3

The workaround I see is to use a managed switch and set the speeds and media options manually.
What concerns me here is that in the event of a real issue on an internet line, the ISP systematically suspects customer equipment (for example, a switch), and in this case, I would no longer be able to demonstrate the proper functioning of my firewall directly on the line.

The other option would be to revert to pfSense version 2.7.2, which is older and suffers from certain vulnerabilities.

Do you think that paid support could be a solution?

Kind regards,

stephenw10

Well in Plus 25.11 dev snapshots are now available and they contain the commit that revered that change. So you could test it now.

I'm looking at what we can do before that....

stephenw10

Opened a bug to track: https://redmine.pfsense.org/issues/16449

mer

From a long time ago, PHYs always had issues with "negotiation" if one side was set to fixed configurations and the other set to autonegotiate.
The way the hardware works (worked?) is speed can be figured out, duplex can't.
A way around this is leave both sides at autoneg, but you only present the options you want.
Example:
HW can do 10,100,1000 speeds, half, full. You want to only do 100, full so the autoneg options presented are "100, full" and the autoneg process does it's thing.
Presenting autoneg options is different than setting an interface to fixed. Setting to fixed means "i'm not even trying autoneg".

I think (believe/hope) most manufacturers leave autoneg on and interfaces restrict what is presented.
All that at least for copper.
Fiber I think really wants both sides saying "fixed".

stephenw10

Yup. It appears this patch attempted to allow linking to ports where speed is fixed but duplex is still negotiated. Which seems like an extreme edge case, I don't think I've ever seen it.

It has wound up introducing this but where igb is trying to negotiate the duplex and the other side is set fixed. Hence it falls back to half-duplex. Obviously it should be possible to just set it fixed because, yes, both sides must be set the same.