Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1

randman76

I am experiencing the same problem. Pfsense CE 2.7.2 everything was great with download speeds of 2300mpbs. After upgrading to 2.8.0 or 2.8.1 I get around 700mpbs down. I even tried pfsense + and getting the same results. For the time being I went back to 2.7.2

patient0

@randman76 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

I am experiencing the same problem.

You got issues with the WAN interface speed negotiation, too? To what speed did it negotiate before and to what does it negotiate now? And what are the NICs in pfSense and to what is it connected?

C80SGEEK

@patient0 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

@C80SGEEK what is the actual link speed that is negotiated between firewall <-> switch and switch <-> router?

Hello,

Please note that normally there is no switch between Pfsense and router, I set this up to look closer into port negiciations :

On Pfsense 2.8.1

Wan is configured as 100BasteTXFullduplex.

Our internet provider sais that the router is configured to 100FDX

On the switch we can see that ports are not negociated automatically like they should :
The Router is automaic negiciated to 100HDx half duplex

                       Intrusion                               MDI   Flow *
  Port         Type       Alert    Enabled  Status     Mode     Mode  Ctrl *

------------ ---------- --------- ------- ------ ---------- ---- ---- *
1 100/1000T No Yes Up 100FDx MDI off
2 100/1000T No Yes Up 100HDx MDI off

Port 1 is Pfsense
Port 2 is router

After forcing the switch to configure Port 2 AS 100FDx

Everything start working like it should having up and down load speed at 90Mb/s

  Port         Type       Alert    Enabled  Status     Mode     Mode  Ctrl *

------------ ---------- --------- ------- ------ ---------- ---- ---- *
1 100/1000T No Yes Up 100FDx MDI off
2 100/1000T No Yes Up 100FDx MDI off

Our internet provider said that manual port setting are needed to make the router work normally on its lan section where pfsense is connected to. This worked since years and on PFsense 2.7.2 the wan port is forced to 100BasteTXFullduplex and working well with the router.

Which makes me think that the Web GUI configuration "Speed and Duplex" in pfSense 2.8.1 is not being applied correctly to the network card.

here is ifconfig for wan from Pfsense 2.8.1 :

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

here is ifconfig for wan on Pfsense 2.7.2 :

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:70:a5:ba
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Kind Regards

C80SGEEK

I have made some researches about speed negociation :

Our internet provider said that manual port setting are needed to make the router work normally on its lan section where pfsense is connected to so i think this is hard coded.

Pfsense documentation said : 

[https://docs.netgate.com/pfsense/en/latest/troubleshooting/low-throughput.html](link url)

If the CPE is hard-coded, but the firewall is not, it would show as using 100Mbit/s half-duplex on Status > Interfaces. The duplex mismatch will lead to interface errors, collisions, and low throughput.

We have set "Speed and Duplex" to 100BaseTXFullDuplex on Pfsense 2.7.2 and 2.8.1.This worked well since years and on PFsense 2.7.2 and previous versions with the router.

stephenw10

It's absurd to find ISPs still supplying devices with fixed rate ports like it's 1998! But.... ISPs what can you do.

So, yes, if the ISP modem/router is set to fixed speed and duplex you will need to configured whatever is connected to it to match that. So in pfSense or in the switch if you have that in between.

What media options are you offered in ifconfig -vvm igb0?

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

ifconfig -vvm igb0

So, yes, if the ISP modem/router is set to fixed speed and duplex you will need to configured whatever is connected to it to match that

We do not have a switch between ISP modem and Pfsense. we always had configured igb0 through web Gui Interfaces --> igb0 --> Speed and Duplex = 100baseTX full-duplex.

This normally configure igb0 to force speed and douplex matching the ISP Router. Note that this is working well on Pfsense 2.7.2.

This is why i thing that there must be a problem with the Gui configurations not well applied on Pfsense 2.8.1.

ifconfig -vvm igb0

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: igb0

So we can see that the NIC is 100baseTXfull-duplex capable. (like it worked on pfsense 2.7.2)

Testing to set it from command line with no luck :
I launched the commands through ssh to be sure to have the right feedback and the command completes normally.

ifconfig igb0 media 100baseTX mediaopt full-duplex
ifconfig -vvm igb0

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: igb0

stephenw10

Yes, that doesn't look like a gui issue. You can see the NIC is set to 100M Full-duplex but is still linking at half for some reason:
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)

With the switch in between it should pass traffic correctly as long as the switch port connected to the ISP router is set to 100M-Full. Have you demonstrated that?

It looks like this is a change in the igb driver somehow but I'm not aware of anything there.

stephenw10

Aha, this looks a likely suspect: https://github.com/pfsense/FreeBSD-src/commit/3ff0231c87f360afa4521e635b46f6c711dc4ee3

stephenw10

What happens if you set the media to 100M without setting the mediaopt value so it still tries to negotiate that?

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

Yes, that doesn't look like a gui issue. You can see the NIC is set to 100M Full-duplex but is still linking at half for some reason:
media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)

With the switch in between it should pass traffic correctly as long as the switch port connected to the ISP router is set to 100M-Full. Have you demonstrated that?

It looks like this is a change in the igb driver somehow but I'm not aware of anything there.


With the switch in between it should pass traffic correctly as long as the switch port connected to the ISP router is set to 100M-Full. Have you demonstrated that?

Yes, this test was already done 

After forcing the switch to configure Port 2 AS 100FDx

Everything start working like it should having up and down load speed at 90Mb/s

  Port         Type       Alert    Enabled  Status     Mode     Mode  Ctrl *

------------ ---------- --------- ------- ------ ---------- ---- ---- *
1 100/1000T No Yes Up 100FDx MDI off
2 100/1000T No Yes Up 100FDx MDI off

stephenw10

OK good.

It certainly looks like that reverted patch I linked to above is the culprit here. Just trying to see if it can be worked around with the existing code.

Otherwise it should be fix in 2.9 dev snapshots when they are available.

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

What happens if you set the media to 100M without setting the mediaopt value so it still tries to negotiate that?

I made the test by not specifying mediaportopt : from GUI and SSH

ifconfig igb0 media 100baseTX

ifconfig -vvm igb0

gb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:90:0b:72:d8:83
inet X.X.X.X netmask X.X.X.X broadcast X.X.X.X
inet6 X.X.X.X prefixlen 64 scopeid 0x3
media: Ethernet 100baseTX (100baseTX <half-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: igb0

No luck here proble still remains the same

C80SGEEK

@stephenw10 said in Issue with WAN speed negotiation after upgrading from 2.7 to 2.8 or 2.8.1:

Just trying to see if it can be worked around with the existing code.

This is an embarrassing situation, considering that this is a stable version of pfSense and that my hardware is in production. I also note that we had to wait a very long time between pfSense 2.7.2 and pfSense 2.8.

I am not sure I fully understand what the source code indicates and what it represents in the development branch: https://github.com/pfsense/FreeBSD-src/commit/3ff0231c87f360afa4521e635b46f6c711dc4ee3

The workaround I see is to use a managed switch and set the speeds and media options manually.
What concerns me here is that in the event of a real issue on an internet line, the ISP systematically suspects customer equipment (for example, a switch), and in this case, I would no longer be able to demonstrate the proper functioning of my firewall directly on the line.

The other option would be to revert to pfSense version 2.7.2, which is older and suffers from certain vulnerabilities.

Do you think that paid support could be a solution?

Kind regards,

stephenw10

Well in Plus 25.11 dev snapshots are now available and they contain the commit that revered that change. So you could test it now.

I'm looking at what we can do before that....

stephenw10

Opened a bug to track: https://redmine.pfsense.org/issues/16449

mer

From a long time ago, PHYs always had issues with "negotiation" if one side was set to fixed configurations and the other set to autonegotiate.
The way the hardware works (worked?) is speed can be figured out, duplex can't.
A way around this is leave both sides at autoneg, but you only present the options you want.
Example:
HW can do 10,100,1000 speeds, half, full. You want to only do 100, full so the autoneg options presented are "100, full" and the autoneg process does it's thing.
Presenting autoneg options is different than setting an interface to fixed. Setting to fixed means "i'm not even trying autoneg".

I think (believe/hope) most manufacturers leave autoneg on and interfaces restrict what is presented.
All that at least for copper.
Fiber I think really wants both sides saying "fixed".

stephenw10

Yup. It appears this patch attempted to allow linking to ports where speed is fixed but duplex is still negotiated. Which seems like an extreme edge case, I don't think I've ever seen it.

It has wound up introducing this but where igb is trying to negotiate the duplex and the other side is set fixed. Hence it falls back to half-duplex. Obviously it should be possible to just set it fixed because, yes, both sides must be set the same.

C80SGEEK

@mer
Hi,

As mentioned, our ISP router is a fiber optic router and has a fixed configuration.
We have always configured our firewall to match the router, namely 100BaseTX full-duplex. This had been working perfectly for more than ten years, up until pfSense version 2.8.0.

Kind regards

C80SGEEK

@stephenw10
What I understand is that, as of today, pfSense 2.8 no longer allows fixing the mediaopt configuration.
Do you think this will be corrected? Does the pfSense Plus version fix this bug?

stephenw10

The commit that reverted that patch is in 25.11-dev snapshots in Plus. But 25.07.1 is the same base as 2.8.1. I'm still looking at what we can do here. I'm sure it affects more users than just you.