[RESOLVED] pfSense with CenturyLink - working but only 90Mbps on 940Mbps service

jhg-goow

I have set up pfSense 2.6.0-RELEASE (amd64) on a Minisforum GK41 (Celeron J4125). This replaces a Zyxel 3000CZ, which was providing 940Mbps service.

pfSense is working correctly (I followed the VLAN configuration steps), EXCEPT I'm getting only 90Mbps in both directions using speedtest.net. This is from a host on the same LAN (same switch) as pfSense with GBEthernet. I tried several different speedtest servers and the result is the same.

This isn't a CPU issue as the load never goes above 1% even while running a speedtest.

Here's the device info from pciconf

re0@pci0:2:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x15 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 10[70] = PCI-Express 2 endpoint MSI 1 max data 128(128) RO
                 max read 4096
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1) ClockPM disabled
    cap 11[b0] = MSI-X supports 4 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x800]
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0002[140] = VC 1 max VC0
    ecap 0003[160] = Serial 1 01000000684ce000
    ecap 0018[170] = LTR 1
    ecap 001e[178] = L1 PM Substates 1

and here are the lines from dmesg for re0

re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xc1304000-0xc1304fff,0xc1300000-0xc1303fff at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: Chip rev. 0x54000000
re0: MAC rev. 0x00100000
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 84:47:09:15:cd:5d
re0: netmap queues/slots: TX 1/256, RX 1/256
re0: link state changed to DOWN
re0: link state changed to UP
re0.201: link state changed to UP

And the PCI slot info from dmidecode

Handle 0x0019, DMI type 9, 19 bytes
System Slot Information
        Designation: J7H1
        Type: x4 PCI Express 2 x4
        Current Usage: In Use
        Length: Short
        ID: 0
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:00:13.0
        Data Bus Width: 10
        Peer Devices: 0

Handle 0x001A, DMI type 9, 19 bytes
System Slot Information
        Designation: J8H1
        Type: x2 PCI Express 2 x2
        Current Usage: In Use
        Length: Short
        ID: 1
        Characteristics:
                3.3 V is provided
                Opening is shared
                PME signal is supported
        Bus Address: 0000:00:14.0
        Data Bus Width: 9
        Peer Devices: 0

I also tried running /usr/local/bin/speedtest from the pfSense console interface. It got to about 80% and then crashed the system hard (had to power cycle to get it back up).

Also, I found references to poor PPPoE performance in several places, including the FreeBSD bug tracker, but those seem to affect the igb(8) driver, which AFAICT isn't used here.

I would like suggestions on how to troubleshoot this 90% network speed reduction. Or is the only solution to put the Zyxel into bridge mode between the ONT and pfSense, and let it handle the PPPoE? I'd really rather not have an extra device in the mix.

jhg-goow

I found this post from 2015, but it applies to the igb driver.

Is there a way to get similar statistics (packets tx/rx by queue) for the re driver? I couldn't find anything in sysctl output listing packet counts.

jhg-goow

Setting net.isr.dispatch=deferred has no effect.

chpalmer

@jhg-goow

What interfaces are you trying to test with? There seems to be two options on that box for a second interface.. USB or Wireless..

jhg-goow

@chpalmer said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

@jhg-goow

What interfaces are you trying to test with? There seems to be two options on that box for a second interface.. USB or Wireless..

The GK41 has two Ethernet ports, re0 and re1. I have re0 set up as the WAN port and re1 as LAN. Everything's working, just at only 10% of the rated speed.

rcoleman-netgate

@jhg-goow said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

The GK41 has two Ethernet ports, re0 and re1. I have re0 set up as the WAN port and re1 as LAN. Everything's working, just at only 10% of the rated speed.

Do you have hardware that doesn't rely on RealTek chipsets to test with?

NollipfSense

RealTek chipsets...my thinking as well and these two don't work well together and strong documentation on forum: re0: netmap queues/slots:

jhg-goow

@nollipfsense said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

RealTek chipsets...my thinking as well and these two don't work well together and strong documentation on forum: re0: netmap queues/slots:

So is the problem specifically with the Realtek chipset, or is there a general issue with PPPoE being a resource hog?

If I were to buy, say, a Netgate 2100 appliance would I have the same issues with a PPPoE connection? That's all CenturyLink provides here (Portland, OR).

rcoleman-netgate

@jhg-goow said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

So is the problem specifically with the Realtek chipset, or is there a general issue with PPPoE being a resource hog?

Could be that, too, but I have pretty good PPPoE FTTP from Lumen on my 7100.

If I were to buy, say, a Netgate 2100 appliance would I have the same issues with a PPPoE connection? That's all CenturyLink provides here (Portland, OR).

No. The 2100 will not push FTTP Gigabit, regardless of the PPPoE factor.
In our current lineup of hardware you want a minimum of a 4100.

jhg-goow

@rcoleman-netgate Will a 4100 do 1Gbps over PPPoE? I'm seeing references online to the fact that the PPPoE slowness is a known issue in FreeBSD, logged as a bug, and has been marked "won't fix" because PPPoE is so ancient.

rcoleman-netgate

@jhg-goow I cannot test that but I think that @stephenw10 might have previously.

jhg-goow

@rcoleman-netgate On a slightly different tack (the 4100 is too rich for our non-profit budget right now), are there Ethernet adapters (Intel?) that are known to not cause problems with PPPoE?

jhg-goow

I cannot test that but I think that @stephenw10 might have previously.

@rcoleman-netgate As a new user I can't contact him directly. I guess I have to wait until he sees the mention?

I just had another thought. The speed I'm seeing is just below 100Mbps, which sounds suspiciously like one of the connections to the router is running at 100 instead of GbE.

The ifconfig output for both ethernet adapters still says

media: Ethernet autoselect (1000baseT <full-duplex>)

Are you aware of any failure mode in a cable which would result in the OS still believing it's running at 1000Mbps while actually being negotiated down to 100Mbps?

I am not at the location again until Thursday but will try another cable.

stephenw10

Despite some initial reports the issue with PPPoE is not restricted to any one driver or hardware. However you will not see it on some hardware (like Realtek) because it only supports a single receive queue. The fact that PPPoE can only use one queue makes no significant difference.

However I don't believe that's all you're seeing here. Bothe the fact you're using Realtek NICs and that you have a PPPoE connection will reduced throughput but not to 90Mbps. I would expect to see at least 500Mbps given no other issues.

Are you seeing errors on the WAN in Status > Interfaces? Is it actually linked at 1G?

What does ifconfig -vvvm re0 show?

The ISP requires a VLAN, does it require priority tagging on that?

Steve

jhg-goow

@stephenw10 Here's the output:

]/root: ifconfig -vvvm re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        capabilities=18399b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST,WOL_MAGIC,LINKSTATE,NETMAP>
        ether 84:47:09:15:cd:5d
        inet6 fe80::8647:9ff:fe15:cd5d%re0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,flowcontrol,master
                media 1000baseT mediaopt full-duplex,flowcontrol
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex,flowcontrol
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex,flowcontrol
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
                media none
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I'm now thinking I might have a bad cable that only supports 100Mbps, but I won't be at the site until Thursday to swap cables. Although, I would expect that if the speed had been negotiated down that would show up in the ifconfig output.

For completeness, here's the output for re0.201

re0.201: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        capabilities=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 84:47:09:15:cd:5d
        inet6 fe80::8647:9ff:fe15:cd5d%re0.201 prefixlen 64 scopeid 0x7
        groups: vlan
        vlan: 201 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

rcoleman-netgate

@stephenw10 said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

The ISP requires a VLAN, does it require priority tagging on that?

No, I have the same ISP.

stephenw10

Yes I would expect it to show in the media: line there and it shows 1G.
90Mbps sure seems suspiciously like what you'd see if something were linked at 100M though. Check the LAN side.

I would still check the ISP supplied router for any priority tagging to be sure. We have seen ISPs do exactly that sort of throttling but still allow the connection when you don't apply it. Maybe some legacy requirement from an earlier company.

Steve

rcoleman-netgate

This is my FTTP from Lumen for PPPoE:

jhg-goow

@rcoleman-netgate Mine looks the same except for the Parent Interface:

rcoleman-netgate

@jhg-goow I would swap interfaces, cables, switch ports, whatever you can to verify it's not a failure that's causing the slowness.