[RESOLVED] pfSense with CenturyLink - working but only 90Mbps on 940Mbps service

jhg-goow

@rcoleman-netgate Will a 4100 do 1Gbps over PPPoE? I'm seeing references online to the fact that the PPPoE slowness is a known issue in FreeBSD, logged as a bug, and has been marked "won't fix" because PPPoE is so ancient.

rcoleman-netgate

@jhg-goow I cannot test that but I think that @stephenw10 might have previously.

jhg-goow

@rcoleman-netgate On a slightly different tack (the 4100 is too rich for our non-profit budget right now), are there Ethernet adapters (Intel?) that are known to not cause problems with PPPoE?

jhg-goow

I cannot test that but I think that @stephenw10 might have previously.

@rcoleman-netgate As a new user I can't contact him directly. I guess I have to wait until he sees the mention?

I just had another thought. The speed I'm seeing is just below 100Mbps, which sounds suspiciously like one of the connections to the router is running at 100 instead of GbE.

The ifconfig output for both ethernet adapters still says

media: Ethernet autoselect (1000baseT <full-duplex>)

Are you aware of any failure mode in a cable which would result in the OS still believing it's running at 1000Mbps while actually being negotiated down to 100Mbps?

I am not at the location again until Thursday but will try another cable.

stephenw10

Despite some initial reports the issue with PPPoE is not restricted to any one driver or hardware. However you will not see it on some hardware (like Realtek) because it only supports a single receive queue. The fact that PPPoE can only use one queue makes no significant difference.

However I don't believe that's all you're seeing here. Bothe the fact you're using Realtek NICs and that you have a PPPoE connection will reduced throughput but not to 90Mbps. I would expect to see at least 500Mbps given no other issues.

Are you seeing errors on the WAN in Status > Interfaces? Is it actually linked at 1G?

What does ifconfig -vvvm re0 show?

The ISP requires a VLAN, does it require priority tagging on that?

Steve

jhg-goow

@stephenw10 Here's the output:

]/root: ifconfig -vvvm re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        capabilities=18399b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_UCAST,WOL_MCAST,WOL_MAGIC,LINKSTATE,NETMAP>
        ether 84:47:09:15:cd:5d
        inet6 fe80::8647:9ff:fe15:cd5d%re0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect mediaopt flowcontrol
                media autoselect
                media 1000baseT mediaopt full-duplex,flowcontrol,master
                media 1000baseT mediaopt full-duplex,flowcontrol
                media 1000baseT mediaopt full-duplex,master
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex,flowcontrol
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex,flowcontrol
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
                media none
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I'm now thinking I might have a bad cable that only supports 100Mbps, but I won't be at the site until Thursday to swap cables. Although, I would expect that if the speed had been negotiated down that would show up in the ifconfig output.

For completeness, here's the output for re0.201

re0.201: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN
        options=80003<RXCSUM,TXCSUM,LINKSTATE>
        capabilities=80003<RXCSUM,TXCSUM,LINKSTATE>
        ether 84:47:09:15:cd:5d
        inet6 fe80::8647:9ff:fe15:cd5d%re0.201 prefixlen 64 scopeid 0x7
        groups: vlan
        vlan: 201 vlanpcp: 0 parent interface: re0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

rcoleman-netgate

@stephenw10 said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

The ISP requires a VLAN, does it require priority tagging on that?

No, I have the same ISP.

stephenw10

Yes I would expect it to show in the media: line there and it shows 1G.
90Mbps sure seems suspiciously like what you'd see if something were linked at 100M though. Check the LAN side.

I would still check the ISP supplied router for any priority tagging to be sure. We have seen ISPs do exactly that sort of throttling but still allow the connection when you don't apply it. Maybe some legacy requirement from an earlier company.

Steve

rcoleman-netgate

This is my FTTP from Lumen for PPPoE:

jhg-goow

@rcoleman-netgate Mine looks the same except for the Parent Interface:

rcoleman-netgate

@jhg-goow I would swap interfaces, cables, switch ports, whatever you can to verify it's not a failure that's causing the slowness.

stephenw10

Yup, check re1 is not linked at 100M.

If you have access to the Zyxel router check the WAN config there to be sure.

Steve

jhg-goow

If you have access to the Zyxel router check the WAN config there to be sure.

@stephenw10 I do, and the connection was definitely operating at 1Gbps (getting about 900Mbps) prior to being replaced with pfSense.

At this point, if it does not turn out to be a bad cable, I'll be forced to abandon pfSense and go back to the Zyxel router. I can't afford to buy several different mini-pcs with different ethernet adapters to troubleshoot this (non-profit, shoestring budget).

At home I have a Zotac CI323 (RTL8111 ethernet) running CentOS Stream 8 as a router/firewall, and it comfortably does 950Mbps on Comcast in bridge mode. But Comcast doesn't use PPPoE, it's a raw Ethernet connection directly to the Internet. CL's insistence on such an antiquated system seems counterproductive.

stephenw10

And you can see the VLAN setup on the Zyxel is not using priority tags?

jhg-goow

@stephenw10 said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

And you can see the VLAN setup on the Zyxel is not using priority tags?

There's nothing in the Zyxel UI that mentions priority tags other than on the screenshot I posted earlier. My pfSense config matches @rcoleman-netgate and he says he sees the full 900Mbps+ speed.

rcoleman-netgate

@jhg-goow said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

and he says he sees the full 900Mbps+ speed.

On intel drivers.

I highly suspect it's the Realtek chipset.

stephenw10

@jhg-goow said in pfSense with CenturyLink - working but only 90Mbps on 940Mbps service:

There's nothing in the Zyxel UI that mentions priority tags other than on the screenshot I posted earlier.

Not seeing any screenshots from the Zyxel but if it shows nothing about tagging that won't help.

jhg-goow

@stephenw10 After all the wailing and gnashing of teeth, the problem turned out to be a combination of a bad cable, lying hardware and lying software.

The bad cable was the one between the ONT and the WAN port on the router.
The lights on the router's WAN ethernet port were both lit, supposedly indicating connection at 1000Mbps (the ONT doesn't have a speed indicating LED).
In the pfSense shell, ifConfig re0 claimed the adapter was connected at 1000Mbps.

As soon as I swapped in a different cable, Internet speed went up to 930Mbps.

Lessons learned:

ALWAYS check the simplest things first, including cables.
Ethernet hardware and software can lie to you about connection speed in some cases.
A Celeron J4125 @ 2.00GHz with RTL8111 ethernet is capable of running pfSense and handling PPPoE at 1Gbps quite comfortably. At one point early in the troubleshooting I did set net.isr.dispatch=deferred, which may be necessary.
Pay attention to the numbers. It took me embarrassingly long to realize 90Mbps speed was telling me one of the adapters was running at 100Mbps instead of 1G.

It is a big relief to have this working (and prove my hardware and FreeBSD weren't the fundamental limitations), and I really appreciate all the suggestions and help I got on this forum.

Thank you all.

stephenw10

Nice. Yeah the interesting thing is that the Ethernet negotiation only requires 2 pairs but Gigabit requires all 4. So it's possible for both ends to negotiate 1G when the cable cannot support it. Some NICs will detect that and prevent it. Not Realtek apparently.

jhg-goow

@stephenw10 @stephenw10 It turns out the problem wasn't quite resolved with the new cable. After a few hours I noticed the speed had dropped again to 100Mbps.

I remembered that there was an Eero WiFi access point in the same general area, and it was quite close to the firewall box. I unplugged the AP and the connection immediately reverted to 1Gbps.

I have never heard of WiFi (2.4GHz, 5-6Ghz) interfering with wired Ethernet (125MHz), but I suppose the Eero device could be poorly designed and leak significantly at other frequencies at close range.

I moved the WiFi AP several feet away and also replaced the ONT<--> pfSense and AP<-->switch cables with CAT6a shielded twisted pair for good measure. The connection has been stable at 1GBps for several hours now, so I believe (hope:-) the problem is finally resolved.