Windstream fiber -> Adtran 411 -> PfSense slow upload speed

nynx64

Just had windstream fiber installed. Using their gateway I get full synchronous speeds. When I connect their ONT directly to Pfsense; Downstream speed is fine. Upload is crawling at 3mbps. Are there any special settings I need to get this working properly?

pfsense box:
2.6.0 on a hyve zeus v1
riser card with CX312A Mellanox MCX312A-XCBT ConnectX-3 EN Dual Port 10 Gigabit. One is LAN other is WAN.

I have the ONT -> cat 6 -> 10G SFP+ RJ45 Copper -> mellanox card

It's been working great with cox for years up to 40mbps upload. Not sure why this is going so slow.

Here are my network interface settings:

Also I've tried to force link speed and duplex and still no go.. right now its auto negotiating to 10Gbase-CX4 <full-duplex,rxpause,txpause>

nynx64

Also just being clear.. the full synchronous is my laptop connected to their gateway.. if i connect pfsense to it i get the same issue. Something definitely going on with pfsense.

stephenw10

I would fisrt test a different NIC. I have mixed results with Mellanox NICs. Does it show linked correctly with the ONT? Are there any errors shown in Status > Interfaces?

I imagine your laptop is using a 1G NIC of some sort?

Steve

nynx64

@stephenw10

Yea 1G rj45 on laptop.

0/0 in out errors and 0 collisions.

When I disconnect the ONT and hook my old Cox internet back up the upstream is 40mbps (as it should be). Not sure what's bottlenecking the upstream just on windstream. I tried upgrading to 2.7.0 and then downgrading to pfsense 2.5.2 and the issue persists on both.

Wouldn't the upstream suffer regardless of connection if it was the NIC? There has to be a piece I'm missing. Like a special system tunable that only applies to fiber users.

The hyve zeus has two regular on board LAN ports.. but pfsense doesn't detect them.. just the mlxen0 and mlxen1.

JonathanLee

Are you using an SFP adapter? If so what brand? I just want to ask have you cleaned your connections with an approved fiber cloth or cleaning kit? Do you have a fiber scope? You can really inspect the connections with a scope too.

stephenw10

Does the Mellanox NIC link at 1G to the ONT? If it's trying to link to at 2.5G or 5G etc it could easily be causing issues.

I can't see exactly what the onboard NICs are on that server but they look like older Intel NICs that I'd expect to be well supported.

Are you running bare metal? That looks to be massively over powered for most pfSense installs. It should work though.

Steve

JKnott

@nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

I get full synchronous speeds.

I believe you mean symmetrical.

nynx64

@JonathanLee said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

Are you using an SFP adapter? If so what brand? I just want to ask have you cleaned your connections with an approved fiber cloth or cleaning kit? Do you have a fiber scope? You can really inspect the connections with a scope too.

the fiber was JUST installed.. and speeds are fine when I take pfsense out of the picture.. also the brand of the rj45 sfp is ipolex

@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

Does the Mellanox NIC link at 1G to the ONT? If it's trying to link to at 2.5G or 5G etc it could easily be causing issues.

I can't see exactly what the onboard NICs are on that server but they look like older Intel NICs that I'd expect to be well supported.

Are you running bare metal? That looks to be massively over powered for most pfSense installs. It should work though.

Steve

when i try to force 1000baseT, it actually doesn't work.. just keeps going at the 10Gbase-CX4... the only thing I know about the NICs is that they are intel and seem to work fine in a debian live cd.

yes running bare metal.. i figured it would be OP but those hyve zeus are way cheaper than alternatives.. i think i picked that one up for $100.

As for an update.. it is indeed the mellanox NIC that windstream doesn't like. I tested a usb NIC and assigned it to WAN.. although slow.. they were symmetrical. I ordered a Intel X520-DA2 off ebay.. hopefully that fares better once it gets here.. Switched back to cox internet in the meantime.

@JKnott said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

@nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

I get full synchronous speeds.

I believe you mean symmetrical.

Thank you.. i'll be sure to use the correct term for now on.

stephenw10

Mmm, that 'feels' like a link issue with the Mellanox. Possibly a flow control mismatch. The X520 should work.

What do you see for on-board NICs from pciconf -lv at the command line?

fathead

@nynx64
@stephenw10
[https://www.reddit.com/r/PFSENSE/comments/w2ts56/frontier_fiber_vlan0/igsf66n/](link url)
Some isp require upload tag on vlan 0 or 1
Have you set the vlan tag and priority bit?
What is your wan vlan config?

stephenw10

That's true but if it applied here it would also restrict a directly connected laptop.

steve10240

I'm also facing this issue, but with a different equipment and ISP (Frontier Fiber ISP -> Nokia XS-010X-Q ONT -> PFSense Asrock NUC BOX-1360P/D5 Fanned BOX with dual i226 NICs). When I am connected directly to the Nokia XS-010X-Q ONT box, I get symmetrical speeds both down and up (1gbps). When my PFSense box WAN port is connected to the Fiber ISP ONT box, download stays at 1gbps but upload speeds tank to 140mbps.

I am curious what the solution ends up being for the OP. I had another router on hand, Edgerouter PoE4, and when that router is hooked up to the ONT, it also gets full 1gbps/1gbps for download and upload. I am trying to figure out what is preventing the PFSense box from reaching its full upload speed capability.

nynx64

@fathead said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

@nynx64
@stephenw10
[https://www.reddit.com/r/PFSENSE/comments/w2ts56/frontier_fiber_vlan0/igsf66n/](link url)
Some isp require upload tag on vlan 0 or 1
Have you set the vlan tag and priority bit?
What is your wan vlan config?

Didn't tag WAN with any vlan.. just set the interface and left the rest default.

@steve10240

I have a frankenstein PC i use for testing. I pulled the mellanox card and ssd out of the zeus and transplanted them into that.. except i'm using the onboard 1G for WAN.

So it's definitely the NIC. This should get me by till that intel card delivers sometime later.

steve10240

@nynx64 I think it's my NIC too, but that is the last thing I haven't tried yet on the PFSense box. I tried placing a TrendNet 2.5G unmanaged switch between my Fiber ONT and PFSense router to see if that would resolve the upload speeds, but it didn't. I'm going to try a USB NIC.

I really thought there were some tunables that were needed to get the intel nics working right, but I don't know. Hopefully it works with your intel card!

stephenw10

An i226 NIC should be fine. Are you seeing errors on the NIC? Are you seeing multiple queues for bot Tx and Rx?

You might try setting hw.igc.max_interrupt_rate: 20000 though the default there should restrict anything at 140Mbps.

Steve

steve10240

@stephenw10 How do I check for the errors and the queues? Is that under diagnostics?

stephenw10

Errors would be shown in Status > Interfaces.

Queues are shown in the boot log whn the driver attaches but you can also check vmstat -i to make sure they are being used when under load.

steve10240

@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

Errors would be shown in Status > Interfaces.

Queues are shown in the boot log whn the driver attaches but you can also check vmstat -i to make sure they are being used when under load.

While running the Upload portion of the speedtest, this is the output of the vmstat -i command from my PFSense router:

interrupt                          total       rate
cpu0:timer                         29037        151
cpu1:timer                         13165         68
cpu2:timer                         26479        137
cpu3:timer                         12253         64
cpu4:timer                         22329        116
cpu5:timer                         10573         55
cpu6:timer                         23061        120
cpu7:timer                         11436         59
cpu8:timer                         26434        137
cpu9:timer                         24551        127
cpu10:timer                        20072        104
cpu11:timer                        20522        107
cpu12:timer                        17992         93
cpu13:timer                        19231        100
cpu14:timer                        18226         95
cpu15:timer                        19063         99
irq128: nvme0:admin                   39          0
irq129: nvme0:io0                    694          4
irq130: nvme0:io1                    378          2
irq131: nvme0:io2                    599          3
irq132: nvme0:io3                    436          2
irq133: nvme0:io4                    735          4
irq134: nvme0:io5                    543          3
irq135: nvme0:io6                    675          4
irq136: nvme0:io7                    369          2
irq137: nvme0:io8                    595          3
irq138: nvme0:io9                    857          4
irq139: nvme0:io10                   331          2
irq140: nvme0:io11                  1302          7
irq141: nvme0:io12                   425          2
irq142: nvme0:io13                   396          2
irq143: nvme0:io14                   363          2
irq144: nvme0:io15                   541          3
irq147: xhci1                        125          1
irq149: igc0:rxq0                 106842        554
irq150: igc0:rxq1                 149145        774
irq151: igc0:rxq2                 217711       1130
irq152: igc0:rxq3                 194107       1007
irq153: igc0:aq                        2          0
irq154: igc1:rxq0                 330733       1716
irq155: igc1:rxq1                 338179       1755
irq156: igc1:rxq2                 301578       1565
irq157: igc1:rxq3                 316481       1642
irq158: igc1:aq                        2          0
irq159: hdac0                         11          0
irq160: iwlwifi0                       8          0
irq175: iwlwifi0                       2          0
Total                            2278628      11826

.
My WAN interface errors were the following:

.

The hw.igc.max_interrupt_rate: 20000 setting is present in the /boot/loader.conf.local config file. I restarted the PFSense router after saving the file:

.
The Upload speeds are still capped at 140-150 Mbps. Download is unaffected.

stephenw10

Any IN errors on the LAN interface?

Have you tried setting it to 1G so it doesn't link at 2.5G?

steve10240

@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

Any IN errors on the LAN interface?

Have you tried setting it to 1G so it doesn't link at 2.5G?

Yes, I did see IN errors on the LAN interface as the Upload speed was being tested. The errors were increasing very rapidly:

VLAN 100 is the LAN where the speedtest traffic is being uploaded from:

I will try a 1G link enforcement of the WAN interface next.