Windstream fiber -> Adtran 411 -> PfSense slow upload speed
-
Are you using an SFP adapter? If so what brand? I just want to ask have you cleaned your connections with an approved fiber cloth or cleaning kit? Do you have a fiber scope? You can really inspect the connections with a scope too.
-
Does the Mellanox NIC link at 1G to the ONT? If it's trying to link to at 2.5G or 5G etc it could easily be causing issues.
I can't see exactly what the onboard NICs are on that server but they look like older Intel NICs that I'd expect to be well supported.
Are you running bare metal? That looks to be massively over powered for most pfSense installs. It should work though.
Steve
-
@nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
I get full synchronous speeds.
I believe you mean symmetrical.
-
@JonathanLee said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Are you using an SFP adapter? If so what brand? I just want to ask have you cleaned your connections with an approved fiber cloth or cleaning kit? Do you have a fiber scope? You can really inspect the connections with a scope too.
the fiber was JUST installed.. and speeds are fine when I take pfsense out of the picture.. also the brand of the rj45 sfp is ipolex
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Does the Mellanox NIC link at 1G to the ONT? If it's trying to link to at 2.5G or 5G etc it could easily be causing issues.
I can't see exactly what the onboard NICs are on that server but they look like older Intel NICs that I'd expect to be well supported.
Are you running bare metal? That looks to be massively over powered for most pfSense installs. It should work though.
Steve
when i try to force 1000baseT, it actually doesn't work.. just keeps going at the 10Gbase-CX4... the only thing I know about the NICs is that they are intel and seem to work fine in a debian live cd.
yes running bare metal.. i figured it would be OP but those hyve zeus are way cheaper than alternatives.. i think i picked that one up for $100.
As for an update.. it is indeed the mellanox NIC that windstream doesn't like. I tested a usb NIC and assigned it to WAN.. although slow.. they were symmetrical. I ordered a Intel X520-DA2 off ebay.. hopefully that fares better once it gets here.. Switched back to cox internet in the meantime.
@JKnott said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
@nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
I get full synchronous speeds.
I believe you mean symmetrical.
Thank you.. i'll be sure to use the correct term for now on.
-
Mmm, that 'feels' like a link issue with the Mellanox. Possibly a flow control mismatch. The X520 should work.
What do you see for on-board NICs from
pciconf -lvat the command line? -
@nynx64
@stephenw10
[https://www.reddit.com/r/PFSENSE/comments/w2ts56/frontier_fiber_vlan0/igsf66n/](link url)
Some isp require upload tag on vlan 0 or 1
Have you set the vlan tag and priority bit?
What is your wan vlan config? -
That's true but if it applied here it would also restrict a directly connected laptop.
-
I'm also facing this issue, but with a different equipment and ISP (Frontier Fiber ISP -> Nokia XS-010X-Q ONT -> PFSense Asrock NUC BOX-1360P/D5 Fanned BOX with dual i226 NICs). When I am connected directly to the Nokia XS-010X-Q ONT box, I get symmetrical speeds both down and up (1gbps). When my PFSense box WAN port is connected to the Fiber ISP ONT box, download stays at 1gbps but upload speeds tank to 140mbps.
I am curious what the solution ends up being for the OP. I had another router on hand, Edgerouter PoE4, and when that router is hooked up to the ONT, it also gets full 1gbps/1gbps for download and upload. I am trying to figure out what is preventing the PFSense box from reaching its full upload speed capability.
-
@fathead said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
@nynx64
@stephenw10
[https://www.reddit.com/r/PFSENSE/comments/w2ts56/frontier_fiber_vlan0/igsf66n/](link url)
Some isp require upload tag on vlan 0 or 1
Have you set the vlan tag and priority bit?
What is your wan vlan config?Didn't tag WAN with any vlan.. just set the interface and left the rest default.
I have a frankenstein PC i use for testing. I pulled the mellanox card and ssd out of the zeus and transplanted them into that.. except i'm using the onboard 1G for WAN.
So it's definitely the NIC. This should get me by till that intel card delivers sometime later.
-
@nynx64 I think it's my NIC too, but that is the last thing I haven't tried yet on the PFSense box. I tried placing a TrendNet 2.5G unmanaged switch between my Fiber ONT and PFSense router to see if that would resolve the upload speeds, but it didn't. I'm going to try a USB NIC.
I really thought there were some tunables that were needed to get the intel nics working right, but I don't know. Hopefully it works with your intel card!
-
An i226 NIC should be fine. Are you seeing errors on the NIC? Are you seeing multiple queues for bot Tx and Rx?
You might try setting
hw.igc.max_interrupt_rate: 20000though the default there should restrict anything at 140Mbps.Steve
-
@stephenw10 How do I check for the errors and the queues? Is that under diagnostics?
-
Errors would be shown in Status > Interfaces.
Queues are shown in the boot log whn the driver attaches but you can also check
vmstat -ito make sure they are being used when under load. -
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Errors would be shown in Status > Interfaces.
Queues are shown in the boot log whn the driver attaches but you can also check
vmstat -ito make sure they are being used when under load.While running the Upload portion of the speedtest, this is the output of the
vmstat -icommand from my PFSense router:interrupt total rate cpu0:timer 29037 151 cpu1:timer 13165 68 cpu2:timer 26479 137 cpu3:timer 12253 64 cpu4:timer 22329 116 cpu5:timer 10573 55 cpu6:timer 23061 120 cpu7:timer 11436 59 cpu8:timer 26434 137 cpu9:timer 24551 127 cpu10:timer 20072 104 cpu11:timer 20522 107 cpu12:timer 17992 93 cpu13:timer 19231 100 cpu14:timer 18226 95 cpu15:timer 19063 99 irq128: nvme0:admin 39 0 irq129: nvme0:io0 694 4 irq130: nvme0:io1 378 2 irq131: nvme0:io2 599 3 irq132: nvme0:io3 436 2 irq133: nvme0:io4 735 4 irq134: nvme0:io5 543 3 irq135: nvme0:io6 675 4 irq136: nvme0:io7 369 2 irq137: nvme0:io8 595 3 irq138: nvme0:io9 857 4 irq139: nvme0:io10 331 2 irq140: nvme0:io11 1302 7 irq141: nvme0:io12 425 2 irq142: nvme0:io13 396 2 irq143: nvme0:io14 363 2 irq144: nvme0:io15 541 3 irq147: xhci1 125 1 irq149: igc0:rxq0 106842 554 irq150: igc0:rxq1 149145 774 irq151: igc0:rxq2 217711 1130 irq152: igc0:rxq3 194107 1007 irq153: igc0:aq 2 0 irq154: igc1:rxq0 330733 1716 irq155: igc1:rxq1 338179 1755 irq156: igc1:rxq2 301578 1565 irq157: igc1:rxq3 316481 1642 irq158: igc1:aq 2 0 irq159: hdac0 11 0 irq160: iwlwifi0 8 0 irq175: iwlwifi0 2 0 Total 2278628 11826.
My WAN interface errors were the following:
.The
hw.igc.max_interrupt_rate: 20000setting is present in the/boot/loader.conf.localconfig file. I restarted the PFSense router after saving the file:
.
The Upload speeds are still capped at 140-150 Mbps. Download is unaffected. -
Any IN errors on the LAN interface?
Have you tried setting it to 1G so it doesn't link at 2.5G?
-
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Any IN errors on the LAN interface?
Have you tried setting it to 1G so it doesn't link at 2.5G?
Yes, I did see IN errors on the LAN interface as the Upload speed was being tested. The errors were increasing very rapidly:
VLAN 100 is the LAN where the speedtest traffic is being uploaded from:
I will try a 1G link enforcement of the WAN interface next.
-
Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?
You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.
-
@steve10240 that's got some packet loss I am seeing on errors. Have you checked the cables, cleaned the fiber connectors also?
-
Update:
Intel X520-DA2 delivered and installed. Same exact issue with download being fine but upload crawling; but this time (not sure why) pfsense is harping loudly about "Unsupported SFP+ module detected".
Are Inpolex ASF-10G-T not supported? If it was unsupported why didn't the mellanox tell me that? Or is it just unsupported on that intel card and unrelated to the root cause?
Can the ASF-10G-T not negotiation down to 1000base-t? Anyone have a similar setup working with a sfp -> rj45?
Should I try to pickup a sfp module that specifically says it can do it?
I'm kind of at a loss here.
-
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?
You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.
I tried limiting the WAN interface to 1G and my download speed is no longer at 1Gbps:
The LAN side is auto-linked at 1gbps and yes it is a switch. The switch is connected to the router on port 0/1. Here are the interface error counters I observed after running show interface errors:
However, running more speed tests did not increment the error numbers for interface 0/1 on the Edgeswitch Lite 24.For your suggestion, like, directly connecting my laptop to the PFSense router LAN interface, bypassing the switch? I will give that a try and report back.
