Windstream fiber -> Adtran 411 -> PfSense slow upload speed
-
@stephenw10 How do I check for the errors and the queues? Is that under diagnostics?
-
Errors would be shown in Status > Interfaces.
Queues are shown in the boot log whn the driver attaches but you can also check
vmstat -ito make sure they are being used when under load. -
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Errors would be shown in Status > Interfaces.
Queues are shown in the boot log whn the driver attaches but you can also check
vmstat -ito make sure they are being used when under load.While running the Upload portion of the speedtest, this is the output of the
vmstat -icommand from my PFSense router:interrupt total rate cpu0:timer 29037 151 cpu1:timer 13165 68 cpu2:timer 26479 137 cpu3:timer 12253 64 cpu4:timer 22329 116 cpu5:timer 10573 55 cpu6:timer 23061 120 cpu7:timer 11436 59 cpu8:timer 26434 137 cpu9:timer 24551 127 cpu10:timer 20072 104 cpu11:timer 20522 107 cpu12:timer 17992 93 cpu13:timer 19231 100 cpu14:timer 18226 95 cpu15:timer 19063 99 irq128: nvme0:admin 39 0 irq129: nvme0:io0 694 4 irq130: nvme0:io1 378 2 irq131: nvme0:io2 599 3 irq132: nvme0:io3 436 2 irq133: nvme0:io4 735 4 irq134: nvme0:io5 543 3 irq135: nvme0:io6 675 4 irq136: nvme0:io7 369 2 irq137: nvme0:io8 595 3 irq138: nvme0:io9 857 4 irq139: nvme0:io10 331 2 irq140: nvme0:io11 1302 7 irq141: nvme0:io12 425 2 irq142: nvme0:io13 396 2 irq143: nvme0:io14 363 2 irq144: nvme0:io15 541 3 irq147: xhci1 125 1 irq149: igc0:rxq0 106842 554 irq150: igc0:rxq1 149145 774 irq151: igc0:rxq2 217711 1130 irq152: igc0:rxq3 194107 1007 irq153: igc0:aq 2 0 irq154: igc1:rxq0 330733 1716 irq155: igc1:rxq1 338179 1755 irq156: igc1:rxq2 301578 1565 irq157: igc1:rxq3 316481 1642 irq158: igc1:aq 2 0 irq159: hdac0 11 0 irq160: iwlwifi0 8 0 irq175: iwlwifi0 2 0 Total 2278628 11826.
My WAN interface errors were the following:
.The
hw.igc.max_interrupt_rate: 20000setting is present in the/boot/loader.conf.localconfig file. I restarted the PFSense router after saving the file:
.
The Upload speeds are still capped at 140-150 Mbps. Download is unaffected. -
Any IN errors on the LAN interface?
Have you tried setting it to 1G so it doesn't link at 2.5G?
-
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Any IN errors on the LAN interface?
Have you tried setting it to 1G so it doesn't link at 2.5G?
Yes, I did see IN errors on the LAN interface as the Upload speed was being tested. The errors were increasing very rapidly:
VLAN 100 is the LAN where the speedtest traffic is being uploaded from:
I will try a 1G link enforcement of the WAN interface next.
-
Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?
You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.
-
@steve10240 that's got some packet loss I am seeing on errors. Have you checked the cables, cleaned the fiber connectors also?
-
Update:
Intel X520-DA2 delivered and installed. Same exact issue with download being fine but upload crawling; but this time (not sure why) pfsense is harping loudly about "Unsupported SFP+ module detected".
Are Inpolex ASF-10G-T not supported? If it was unsupported why didn't the mellanox tell me that? Or is it just unsupported on that intel card and unrelated to the root cause?
Can the ASF-10G-T not negotiation down to 1000base-t? Anyone have a similar setup working with a sfp -> rj45?
Should I try to pickup a sfp module that specifically says it can do it?
I'm kind of at a loss here.
-
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?
You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.
I tried limiting the WAN interface to 1G and my download speed is no longer at 1Gbps:
The LAN side is auto-linked at 1gbps and yes it is a switch. The switch is connected to the router on port 0/1. Here are the interface error counters I observed after running show interface errors:
However, running more speed tests did not increment the error numbers for interface 0/1 on the Edgeswitch Lite 24.For your suggestion, like, directly connecting my laptop to the PFSense router LAN interface, bypassing the switch? I will give that a try and report back.
-
@nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
is it just unsupported on that intel card and unrelated to the root cause?
Probably. Though if your WAN is only 1G I would try a 1G RJ-45 adapter. You might also try setting the loader value:
hw.ix.unsupported_sfp=1 -
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
@nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
is it just unsupported on that intel card and unrelated to the root cause?
Probably. Though if your WAN is only 1G I would try a 1G RJ-45 adapter. You might also try setting the loader value:
hw.ix.unsupported_sfp=1It stopped griping about unsupported sfps but still no bueno.
Guess i'll go looking for a 1g sfp. I know eventually they will be pushing 2 and 5 gig, which is why I was trying to get this sfp to work. Ready for the upgrade etc.
-
Still in the same motherboard? Can you test that NIC in the other PC you set up.
Something there is very wrong to see that sort of throttling. Those are the sort of speeds you see when something is falling back to half-duplex for example.
Check for a flow-control mismatch. I have seen links where it needed to be set off and also where is needed to be set on
-
It was the sfp. I purchased https://www.amazon.com/dp/B00U77IJ2M, which delivered today.
Running fine back on the mellanox. Gonna hang on to the intel card in case this one dies I guess. It was rather cheap.
Thank you all for the help troubleshooting.
-
I was thinking it was the cable or connectors/SFP. At my old job we would have to clean the connectors on Ciena backbone routers and also with lots of other fiber optic routers anytime we touched them. Ciena they had monster MPO cables on them. We even had to photograph the digital scope screen with the cable tips after we cleaned them to put in the work orders as proof we cleaned and checked them. It's that important. A lot of problems we had came back to the cable or connectors as the DB loss at that point is very high. I bet that old SFP is really dirty inside it. We had this installer once who left french fry salt and fry oil all over a cable tip by accident and just jammed it in. That guy was over worked. We had to go back and change the SFP out, and clean all the cables on it.
For my Cisco exam for that job we also had to know all the fiber optic equations for their fiber optic routers too. I was lucky and got to get good training on alot of fiber optic equipment over the many years at that job.
This is a great reference for equations:
https://www.cisco.com/c/en/us/support/docs/optical/synchronous-digital-hierarchy-sdh/29000-db-29000.htmlLong story short make sure you clean the connectors with a fiber cleaning kit. Amazon has really affordable ones that work good. You going to need one eventually. Cisco had this press button connector cleaner one that you just held the cable to and click it would drag the cloth over the tips. I still wonder why it said it was unsupported but it still could use it
. -
@steve10240 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?
You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.
I tried limiting the WAN interface to 1G and my download speed is no longer at 1Gbps:
The LAN side is auto-linked at 1gbps and yes it is a switch. The switch is connected to the router on port 0/1. Here are the interface error counters I observed after running show interface errors:
However, running more speed tests did not increment the error numbers for interface 0/1 on the Edgeswitch Lite 24.For your suggestion, like, directly connecting my laptop to the PFSense router LAN interface, bypassing the switch? I will give that a try and report back.
@stephenw10 @JonathanLee
I had to wait until the weekend to take down the Internet at this site for testing. I tried the following interface setups on the Pfsense firewall:-
WAN interface on USB-C NIC and Laptop NIC directly connected to integrated i226 LAN, no vlans
-
WAN Interface on integrated i226 and laptop connected to 2nd integrated i226 LAN, no vlans
In both tests, I was able to pull the 900mbps download and 900mbps upload. When I put back the VLANs on the LAN interface and do intervlan routing with the pfsense firewall, the speed drops back to 140mbps max. I wonder why it's only affecting the upload speed. When the router is intervlan routing, that's where I see the interface errors start going up.
-
-
I'd have to guess it's a bad vlan config somewhere. Since it's upload only I'd look at a PVID conflict.
Could also be an MTU issue since adding the VLAN tags increases that.
-
@stephenw10 Would I configure those changes on the pfSense router point of view or on the switch that is connecting the computers to pfSense?
In regards to the PVID conflict, does this mean something that's wrong with the trunkport that connects the switch and pfsense together?
-
It could be a mismtach between the switch and pfSense but unless you have set something weird in pfSense it should 'just work'. I'd double check the switch config and/or try a different switch if you can.
-
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
It could be a mismtach between the switch and pfSense but unless you have set something weird in pfSense it should 'just work'. I'd double check the switch config and/or try a different switch if you can.
@stephenw10 I am getting a new switch which is arriving today and will test with that.
I will set the port that connects the pfSense system and the switch as a trunk that allows VLANS 20, 40, 50, 99 and 100 and will match that config on pfsense with the parent interface, igc1, unassigned and create matching VLAN sub-interfaces. I hope it works this time with a different switch.
-
@steve10240 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
@stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:
It could be a mismtach between the switch and pfSense but unless you have set something weird in pfSense it should 'just work'. I'd double check the switch config and/or try a different switch if you can.
@stephenw10 I am getting a new switch which is arriving today and will test with that.
I will set the port that connects the pfSense system and the switch as a trunk that allows VLANS 20, 40, 50, 99 and 100 and will match that config on pfsense with the parent interface, igc1, unassigned and create matching VLAN sub-interfaces. I hope it works this time with a different switch.
@stephenw10 I checked the VLAN configs on the EdgeSwitch and everything looks okay. I received a new TrendNET TEG-S350 unmanaged switch and it was connected in the following way:
Fiber ISP ---> TEG-S350 2.5G switch ---> PFSense Asrock NUC BOX-1360P/D5 ---> EdgeSwitch Lite 24
I am still getting the same issues with slow upload speed and In/Out errors increasing. In addition, now the PFSense router is having trouble keeping the 2.5G link on its interface in an UP state. The WAN port connected to the switch and the switch leading into the Fiber ONT, it keeps going through cycles of UP and DOWN states every couple seconds. This was also happening before the new TrendNET switch was introduced. This has been causing a lot of headache for the users where I placed this PFSense router. I put back the original router equipment - the Edgerouter POE5 - and I am once again getting full speeds in both Up/Down directions with no errors to report.
I called my ISP and they kept telling me that my router is the issue. This NUC does have the Intel I226 NIC, which I read online still has some problems keeping a link and for others, its working just fine. I'm not sure what is causing me so much trouble, but it has only gotten worse with time. Thank you for the troubleshooting tips you provided me to try.
