Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Windstream fiber -> Adtran 411 -> PfSense slow upload speed

    Scheduled Pinned Locked Moved General pfSense Questions
    39 Posts 6 Posters 3.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      An i226 NIC should be fine. Are you seeing errors on the NIC? Are you seeing multiple queues for bot Tx and Rx?

      You might try setting hw.igc.max_interrupt_rate: 20000 though the default there should restrict anything at 140Mbps.

      Steve

      S 1 Reply Last reply Reply Quote 0
      • S
        steve10240 @stephenw10
        last edited by

        @stephenw10 How do I check for the errors and the queues? Is that under diagnostics?

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Errors would be shown in Status > Interfaces.

          Queues are shown in the boot log whn the driver attaches but you can also check vmstat -i to make sure they are being used when under load.

          S 1 Reply Last reply Reply Quote 1
          • S
            steve10240 @stephenw10
            last edited by

            @stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

            Errors would be shown in Status > Interfaces.

            Queues are shown in the boot log whn the driver attaches but you can also check vmstat -i to make sure they are being used when under load.

            While running the Upload portion of the speedtest, this is the output of the vmstat -i command from my PFSense router:

            interrupt                          total       rate
            cpu0:timer                         29037        151
            cpu1:timer                         13165         68
            cpu2:timer                         26479        137
            cpu3:timer                         12253         64
            cpu4:timer                         22329        116
            cpu5:timer                         10573         55
            cpu6:timer                         23061        120
            cpu7:timer                         11436         59
            cpu8:timer                         26434        137
            cpu9:timer                         24551        127
            cpu10:timer                        20072        104
            cpu11:timer                        20522        107
            cpu12:timer                        17992         93
            cpu13:timer                        19231        100
            cpu14:timer                        18226         95
            cpu15:timer                        19063         99
            irq128: nvme0:admin                   39          0
            irq129: nvme0:io0                    694          4
            irq130: nvme0:io1                    378          2
            irq131: nvme0:io2                    599          3
            irq132: nvme0:io3                    436          2
            irq133: nvme0:io4                    735          4
            irq134: nvme0:io5                    543          3
            irq135: nvme0:io6                    675          4
            irq136: nvme0:io7                    369          2
            irq137: nvme0:io8                    595          3
            irq138: nvme0:io9                    857          4
            irq139: nvme0:io10                   331          2
            irq140: nvme0:io11                  1302          7
            irq141: nvme0:io12                   425          2
            irq142: nvme0:io13                   396          2
            irq143: nvme0:io14                   363          2
            irq144: nvme0:io15                   541          3
            irq147: xhci1                        125          1
            irq149: igc0:rxq0                 106842        554
            irq150: igc0:rxq1                 149145        774
            irq151: igc0:rxq2                 217711       1130
            irq152: igc0:rxq3                 194107       1007
            irq153: igc0:aq                        2          0
            irq154: igc1:rxq0                 330733       1716
            irq155: igc1:rxq1                 338179       1755
            irq156: igc1:rxq2                 301578       1565
            irq157: igc1:rxq3                 316481       1642
            irq158: igc1:aq                        2          0
            irq159: hdac0                         11          0
            irq160: iwlwifi0                       8          0
            irq175: iwlwifi0                       2          0
            Total                            2278628      11826
            

            .
            My WAN interface errors were the following:
            interface-error.png
            .

            The hw.igc.max_interrupt_rate: 20000 setting is present in the /boot/loader.conf.local config file. I restarted the PFSense router after saving the file:

            conflocal-settings.png
            .
            The Upload speeds are still capped at 140-150 Mbps. Download is unaffected.

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Any IN errors on the LAN interface?

              Have you tried setting it to 1G so it doesn't link at 2.5G?

              S 1 Reply Last reply Reply Quote 0
              • S
                steve10240 @stephenw10
                last edited by

                @stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                Any IN errors on the LAN interface?

                Have you tried setting it to 1G so it doesn't link at 2.5G?

                Yes, I did see IN errors on the LAN interface as the Upload speed was being tested. The errors were increasing very rapidly:
                err_lan.png

                VLAN 100 is the LAN where the speedtest traffic is being uploaded from:
                err-vlan100.png

                I will try a 1G link enforcement of the WAN interface next.

                JonathanLeeJ 1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?

                  You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.

                  S 1 Reply Last reply Reply Quote 0
                  • JonathanLeeJ
                    JonathanLee @steve10240
                    last edited by

                    @steve10240 that's got some packet loss I am seeing on errors. Have you checked the cables, cleaned the fiber connectors also?

                    Make sure to upvote

                    N 1 Reply Last reply Reply Quote 0
                    • N
                      nynx64 @JonathanLee
                      last edited by nynx64

                      Update:

                      Intel X520-DA2 delivered and installed. Same exact issue with download being fine but upload crawling; but this time (not sure why) pfsense is harping loudly about "Unsupported SFP+ module detected".

                      Are Inpolex ASF-10G-T not supported? If it was unsupported why didn't the mellanox tell me that? Or is it just unsupported on that intel card and unrelated to the root cause?

                      Can the ASF-10G-T not negotiation down to 1000base-t? Anyone have a similar setup working with a sfp -> rj45?

                      Should I try to pickup a sfp module that specifically says it can do it?

                      80364c7c-dbdc-4ae5-83ea-77b26163fcd8-image.png

                      I'm kind of at a loss here.

                      stephenw10S 1 Reply Last reply Reply Quote 0
                      • S
                        steve10240 @stephenw10
                        last edited by

                        @stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                        Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?

                        You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.

                        I tried limiting the WAN interface to 1G and my download speed is no longer at 1Gbps:
                        a112c630-ba1c-44e5-804c-c949e72287f0-image.png

                        The LAN side is auto-linked at 1gbps and yes it is a switch. The switch is connected to the router on port 0/1. Here are the interface error counters I observed after running show interface errors:
                        aa5583fe-6f94-4d2b-9984-de198d9d2225-image.png
                        However, running more speed tests did not increment the error numbers for interface 0/1 on the Edgeswitch Lite 24.

                        For your suggestion, like, directly connecting my laptop to the PFSense router LAN interface, bypassing the switch? I will give that a try and report back.

                        N S 2 Replies Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator @nynx64
                          last edited by

                          @nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                          is it just unsupported on that intel card and unrelated to the root cause?

                          Probably. Though if your WAN is only 1G I would try a 1G RJ-45 adapter. You might also try setting the loader value: hw.ix.unsupported_sfp=1

                          1 Reply Last reply Reply Quote 0
                          • N
                            nynx64 @steve10240
                            last edited by

                            @stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                            @nynx64 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                            is it just unsupported on that intel card and unrelated to the root cause?

                            Probably. Though if your WAN is only 1G I would try a 1G RJ-45 adapter. You might also try setting the loader value: hw.ix.unsupported_sfp=1

                            It stopped griping about unsupported sfps but still no bueno.

                            7ef87ff3-a895-44ae-8ba8-8d7ac68d6359-image.png

                            6c109fc7-1dcf-4b39-a0b0-2b53994a00f8-image.png

                            Guess i'll go looking for a 1g sfp. I know eventually they will be pushing 2 and 5 gig, which is why I was trying to get this sfp to work. Ready for the upgrade etc.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by stephenw10

                              Still in the same motherboard? Can you test that NIC in the other PC you set up.

                              Something there is very wrong to see that sort of throttling. Those are the sort of speeds you see when something is falling back to half-duplex for example.

                              Check for a flow-control mismatch. I have seen links where it needed to be set off and also where is needed to be set on

                              N 1 Reply Last reply Reply Quote 0
                              • N
                                nynx64 @stephenw10
                                last edited by

                                @stephenw10

                                It was the sfp. I purchased https://www.amazon.com/dp/B00U77IJ2M, which delivered today.

                                Running fine back on the mellanox. Gonna hang on to the intel card in case this one dies I guess. It was rather cheap.

                                f995ec5a-5286-4e8c-a4e0-f14371da9ca7-image.png

                                Thank you all for the help troubleshooting.

                                1 Reply Last reply Reply Quote 2
                                • JonathanLeeJ
                                  JonathanLee
                                  last edited by JonathanLee

                                  I was thinking it was the cable or connectors/SFP. At my old job we would have to clean the connectors on Ciena backbone routers and also with lots of other fiber optic routers anytime we touched them. Ciena they had monster MPO cables on them. We even had to photograph the digital scope screen with the cable tips after we cleaned them to put in the work orders as proof we cleaned and checked them. It's that important. A lot of problems we had came back to the cable or connectors as the DB loss at that point is very high. I bet that old SFP is really dirty inside it. We had this installer once who left french fry salt and fry oil all over a cable tip by accident and just jammed it in. That guy was over worked. We had to go back and change the SFP out, and clean all the cables on it.

                                  For my Cisco exam for that job we also had to know all the fiber optic equations for their fiber optic routers too. I was lucky and got to get good training on alot of fiber optic equipment over the many years at that job.

                                  This is a great reference for equations:
                                  https://www.cisco.com/c/en/us/support/docs/optical/synchronous-digital-hierarchy-sdh/29000-db-29000.html

                                  Long story short make sure you clean the connectors with a fiber cleaning kit. Amazon has really affordable ones that work good. You going to need one eventually. Cisco had this press button connector cleaner one that you just held the cable to and click it would drag the cloth over the tips. I still wonder why it said it was unsupported but it still could use it 🤔.

                                  Make sure to upvote

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    steve10240 @steve10240
                                    last edited by

                                    @steve10240 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                                    @stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                                    Is the LAN side auto linked at 1G? What's it actually connected to? If it's a switch are there errors shown there?

                                    You might try an internal interface on a different port with a client connected to it directly to eliminate anything the switch or vlans might be doing.

                                    I tried limiting the WAN interface to 1G and my download speed is no longer at 1Gbps:
                                    a112c630-ba1c-44e5-804c-c949e72287f0-image.png

                                    The LAN side is auto-linked at 1gbps and yes it is a switch. The switch is connected to the router on port 0/1. Here are the interface error counters I observed after running show interface errors:
                                    aa5583fe-6f94-4d2b-9984-de198d9d2225-image.png
                                    However, running more speed tests did not increment the error numbers for interface 0/1 on the Edgeswitch Lite 24.

                                    For your suggestion, like, directly connecting my laptop to the PFSense router LAN interface, bypassing the switch? I will give that a try and report back.

                                    @stephenw10 @JonathanLee
                                    I had to wait until the weekend to take down the Internet at this site for testing. I tried the following interface setups on the Pfsense firewall:

                                    1. WAN interface on USB-C NIC and Laptop NIC directly connected to integrated i226 LAN, no vlans

                                    2. WAN Interface on integrated i226 and laptop connected to 2nd integrated i226 LAN, no vlans

                                    In both tests, I was able to pull the 900mbps download and 900mbps upload. When I put back the VLANs on the LAN interface and do intervlan routing with the pfsense firewall, the speed drops back to 140mbps max. I wonder why it's only affecting the upload speed. When the router is intervlan routing, that's where I see the interface errors start going up.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      I'd have to guess it's a bad vlan config somewhere. Since it's upload only I'd look at a PVID conflict.

                                      Could also be an MTU issue since adding the VLAN tags increases that.

                                      S 1 Reply Last reply Reply Quote 0
                                      • S
                                        steve10240 @stephenw10
                                        last edited by

                                        @stephenw10 Would I configure those changes on the pfSense router point of view or on the switch that is connecting the computers to pfSense?

                                        In regards to the PVID conflict, does this mean something that's wrong with the trunkport that connects the switch and pfsense together?

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          It could be a mismtach between the switch and pfSense but unless you have set something weird in pfSense it should 'just work'. I'd double check the switch config and/or try a different switch if you can.

                                          S 1 Reply Last reply Reply Quote 0
                                          • S
                                            steve10240 @stephenw10
                                            last edited by

                                            @stephenw10 said in Windstream fiber -> Adtran 411 -> PfSense slow upload speed:

                                            It could be a mismtach between the switch and pfSense but unless you have set something weird in pfSense it should 'just work'. I'd double check the switch config and/or try a different switch if you can.

                                            @stephenw10 I am getting a new switch which is arriving today and will test with that.

                                            I will set the port that connects the pfSense system and the switch as a trunk that allows VLANS 20, 40, 50, 99 and 100 and will match that config on pfsense with the parent interface, igc1, unassigned and create matching VLAN sub-interfaces. I hope it works this time with a different switch.

                                            S 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.