gigabit wan download speed much lower than upload



  • Hi all, I'm having no luck figuring out why my Verizon fios symmetrical Gigabit WAN is getting such low download speeds on my pfsense box.
    Speedtest.net is showing consistently around 5ms ping, 50-70Mbps download speeds vs. 800-900Mbps upload speeds.

    I have a new pfsense setup 2.4.4-release-p3
    Running on a HP T620 Plus 16GB M.2 SSD, 4GB of RAM, with a HP NC364T PCI Express Quad Port Gigabit Server Adapter. (Intel chipset, em0 wan, em1 lan interfaces)

    I'm only using two of the four ports on the nic for WAN/LAN.
    Brand new CAT6 cables on port WAN/LAN.
    A 24 port Netgear JGS524 switch for LAN devices.

    I have disabled:
    Hardware Checksum Offloading
    Hardware TCP Segmentation Offloading
    Hardware Large Receive Offloading

    And after some googling decided to add these change to /boot/loader.conf.local
    kern.ipc.nmbclusters="1000000"
    hw.em.num_queues=0
    hw.em.fc_setting=0
    hw.em.rxd=4096
    hw.em.txd=4096

    I have 3 openvpn connections open to iVPN in a gateway group that by default are allowed to NAT outbound to the internet. And I've added an alias "nonvpnd" for a group of LAN ip's that passes these matched ip's to the WAN gateway and does not pass over the ivpn gateway group. In other words all internet bound traffic from my LAN 192.168.137.X goes over the iVPN gateway openvpn connections, unless the source IP matches one in the "nonvpnd" alias, then it gets NAT'd through the WAN/fios connection IP to the internet.

    I also have installed pfblocker-ng dev version. And have tried the speed tests with it on and disabled in every place. And no change in download speed tests.

    CPU usage during a speed test never goes past 20%, RAM usage at 13%
    https://www.verizon.com/speedtest/ shows 15Mpbs download 935Mpbs upload to device. 935 Download to router and fails on upload. Which I've read is because they try to test upload from the Wi-Fi/Router box I took out of the mix and going straight from the ONT with CAT6 to the HP T620 pfsense box.

    I"m baffled at this point what is causing such a huge drop in the download speeds.
    That speed disparity is HUGE! 50Mbps vs. 900Mbps leads me to believe that nothing is wrong with my fios service. And that something is wrong in pfsense's configuration that is crippling the download speeds from the internet.

    Any help anyone could give would be so greatly appreciated! I"m throwing my money away on this speed that I'm not getting.


  • Netgate Administrator

    What CPU is in that?

    Try running top -aSH at the console while you're testing to see how the load is spread across the CPU cores.

    That's a DHCP WAN connection I assume?

    Have you tested with a client connected directly, without pfSense, and can see the full download speed?

    Steve



  • Agree with @stephenw10 - if you connect directly to the ONT, do you see full up and down speeds? That would be the first thing to check to make sure it isn't a provisioning or other issue on Verizon's end.





  • The HP T620 Plus is a thin client PC I've been told was great for a small, low cost, low power, silent pfsense box.
    CPU Type AMD GX-420CA SOC with Radeon(tm) HD Graphics
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    2Ghz

    I tried a speed test with 2 different laptops gigabit Ethernet directly to the ONT and unexpectedly still got the same speeds, worse on subsequent tests.

    Now I'm questioning my fios service quality.

    Before I used the T620 I'd been using a virtualized pfsense install on esxi but wasn't satisfied with speeds.
    And I'd been using the WAN nic with MAC clone of the Verizon provided Wi-Fi box I don't use anymore.

    Do you think that MAC could impact speeds at all? With provisioning?

    Thanks for the input everyone.


  • Netgate Administrator

    Some providers will massively limit the speeds you can get if you are not connected "properly". Such as having to use the vlan or priority tags on outgoing packets. Though I didn't think FiOS did that.

    I would have them reset the ONT or at least check the connection, there's clearly something wrong there if a laptop sees the same speeds.

    Steve



  • I assume you also tried a different cable just in case? Since you can't get full speed connected directly to the ONT, I think you should call Verizon as something doesn't seem be configured properly on their end, or maybe there is an issue with your fiber signal level/quality, etc.

    I have been using Verizon FIOS gigabit service together w/pfSense for over two years and have always gotten full speeds - there is really nothing special you have to do to set it up (e.g. no VLAN's, priority tags, gateway bridging, MAC cloning, etc.) besides maybe a bit of network hardware tuning for high speeds.

    Hope this helps.


  • Netgate Administrator

    Worth checking if your WAN connection linked at 1Gb full duplex. Check the Status > Interfaces page for errors.

    Steve



  • @stephenw10 said in gigabit wan download speed much lower than upload:

    Worth checking if your WAN connection linked at 1Gb full duplex.

    Given he's seeing 800-800 Mb in one direction, the interface is definitely running at 1 Gb. Also, Gb is always full duplex.


  • Netgate Administrator

    Yup it should always be full duplex and it should always be autonegotiate. But we have seen equipment that doesn't stick to that. Though I agree it would have to be something more like a hardware issue to get such asymmetric speeds.

    Steve



  • Well I've taken another look at speeds directly to the ONT
    Started with a power reset of the ONT.
    Then with a new cable and laptop, verified I got 1 Gbps Full Duplex link up.
    Ran speed test through speedtest.net
    Got back PING ms
    7
    DOWNLOAD Mbps
    141.84
    UPLOAD Mbps
    242.88

    Then on verizon.com/speedtest
    DOWNLOAD
    405
    Mbps

    UPLOAD
    476
    Mbps

    both of these are terrible, and indicate something still ins't right hardware wise or provisioning?

    I guess I need to call Verizon now.


  • Netgate Administrator

    Um, yeah that looks.... sub-optimal!

    Seems like an upstream issue.

    Steve



  • @lasermole said in gigabit wan download speed much lower than upload:

    Then with a new cable and laptop, verified I got 1 Gbps Full Duplex link up.

    Was that with the computer by itself? Or through pfSense? Try the computer connected directly, to see if the problem is with pfSense or the ISP. It's best to keep things simple while testing.



  • Yes the tests were with the laptop directly connected to the ONT with new cable and I got those results...



  • @lasermole said in gigabit wan download speed much lower than upload:

    Yes the tests were with the laptop directly connected to the ONT with new cable and I got those results...

    Then it's not a pfSense issue. Call your ISP.



  • Yup, good times with ol Verizon....



  • Just curious what you guys think, but do you think Verizon has reason to dismiss a speed test done directly to their equipment? Dismissing your results, pointing blame at your device. Hence why I'm sure verizon's own speed test is the only one they officially recognize, as it separates "Device" and "Router" in its test. FYI, I don't expect to get 1000/1000, even 800/800 would be fine by me to device.



  • @lasermole - do you still have Verizon router/gateway? If yes, it might be worth plugging it in temporarily and run the Verizon speed test to the router. Given what you have described so far, I doubt that even that test will actually see full speed. If it doesn't, I expect them to help your further since the issue is then more clearly somewhere with their equipment.



  • Yes I do still have the router/gateway device. I could hook it up again and try through that to see what speeds I get. I'll let you know what comes of that test.



  • Tried the speed tests again with the Verizon router in place between the laptop and the ONT.

    Laptop > Verizon router > ONT
    Speedtest.net
    PING ms
    7
    DOWNLOAD Mbps
    256.93
    UPLOAD Mbps
    207.28

    Verizon speed test
    "Device"
    DOWNLOAD
    568
    Mbps

    UPLOAD
    382
    Mbps

    "Router"
    DOWNLOAD
    949
    Mbps

    UPLOAD
    912
    Mbps

    I really feel that "router" test is bull. I mean I don't pay Verizon each month to read how fast their devices are, to their network, on a screen...
    That is not end user usable bandwidth. And I'm sure they will use that result to silence my complaints and shift the blame to me any way they see fit.



  • @lasermole - do you have any other device (besides the laptop) you can test with? What are the specs of the device (laptop) you are testing with? Any third party firewall / AV etc. software running in the background which may be slowing things down?



  • I've tested with two laptops, a Lenovo T520 i5 with 16GB of RAM, and 120GB SSD and onboard intel gigabit nic.
    And a Dell Precision 5530 with an i7 8850H 6 Core with hyper threading. Used a USB 3 Gigabit nic, as it doesn't have onboard.
    Both of which I went through thorough checking to kill anything running that would use CPU power, or bandwidth.
    The T520 had no firewall, or AV running during the test.
    The Dell has Carbon Black AV/Firewall on, its my work laptop.
    And each test the laptop was the only thing connected to the ONT, or the Verizon router.
    And to note I disabled the Wi-Fi on the laptops, and on the Verizon router from day 1. And I don't ever use the Verizon router. I only brought it in to test everything after i setup pfSense.

    Both laptops tested poorly in speed tests. as shown above.
    I have contacted Verzion, and will add another post with those results.



  • So I got Verizon on the phone yesterday morning.
    I explained I was experiencing poor speeds on my Gigabit service.

    The tech immediately asked if I was using my own router, and explained they can't guarantee speeds to any device but theirs. I continue appeasing them and attach the Verizon router to the ONT, then my Dell laptop wired to the router directly, and we proceed to verizon.net/speedtest to do their speed test and get:

    Device (check)
    DOWNLOAD
    642
    Mbps

    UPLOAD
    767
    Mbps

    Router
    DOWNLOAD
    947
    Mbps

    UPLOAD
    911
    Mbps

    This still seemed kinda crappy, but I realized since they only care about the "router" speeds, and that absolves them of all wrongdoing with network on their side I accepted it and said "you can't help me any further thank you"

    Now I plugged my pfSense box back up as I normally have it direct to the ONT.
    And I did another speed test with the Dell laptop again...and mysteriously I got the best speed test I've got yet from Verizon's site with normal network hookup for me:

    Device (good)
    Download
    866

    Upload
    766

    So now I'm leaving things as they are. To note I do have pfsense WAN interface cloning the Verizon router MAC, but I already had that before. So I wonder if something just "clicked" into place now, or if Verizon changed anything silently? I made no new configuration changes to pfsense during this process.


  • Netgate Administrator

    Would not be the first time a connection mysteriously started working after the ISP claimed to have found no fault. 🙄

    Steve



  • Funny now my speeds are crap on download again.
    And I'm sure nothing is eating all the bandwidth during my test.
    (tested through pfsense router )

    Is there anything that could "degrade" speeds over time in pfsense? anything that builds up or needs cleaning? buffers?

    Verizon speed test shows:
    DOWNLOAD
    18
    Mbps

    UPLOAD
    816
    Mbps


  • LAYER 8 Netgate

    @lasermole said in gigabit wan download speed much lower than upload:

    Is there anything that could "degrade" speeds over time in pfsense? anything that builds up or needs cleaning? buffers?

    Nope.


  • Netgate Administrator

    And still bad when putting the Verizon router back in line? Or testing directly with a single client?

    Steve


Log in to reply