PfSense low throughput



  • Hello,
    I recently upgraded my pfSense box from a Intel Celeron 1.2Ghz to an Intel(R) Atom(TM) CPU D2700 @ 2.13GHz (2Cores with Hyper Threading) because the load went skydiving after I upgraded my fiber connection to 500/500Mbit.

    The new pfSense box is freshly installed with the newest 64bit pfSense, has 1 onboard Gbit card and 1 Gbit Intel Dual Head servercard.

    I use one intel port for WAN, the other for LAN (to my servers) and the onboard for OPT1 to my local network.

    It seems pfSense still has issues with lack of throughput. I have tried a second PC (i5, 15GB DDR3 with a PCIe Quad port intel (Gbit)) but got almost exactly the same results.
    First I suspected my Fiber connection was maybe not optimal, also tried different short CAT6 cables but the speed never went above 366Mbit.
    The load on this box when speedtesting almost never gets over 0.3.

    But the thing is when I connect my Mac book pro I get a much better 440Mbit.  :o

    System information:
    Version 2.1.5-RELEASE  (amd64)
    FreeBSD 8.3-RELEASE-p16
    CPU Type Intel(R) Atom(TM) CPU D2700 @ 2.13GHz 4 CPUs: 1 package(s) x 2 core(s) x 2 HTT threads.

    RAM: 4GB DDR 1066Mhz.  (SO-Dimm)

    Disk: SSD 60GB Kingston  (connected via SATA3)

    Network:
    10/100/1000 dual port Intel server NIC.
    10/100/1000 Realtec NIC.

    vmstat output:
    [2.1.5-RELEASE][root@gateway]/root(4): vmstat -i
    interrupt                          total      rate
    irq19: uhci1+                      8338          4
    irq21: em0                        760398        382
    irq22: em1                        350770        176
    cpu0: timer                      3974679      1997
    irq256: re0                      708337        355
    cpu2: timer                      3974614      1997
    cpu1: timer                      3974613      1997
    cpu3: timer                      3974613      1997
    Total                          17726362      8907

    Anyone has a clue what is slowing down this machine?

    Would be much appreciated.



  • Go into System: Advanced: Networking

    In Network Interfaces section disable (check) Hardware Checksum Offloading.  This should fix your low throughput issue.

    If not then you may have to play with the other 2 options.. Hardware TCP Segmentation Offloading & Hardware Large Receive Offloading

    Atom will not be able to handle 500Mbit. i5 should easily do it.



  • How are you speed testing? I don't have a 400mb+ connection with which to play with, but with iperf, I can easily reach the limit of my two test machine's 1.3gb/s with less than 5% cpu load, through firewall+NAT.


  • Netgate Administrator

    The D525 will pass 500Mbps. Some have managed >600Mbps with some tuning. I'd be surprised if the D2700 couldn't manage 500Mbps. It won't do 1Gbps though and that's without any packages running.

    Steve



  • Thanks for the tip. unfortunatly my speed drops to a very poor 70Mbit..  :'(

    @asterix:

    Go into System: Advanced: Networking

    In Network Interfaces section disable (check) Hardware Checksum Offloading.  This should fix your low throughput issue.

    If not then you may have to play with the other 2 options.. Hardware TCP Segmentation Offloading & Hardware Large Receive Offloading

    Atom will not be able to handle 500Mbit. i5 should easily do it.



  • 1.3gb/s? wow… I tested with a couple of servers I have that are on 1gb lines and easily can do 900Mbit. Also from some of the testfiles from hosters but mostly via the cli speedtest.net script.

    @Harvy66:

    How are you speed testing? I don't have a 400mb+ connection with which to play with, but with iperf, I can easily reach the limit of my two test machine's 1.3gb/s with less than 5% cpu load, through firewall+NAT.



  • Exactly!  especially because the D2700 hasnt shown any high load at all no matter what I throw at it..
    I am using NAT but also 1:1 NAT with virtual IP's. (because of the multiple external IP's I have).
    Maybe thats slowing things down in some way..

    @stephenw10:

    The D525 will pass 500Mbps. Some have managed >600Mbps with some tuning. I'd be surprised if the D2700 couldn't manage 500Mbps. It won't do 1Gbps though and that's without any packages running.

    Steve


  • Netgate Administrator

    It shouldn't be.
    First off, are you or have you ever used limiters? Any type of traffic shaping? Captive portal?

    Set the Intel NIC as WAN and then run a test from the pfSense command line so you're only testing one side on the box. Is that what you did with the speedtest.net script? Otherwise try just fetching a large file from a known good source:

    fetch -o /dev/null http://cachefly.cachefly.net/10mb.test
    

    Or maybe try something listed here: http://www.cloudtestfiles.net/

    Steve



  • Hi Steve,
    Thanks for you reply. I tried with the links you sent but I get lower speeds probably because I am located in NL not UK.

    I did once use a limiter for 1 IP but those I removed again, why do you mention this explicitly? Does that leave something that could impact througput even after removal?

    ( ooh and the Intel is on the WAN site. also most of the time my test speed on a server behind the pfSense is higher then the pfsense local tests  :-\ )

    [2.1.5-RELEASE][root@gateway]/root(1): fetch -o /dev/null http://cachefly.cachefly.net/10mb.test
    /dev/null                                    100% of  10 MB 4230 kBps

    [2.1.5-RELEASE][root@gateway]/root(2): fetch -o /dev/null http://download.xs4all.nl/test/1GB.bin
    /dev/null                                    100% of  953 MB  32 MBps 00m00s

    @stephenw10:

    It shouldn't be.
    First off, are you or have you ever used limiters? Any type of traffic shaping? Captive portal?

    Set the Intel NIC as WAN and then run a test from the pfSense command line so you're only testing one side on the box. Is that what you did with the speedtest.net script? Otherwise try just fetching a large file from a known good source:

    fetch -o /dev/null http://cachefly.cachefly.net/10mb.test
    

    Or maybe try something listed here: http://www.cloudtestfiles.net/

    Steve


  • Netgate Administrator

    Ah, sorry. I assmed you were in the US.  :-[  Cloudtestfiles have a site in Amsterdam.

    What speeds do you see on the macbook directly connected using the same files?

    Yes, I have seen odd traffic shaping where none should be before. If you've ever used it I would start from a clean install. Even if just as a test, you can always restore you config afterwards.

    Is ifconfig showing the NICs connected at 'autoselect (1000baseT <full-duplex>)'?

    Steve</full-duplex>



  • Aaah.  I did reinstalled it, I only restored the config from the old setup so I should be good on that one.    8)

    The WAN is a pppoe connection. everything else shows indeed 1000baseT <full-duplex>.

    When I check with ifconfig the NIC who is used by the pppoe is also 1000baseT FD.

    By the way, I tried to install iperf to do some other tests but for some reason I cannot install it.
    I can install other packaged but this one throwes me an error:

    Beginning package installation for iperf .
    Downloading package configuration file… done.
    Saving updated package information... done.
    Downloading iperf and its dependencies...
    Checking for package installation...
    Downloading https://files.pfsense.org/packages/amd64/8/All/iperf-2.0.5-amd64.pbi ...  could not download from there or http://files.pfsense.org/packages/amd64/8/All//iperf-2.0.5-amd64.pbi.
    of iperf-2.0.5-amd64 failed!

    Installation aborted.Removing package...
    Starting package deletion for iperf-2.0.5-amd64...done.
    Removing iperf components...
    Tabs items... done.
    Menu items... done.
    Services... done.
    Auxiliary files... done.
    Package XML... done.
    Configuration... done.
    done.
    Failed to install package.

    Installation halted.

    By the looks of it the URL should be http://files.pfsense.org/packages/amd64/8/All/iperf-2.0.5-amd64.pbi. and not http://files.pfsense.org/packages/amd64/8/All//iperf-2.0.5-amd64.pbi.

    So someone other then me made a typo somewhere..  :-[

    [quote author=stephenw10 link=topic=84731.msg465211#msg465211 date=1417454403]
    Ah, sorry. I assmed you were in the US.  :-[  Cloudtestfiles have a site in Amsterdam.

    What speeds do you see on the macbook directly connected using the same files?

    Yes, I have seen odd traffic shaping where none should be before. If you've ever used it I would start from a clean install. Even if just as a test, you can always restore you config afterwards.

    Is ifconfig showing the NICs connected at 'autoselect (1000baseT <full-duplex>)'?

    Steve
    [/quote]</full-duplex></full-duplex>



  • I would check BIOS settings. I've also read that Hyper-threading should also be turned off if you have it.



  • https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

    also check for errors on the wiring (just in case)



  • Is your Intel card PCI or PCI Express?

    Also make your test using only the intel nics.


  • Netgate Administrator

    I wouldn't suspect hyperthreading (though it's an easy test). The general advise to disable it was based on much older hardware, netburst core, and no longer applies.

    Even with two PCI NICs sharing the same bus I would expect ~500Mbps. Best to get that info though.

    There are various power saving options that can kill performance. I forget exactly what it's called but there is a PCI power saving option that has been found causing problems on atoms before. There was a thread about it here somewhere, I should have book marked it.  ::)

    A clean install to which you restored your config may not be good enough. To rule out anything being carried across try just a very basic setup with nothing restored. Just run it live from an install CD or memstick install image.

    Reading back, what hardware options did you set to reduce your throughout to 70Mbps?

    Steve

    Edit: Found it. This thread sees 600Mbps+ with a D2500: https://forum.pfsense.org/index.php?topic=67411.0



  • A lot of things to keep me busy here in this thread.. Much appreciated.

    The speed went drastically down (70Mbps) when I checked the Hardware Checksum Offloading box. I also unticked for test the Hardware TCP Segmentation Offloading and Hardware Large Receive Offloading but there wasnt any noticable difference.

    I dont have any power savings on in the BIOS and AFAIK also none in pfSense but I will search for powersettings for atoms maybe I indeed missed something there.
    Next I will read the link you posted in your message.

    As mentioned somewhere along the road, I also tested the same setup in an i5 with an PCIe 4port Intel server card and got exactly the same throughput so my conclusion from that was that it is not likely to be the PCI bus and also not the Atom but still it doesnt harm to test further.

    I heard nobody about the NAT 1:1 mapping so I guess thats never been a bottleneck?

    Greetz,
    Override

    @stephenw10:

    I wouldn't suspect hyperthreading (though it's an easy test). The general advise to disable it was based on much older hardware, netburst core, and no longer applies.

    Even with two PCI NICs sharing the same bus I would expect ~500Mbps. Best to get that info though.

    There are various power saving options that can kill performance. I forget exactly what it's called but there is a PCI power saving option that has been found causing problems on atoms before. There was a thread about it here somewhere, I should have book marked it.  ::)

    A clean install to which you restored your config may not be good enough. To rule out anything being carried across try just a very basic setup with nothing restored. Just run it live from an install CD or memstick install image.

    Reading back, what hardware options did you set to reduce your throughout to 70Mbps?

    Steve

    Edit: Found it. This thread sees 600Mbps+ with a D2500: https://forum.pfsense.org/index.php?topic=67411.0



  • Power saving options for Intel CPUs are C-states and SpeedStep.

    Have you also tried resetting your modem to defaults and allowing it to pull down configs again?


  • Netgate Administrator

    I'm not talking about the CPU power saving, in the linked thread the offending option was PCI express ASPM which saves power by somehow sleeping/waking the PCIe bus. However the fact that you are seeing the identical speed on an i5 should be a big clue here. It's very unlikely both would have the same bad bios settings and even if they did the i5 is so much more powerful you'd expect to see at least some difference.

    Both share the same upstream hardware connection, both use the same Intel driver. Some low level hardware incompatibility? Can you put a switch in between the WAN interface and the modem?

    Steve


  • Netgate Administrator

    Actually reading back through the linked thread the PCIe ASPM only helped a little, the biggest performance increase came from using NICs on different buses. You would not have thought there would be any restrictions on a PCIe though. Also doesn't explain the i5 result.

    Steve