10G TCP Performance



  • I have 2.3.2-p1 as the gateway for Comcast Gigabit Pro (2G/2G Fiber).
    The server is a SM mobo with an E5-1620v2 3.7GHz QC, 8GB RAM, SSD storage.
    An Intel x520-DA2 NIC connected to the CPE (Juniper ACX2100) and to a UBNT ES-16-XG (10G switch).

    The problem is, I have very poor TCP performance to/from the internet. UDP is fine.

    I have been testing with iperf and various speed tests, but primarily DSLR Speedtest.  This test host is my gaming PC with a 10G NIC.  You can see here, that the speeds are erratic.

    For the iperf testing, I have a new Ubuntu 16.04VM, fully patched on my ESXi host. R610, dual X5670 2.9Ghz 6c procs, 64GB RAM, etc.  NIC is QLogic QLE8442-CU-CK.  10G connection to the ES-16-XG.  I also have a filer with the same NIC, e3-1220v3 @3.1Ghz, 32G RAM, etc.  Not a question of hardware being able to move the packets.

    iperf3 testing from the VM to the filer.

    == tcp download==
    iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20
    [  4]   0.00-20.00  sec  14.6 GBytes  6278 Mbits/sec    0             sender
    [  4]   0.00-20.00  sec  14.7 GBytes  6293 Mbits/sec                  receiver
    == tcp upload ==
    iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20 -R
    [  4]   0.00-20.00  sec  21.1 GBytes  9070 Mbits/sec  3764             sender
    [  4]   0.00-20.00  sec  21.1 GBytes  9053 Mbits/sec                  receiver
    == udp 10g download==
    iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20 -u -b10g
    [  4]   0.00-20.00  sec  5.90 GBytes  2535 Mbits/sec  0.116 ms  3963/773715 (0.51%)
    == udp 10g upload ==
    iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20 -u -b10g -R
    [  4]   0.00-20.00  sec  18.0 GBytes  7713 Mbits/sec  0.007 ms  489576/2349311 (21%)
    

    Perf isn't perfect, but clearly demonstrates that the hosts I am testing with are capable of the throughput.

    UDP testing from the VM to the internet shows the throughput is there and not an issue with v4 vs. v6.

    == ipv6 udp 2g download to interenet==
    iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G
    [  4]   0.00-20.00  sec  4.66 GBytes  2000 Mbits/sec  0.009 ms  312866/610090 (51%)
    == ipv6 udp 2g upload to interenet==
    iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G -R
    [  4]   0.00-20.00  sec  4.67 GBytes  2006 Mbits/sec  0.013 ms  30043/612286 (4.9%)
    == ipv4 udp 2g download to interenet==
    iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G
    [  4]   0.00-20.00  sec  4.66 GBytes  2001 Mbits/sec  783.302 ms  263909/610712 (43%)
    == ipv4 udp 2g upload to interenet==
    iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G -R
    [  4]   0.00-20.00  sec  4.67 GBytes  2007 Mbits/sec  0.009 ms  4832/612510 (0.79%)
    

    Here is the problem.  Switching to TCP, this is what I get.

    == ipv6 tcp download to internet ==
    iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net
    [  4]   0.00-20.00  sec  2.74 GBytes  1176 Mbits/sec  3052             sender
    [  4]   0.00-20.00  sec  2.75 GBytes  1179 Mbits/sec                  receiver
    == ipv6 tcp upload to internet ==
    iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -R
    [  4]   0.00-20.00  sec  1.06 GBytes   454 Mbits/sec   38             sender
    [  4]   0.00-20.00  sec  1.05 GBytes   452 Mbits/sec                  receiver
    
    == ipv4 tcp download to internet ==
    iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net
    [  4]   0.00-20.00  sec  2.76 GBytes  1186 Mbits/sec  1571             sender
    [  4]   0.00-20.00  sec  2.77 GBytes  1189 Mbits/sec                  receiver
    == ipv4 tcp upload to internet ==
    iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -R
    [  4]   0.00-20.00  sec  2.45 GBytes  1052 Mbits/sec    0             sender
    [  4]   0.00-20.00  sec  2.45 GBytes  1053 Mbits/sec                  receiver
    

    Full Output of those TCP tests:  https://p.bsd-unix.net/view/5fb680b8

    In looking at those, you can see when downloading from the internet, the TCP connection will hit 2G occasionally but bounces off the limiter and perf takes a nose dive until it can recover.

    What I am looking for is help is tuning pfsense and/or QoS to be able to get consistent 2G/2G TCP throughput.  Currently, I have no QoS configured, as my prior attempts significantly degraded throughput.



  • == ipv4 udp 2g download to interenet==
    iperf3 -4 -p5201 -t 20 -P 1 -O 3 –get-server-output -f m -c iperf.he.net -u -b2G
    [  4]  0.00-20.00  sec  4.66 GBytes  2001 Mbits/sec  783.302 ms  263909/610712 (43%)

    I'm not sure an 800ms ping and almost 50% packet-loss is showing that UDP in unaffected. My guess is your ISP can't actually provide anywhere near the bandwidth it has provisioned you in a stable fashion.



  • What is the latency to the server you are testing with?
    TCP speeds degrade quickly as latency increases since each packet needs to wait for the acknowledgement from the receiving end before sending the next packet in the series.  This is why products like Signiant and Aspera exist (and cost big bucks) so that large volumes of data can be sent reliably using UDP between sites that are geographically far apart.



  • @Harvy66:

    == ipv4 udp 2g download to interenet==
    iperf3 -4 -p5201 -t 20 -P 1 -O 3 –get-server-output -f m -c iperf.he.net -u -b2G
    [  4]  0.00-20.00  sec  4.66 GBytes  2001 Mbits/sec  783.302 ms  263909/610712 (43%)

    I'm not sure an 800ms ping and almost 50% packet-loss is showing that UDP in unaffected. My guess is your ISP can't actually provide anywhere near the bandwidth it has provisioned you in a stable fashion.

    That is a fair and valid point.  Since posting this, I had done a lot more testing and found more signs pointing to the issue being ISP.

    @berniecnyc What is the latency to the server you are testing with? Consistent 73ms.



  • @mrjester:

    @Harvy66:

    == ipv4 udp 2g download to interenet==
    iperf3 -4 -p5201 -t 20 -P 1 -O 3 –get-server-output -f m -c iperf.he.net -u -b2G
    [  4]  0.00-20.00  sec  4.66 GBytes  2001 Mbits/sec  783.302 ms  263909/610712 (43%)

    I'm not sure an 800ms ping and almost 50% packet-loss is showing that UDP in unaffected. My guess is your ISP can't actually provide anywhere near the bandwidth it has provisioned you in a stable fashion.

    That is a fair and valid point.  Since posting this, I had done a lot more testing and found more signs pointing to the issue being ISP.

    @berniecnyc What is the latency to the server you are testing with? Consistent 73ms.

    Looks like you need to do some TCP tuning itself in sysctl. I've been doing this on 1Gbe hosts in DC's and constantly get 2.3Gbe bursts all the time and get 100MB/s even over the pond on a little $5 VPS.

    Here is a post a while back who used pfsense with xfinity and a whole slew of gear. Maybe this could help

    http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/



  • He was doing a UDP test and attempting to send 2Gb/s over his 2Gb/s connection was causing almost 50% packetloss on average. His connection cannot support anywhere near his provisioned speed. He also had several performance tests showing he can get 1.95Gb/s over TCP, but the same test may only give him 300Mb/s only minutes later.

    I do agree TCP tuning becomes an issue these rates and typical WAN latencies, but that is not the current bottleneck.

    And TCP tuning PFSense won't gain you almost anything in for most settings. The firewall is not the sender or receiver, it's just a middleman that makes sure the state is valid.


Log in to reply