10G TCP Performance
-
I have 2.3.2-p1 as the gateway for Comcast Gigabit Pro (2G/2G Fiber).
The server is a SM mobo with an E5-1620v2 3.7GHz QC, 8GB RAM, SSD storage.
An Intel x520-DA2 NIC connected to the CPE (Juniper ACX2100) and to a UBNT ES-16-XG (10G switch).The problem is, I have very poor TCP performance to/from the internet. UDP is fine.
I have been testing with iperf and various speed tests, but primarily DSLR Speedtest. This test host is my gaming PC with a 10G NIC. You can see here, that the speeds are erratic.
For the iperf testing, I have a new Ubuntu 16.04VM, fully patched on my ESXi host. R610, dual X5670 2.9Ghz 6c procs, 64GB RAM, etc. NIC is QLogic QLE8442-CU-CK. 10G connection to the ES-16-XG. I also have a filer with the same NIC, e3-1220v3 @3.1Ghz, 32G RAM, etc. Not a question of hardware being able to move the packets.
iperf3 testing from the VM to the filer.
== tcp download== iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20 [ 4] 0.00-20.00 sec 14.6 GBytes 6278 Mbits/sec 0 sender [ 4] 0.00-20.00 sec 14.7 GBytes 6293 Mbits/sec receiver == tcp upload == iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20 -R [ 4] 0.00-20.00 sec 21.1 GBytes 9070 Mbits/sec 3764 sender [ 4] 0.00-20.00 sec 21.1 GBytes 9053 Mbits/sec receiver == udp 10g download== iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20 -u -b10g [ 4] 0.00-20.00 sec 5.90 GBytes 2535 Mbits/sec 0.116 ms 3963/773715 (0.51%) == udp 10g upload == iperf3 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c 10.8.10.20 -u -b10g -R [ 4] 0.00-20.00 sec 18.0 GBytes 7713 Mbits/sec 0.007 ms 489576/2349311 (21%)
Perf isn't perfect, but clearly demonstrates that the hosts I am testing with are capable of the throughput.
UDP testing from the VM to the internet shows the throughput is there and not an issue with v4 vs. v6.
== ipv6 udp 2g download to interenet== iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G [ 4] 0.00-20.00 sec 4.66 GBytes 2000 Mbits/sec 0.009 ms 312866/610090 (51%) == ipv6 udp 2g upload to interenet== iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G -R [ 4] 0.00-20.00 sec 4.67 GBytes 2006 Mbits/sec 0.013 ms 30043/612286 (4.9%) == ipv4 udp 2g download to interenet== iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G [ 4] 0.00-20.00 sec 4.66 GBytes 2001 Mbits/sec 783.302 ms 263909/610712 (43%) == ipv4 udp 2g upload to interenet== iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -u -b2G -R [ 4] 0.00-20.00 sec 4.67 GBytes 2007 Mbits/sec 0.009 ms 4832/612510 (0.79%)
Here is the problem. Switching to TCP, this is what I get.
== ipv6 tcp download to internet == iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net [ 4] 0.00-20.00 sec 2.74 GBytes 1176 Mbits/sec 3052 sender [ 4] 0.00-20.00 sec 2.75 GBytes 1179 Mbits/sec receiver == ipv6 tcp upload to internet == iperf3 -6 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -R [ 4] 0.00-20.00 sec 1.06 GBytes 454 Mbits/sec 38 sender [ 4] 0.00-20.00 sec 1.05 GBytes 452 Mbits/sec receiver == ipv4 tcp download to internet == iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net [ 4] 0.00-20.00 sec 2.76 GBytes 1186 Mbits/sec 1571 sender [ 4] 0.00-20.00 sec 2.77 GBytes 1189 Mbits/sec receiver == ipv4 tcp upload to internet == iperf3 -4 -p5201 -t 20 -P 1 -O 3 --get-server-output -f m -c iperf.he.net -R [ 4] 0.00-20.00 sec 2.45 GBytes 1052 Mbits/sec 0 sender [ 4] 0.00-20.00 sec 2.45 GBytes 1053 Mbits/sec receiver
Full Output of those TCP tests: https://p.bsd-unix.net/view/5fb680b8
In looking at those, you can see when downloading from the internet, the TCP connection will hit 2G occasionally but bounces off the limiter and perf takes a nose dive until it can recover.
What I am looking for is help is tuning pfsense and/or QoS to be able to get consistent 2G/2G TCP throughput. Currently, I have no QoS configured, as my prior attempts significantly degraded throughput.
-
== ipv4 udp 2g download to interenet==
iperf3 -4 -p5201 -t 20 -P 1 -O 3 –get-server-output -f m -c iperf.he.net -u -b2G
[ 4] 0.00-20.00 sec 4.66 GBytes 2001 Mbits/sec 783.302 ms 263909/610712 (43%)I'm not sure an 800ms ping and almost 50% packet-loss is showing that UDP in unaffected. My guess is your ISP can't actually provide anywhere near the bandwidth it has provisioned you in a stable fashion.
-
What is the latency to the server you are testing with?
TCP speeds degrade quickly as latency increases since each packet needs to wait for the acknowledgement from the receiving end before sending the next packet in the series. This is why products like Signiant and Aspera exist (and cost big bucks) so that large volumes of data can be sent reliably using UDP between sites that are geographically far apart. -
== ipv4 udp 2g download to interenet==
iperf3 -4 -p5201 -t 20 -P 1 -O 3 –get-server-output -f m -c iperf.he.net -u -b2G
[ 4] 0.00-20.00 sec 4.66 GBytes 2001 Mbits/sec 783.302 ms 263909/610712 (43%)I'm not sure an 800ms ping and almost 50% packet-loss is showing that UDP in unaffected. My guess is your ISP can't actually provide anywhere near the bandwidth it has provisioned you in a stable fashion.
That is a fair and valid point. Since posting this, I had done a lot more testing and found more signs pointing to the issue being ISP.
@berniecnyc What is the latency to the server you are testing with? Consistent 73ms.
-
== ipv4 udp 2g download to interenet==
iperf3 -4 -p5201 -t 20 -P 1 -O 3 –get-server-output -f m -c iperf.he.net -u -b2G
[ 4] 0.00-20.00 sec 4.66 GBytes 2001 Mbits/sec 783.302 ms 263909/610712 (43%)I'm not sure an 800ms ping and almost 50% packet-loss is showing that UDP in unaffected. My guess is your ISP can't actually provide anywhere near the bandwidth it has provisioned you in a stable fashion.
That is a fair and valid point. Since posting this, I had done a lot more testing and found more signs pointing to the issue being ISP.
@berniecnyc What is the latency to the server you are testing with? Consistent 73ms.
Looks like you need to do some TCP tuning itself in sysctl. I've been doing this on 1Gbe hosts in DC's and constantly get 2.3Gbe bursts all the time and get 100MB/s even over the pond on a little $5 VPS.
Here is a post a while back who used pfsense with xfinity and a whole slew of gear. Maybe this could help
http://www.pcgamer.com/what-its-like-to-have-the-fastest-internet-speeds-in-the-country/
-
He was doing a UDP test and attempting to send 2Gb/s over his 2Gb/s connection was causing almost 50% packetloss on average. His connection cannot support anywhere near his provisioned speed. He also had several performance tests showing he can get 1.95Gb/s over TCP, but the same test may only give him 300Mb/s only minutes later.
I do agree TCP tuning becomes an issue these rates and typical WAN latencies, but that is not the current bottleneck.
And TCP tuning PFSense won't gain you almost anything in for most settings. The firewall is not the sender or receiver, it's just a middleman that makes sure the state is valid.