Wan iPerf UDP max out tcp isnt even .5% of throughput
-
Tested from multiple locations, I would think this physical box should just light up the night sky.
UDP test I can push over 2Gbits UDP between multiple sources, however the second I run TCP (yes single stream) I'm lucky to see 50Mbit....
[ 4] 0.00-1.00 sec 4.75 MBytes 39.8 Mbits/sec
[ 4] 1.00-2.02 sec 6.88 MBytes 56.9 Mbits/sec
[ 4] 2.02-3.00 sec 6.75 MBytes 57.4 Mbits/sec
[ 4] 3.00-4.01 sec 7.00 MBytes 58.3 Mbits/sec
[ 4] 4.01-5.01 sec 6.88 MBytes 57.3 Mbits/sec
[ 4] 5.01-6.00 sec 6.88 MBytes 58.4 Mbits/sec
[ 4] 6.00-7.00 sec 7.00 MBytes 58.7 Mbits/sec
[ 4] 7.00-8.01 sec 6.75 MBytes 56.1 Mbits/sec
[ 4] 8.01-9.00 sec 6.88 MBytes 58.3 Mbits/sec
[ 4] 9.00-10.01 sec 6.88 MBytes 57.3 Mbits/secRunning 2.5 with these settings
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
hw.ix.flow_control=0
net.isr.dispatch=deferredInternal from this same machine to a server I almost 10gbit, everything is 10Gbit so internal is fine
Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
88 CPUs: 2 package(s) x 22 core(s) x 2 hardware threads4x10Gbit Intel 540 (Copper) nics
128GB Ram
4x512SSDs
-
Where are you testing to/from that shows 2Gbps UDP? Lets see that result.
Otherwise that's such a huge throttle it either something deliberate or something like a speed/duplex mismatch.
Steve
-
@stephenw10 10Gbit lines from work testing to my 2.5Gbit ONT
-
And it's linked at 2.5G?
No errors shown on the interface?
Steve
-
@stephenw10 on UDP it ended up having 1.08%, tcp tests though never a drop
CPU use never exceeds 1% so I know I'm not taxing it
-
@stephenw10 its acting like its xamount per single stream, I guess thats the odd part, I add 10 and it hits over 500Mbit. Seems awfully specific
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 2.00 MBytes 16.8 Mbits/sec
[ 6] 0.00-1.00 sec 3.12 MBytes 26.2 Mbits/sec
[ 8] 0.00-1.00 sec 3.38 MBytes 28.3 Mbits/sec
[ 10] 0.00-1.00 sec 2.25 MBytes 18.9 Mbits/sec
[SUM] 0.00-1.00 sec 10.8 MBytes 90.2 Mbits/sec
[ 4] 1.00-2.00 sec 3.62 MBytes 30.4 Mbits/sec
[ 6] 1.00-2.00 sec 6.38 MBytes 53.5 Mbits/sec
[ 8] 1.00-2.00 sec 6.38 MBytes 53.5 Mbits/sec
[ 10] 1.00-2.00 sec 4.12 MBytes 34.6 Mbits/sec
[SUM] 1.00-2.00 sec 20.5 MBytes 172 Mbits/sec
[ 4] 2.00-3.00 sec 5.75 MBytes 48.2 Mbits/sec
[ 6] 2.00-3.00 sec 6.50 MBytes 54.5 Mbits/sec
[ 8] 2.00-3.00 sec 7.00 MBytes 58.7 Mbits/sec
[ 10] 2.00-3.00 sec 6.12 MBytes 51.4 Mbits/sec
[SUM] 2.00-3.00 sec 25.4 MBytes 213 Mbits/sec
[ 4] 3.00-4.00 sec 6.50 MBytes 54.5 Mbits/sec
[ 6] 3.00-4.00 sec 6.75 MBytes 56.6 Mbits/sec
[ 8] 3.00-4.00 sec 6.75 MBytes 56.6 Mbits/sec
[ 10] 3.00-4.00 sec 7.00 MBytes 58.7 Mbits/sec
[SUM] 3.00-4.00 sec 27.0 MBytes 226 Mbits/sec
[ 4] 4.00-5.00 sec 6.50 MBytes 54.5 Mbits/sec
[ 6] 4.00-5.00 sec 6.38 MBytes 53.5 Mbits/sec
[ 8] 4.00-5.00 sec 6.88 MBytes 57.7 Mbits/sec
[ 10] 4.00-5.00 sec 6.75 MBytes 56.6 Mbits/sec
[SUM] 4.00-5.00 sec 26.5 MBytes 222 Mbits/sec
[ 4] 5.00-6.00 sec 6.38 MBytes 53.5 Mbits/sec
[ 6] 5.00-6.00 sec 4.75 MBytes 39.9 Mbits/sec
[ 8] 5.00-6.00 sec 6.75 MBytes 56.6 Mbits/sec
[ 10] 5.00-6.00 sec 7.00 MBytes 58.7 Mbits/sec
[SUM] 5.00-6.00 sec 24.9 MBytes 209 Mbits/sec
[ 4] 6.00-7.00 sec 6.50 MBytes 54.5 Mbits/sec
[ 6] 6.00-7.00 sec 4.12 MBytes 34.6 Mbits/sec
[ 8] 6.00-7.00 sec 7.00 MBytes 58.7 Mbits/sec
[ 10] 6.00-7.00 sec 6.88 MBytes 57.7 Mbits/sec
[SUM] 6.00-7.00 sec 24.5 MBytes 206 Mbits/sec
[ 4] 7.00-8.00 sec 6.50 MBytes 54.5 Mbits/sec
[ 6] 7.00-8.00 sec 5.12 MBytes 43.0 Mbits/sec
[ 8] 7.00-8.00 sec 6.75 MBytes 56.6 Mbits/sec
[ 10] 7.00-8.00 sec 6.88 MBytes 57.7 Mbits/sec
[SUM] 7.00-8.00 sec 25.2 MBytes 212 Mbits/sec
[ 4] 8.00-9.00 sec 6.50 MBytes 54.5 Mbits/sec
[ 6] 8.00-9.00 sec 5.88 MBytes 49.2 Mbits/sec
[ 8] 8.00-9.00 sec 6.75 MBytes 56.6 Mbits/sec
[ 10] 8.00-9.00 sec 6.75 MBytes 56.6 Mbits/sec
[SUM] 8.00-9.00 sec 25.9 MBytes 217 Mbits/sec
[ 4] 9.00-10.00 sec 6.50 MBytes 54.6 Mbits/sec
[ 6] 9.00-10.00 sec 6.62 MBytes 55.6 Mbits/sec
[ 8] 9.00-10.00 sec 6.75 MBytes 56.7 Mbits/sec
[ 10] 9.00-10.00 sec 7.00 MBytes 58.8 Mbits/sec
[SUM] 9.00-10.00 sec 26.9 MBytes 226 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 56.8 MBytes 47.6 Mbits/sec sender
[ 4] 0.00-10.00 sec 56.8 MBytes 47.6 Mbits/sec receiver
[ 6] 0.00-10.00 sec 55.6 MBytes 46.7 Mbits/sec sender
[ 6] 0.00-10.00 sec 55.6 MBytes 46.7 Mbits/sec receiver
[ 8] 0.00-10.00 sec 64.4 MBytes 54.0 Mbits/sec sender
[ 8] 0.00-10.00 sec 64.4 MBytes 54.0 Mbits/sec receiver
[ 10] 0.00-10.00 sec 60.8 MBytes 51.0 Mbits/sec sender
[ 10] 0.00-10.00 sec 60.8 MBytes 51.0 Mbits/sec receiver
[SUM] 0.00-10.00 sec 238 MBytes 199 Mbits/sec sender
[SUM] 0.00-10.00 sec 238 MBytes 199 Mbits/sec receiver -
It was dropping 1% packets at 2Gbps target?
Can we see that result?
-
@stephenw10 Again thats a udp test, oddly even faster now, but UDP isnt a perfect example
intesting is that its a GPON terminating on an XG16, XG is connected to a 3850 stack over an lacp trunk, then the pfsense box connects and does vlans on the 3850 for everything.
In this case, I expect dropped packets for UDP as its going nuts
TCP, I never have dropped packets, but as I said, VERY limiting speeds, its a PPPOE
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 335 MBytes 2.81 Gbits/sec 42920
[ 4] 1.00-2.00 sec 413 MBytes 3.46 Gbits/sec 52810
[ 4] 2.00-3.00 sec 445 MBytes 3.73 Gbits/sec 56950
[ 4] 3.00-4.00 sec 464 MBytes 3.89 Gbits/sec 59340
[ 4] 4.00-5.00 sec 462 MBytes 3.88 Gbits/sec 59170
[ 4] 5.00-6.00 sec 467 MBytes 3.91 Gbits/sec 59720
[ 4] 6.00-7.00 sec 483 MBytes 4.05 Gbits/sec 61830
[ 4] 7.00-8.00 sec 491 MBytes 4.12 Gbits/sec 62910
[ 4] 8.00-9.00 sec 504 MBytes 4.23 Gbits/sec 64510
[ 4] 9.00-10.00 sec 493 MBytes 4.13 Gbits/sec 63050
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 4.45 GBytes 3.82 Gbits/sec 0.098 ms 581070/581783 (1e+02%)
[ 4] Sent 581783 datagrams -
-
@techanalyst said in Wan iPerf UDP max out tcp isnt even .5% of throughput:
Lost/Total Datagrams
581070/581783You realise that's showing it lost 581070 packets?
It is losing 99.8% of packets at 2Gbps. Something on that connection is very broken!
Given that the TCP throughput is quite good.
Steve
-
@stephenw10 said in Wan iPerf UDP max out tcp isnt even .5% of throughput:
You realise that's showing it lost 581070 packets?
It is losing 99.8% of packets at 2Gbps. Something on that connection is very broken!
Given that the TCP throughput is quite good.So thats a 2.5GPON, the speeds are roughly 1700 down and 1000 up
thats why I said UDP test which shows all this is kinda useless
So these throughput iperfs are UDP.....TCP iperf is absolute trash
-
Ok, so run a UDP test at 500Mbps and see what packet loss you get then. It should be close to zero of the connection is good.
Hard to imagine it will be though when you have nearly 100% loss at 2GbpsSteve
-
@stephenw10 said in Wan iPerf UDP max out tcp isnt even .5% of throughput:
Ok, so run a UDP test at 500Mbps and see what packet loss you get then. It should be close to zero of the connection is good.
Hard to imagine it will be though when you have nearly 100% loss at 2GbpsOddly enough from an Enterprise line to myself packet drops stop at 109Mbit :( thats kinda crappy
So you feels its ISP related?
**[ 4] 0.00-1.00 sec 346 MBytes 2.90 Gbits/sec
[ 4] 1.00-2.00 sec 419 MBytes 3.52 Gbits/sec
[ 4] 2.00-3.00 sec 419 MBytes 3.52 Gbits/sec
[ 4] 3.00-4.00 sec 399 MBytes 3.35 Gbits/sec
[ 4] 4.00-5.00 sec 412 MBytes 3.46 Gbits/sec
[ 4] 5.00-6.00 sec 418 MBytes 3.51 Gbits/sec
[ 4] 6.00-7.00 sec 420 MBytes 3.52 Gbits/sec
[ 4] 7.00-8.00 sec 420 MBytes 3.52 Gbits/sec
[ 4] 8.00-9.00 sec 405 MBytes 3.40 Gbits/sec
[ 4] 9.00-10.00 sec 415 MBytes 3.48 Gbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.98 GBytes 3.42 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.98 GBytes 3.42 Gbits/sec receiver
**Internally it seems both an x540 and x550 is kinda slow
My workstation or servers to each other usually work perfectr
[ 4] 0.00-1.00 sec 1.05 GBytes 8.99 Gbits/sec
[ 4] 1.00-2.00 sec 1.04 GBytes 8.95 Gbits/sec
[ 4] 2.00-3.00 sec 1.05 GBytes 9.01 Gbits/sec
[ 4] 3.00-4.00 sec 1.05 GBytes 9.04 Gbits/sec
[ 4] 4.00-5.00 sec 1.05 GBytes 9.01 Gbits/sec
[ 4] 5.00-6.00 sec 1.05 GBytes 9.00 Gbits/sec
[ 4] 6.00-7.00 sec 1.05 GBytes 9.01 Gbits/sec
[ 4] 7.00-8.00 sec 1.04 GBytes 8.94 Gbits/sec
[ 4] 8.00-9.00 sec 1.05 GBytes 9.05 Gbits/sec
[ 4] 9.00-10.00 sec 1.04 GBytes 8.96 Gbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 10.5 GBytes 9.00 Gbits/sec sender
[ 4] 0.00-10.00 sec 10.5 GBytes 9.00 Gbits/sec receiver -
Well from the results you've posted you see to have massive packet loss on the connection.
Nothing we can do about that in pfSense other than limit the speed to something bellow the loss level. Which is not a great solution!
Can we see the output from a UDP test just about that threshold? Say 150Mbps.
Do you see loss if you run the test reversed? (-R)
Steve
-
@stephenw10 my speed tests are over 1700/1000, i dont see value in lowering the throughput when it seems to be specific to locations
-
Because you are seeing almost 100% packet loss at anywhere close to those speeds.
Either you're testing completely wrong or the connection is very broken.
If it's the later knowing what value the packet loss seems to kick in at is useful.
Steve