iperf3 speeds confusing - fast = bad internet slow = good internet?
-
@notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
Yesterday I spent a lot of time going through my network settings and attempting to optimize certain things.
Hi,
So, I haven’t found a better guide to what you set out to achieve:
https://calomel.org/freebsd_network_tuning.html
https://calomel.org/network_performance.html
+++edit:
we mention this a lot:
TSO / LRO is not good for you if you are a router (pfSense), if you are an endpoint = YES -
@daddygo thanks for the links...
However I suppose I don't see the relationship to my iperf3 speeds in this case.
BTW, the ESXi box is very capable in terms of CPU horsepower IMO:
2x Xeon E5-2695 v2 @ 2.40GHz
Don't get me wrong, my current config is obviously capable of saturating my internet connection (at least downstream) with speed tests consistently around 1500-1600 Mbit/s. But ingress isn't always so stable (which I know is not so easy).
I just did a quick iperf3 test with the offloads disabled (current working config).
Interestingly enough the slow iperf3 speeds are only TO pfsense (pfsense being the ipef3 server). Going the other way (CentOs box being the iperf3 server) I'm seeing very good speeds as I would expect.
I'm not quite sure where I can make the improvements reuiqred in the incoming connections to pfsense where I'm seeing only ~2.8Gbit/s
See results (from the CentOS box):
Connecting to host 10.10.0.254, port 5201 [ 4] local 10.10.1.32 port 40738 connected to 10.10.0.254 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec 34 509 KBytes [ 4] 1.00-2.00 sec 345 MBytes 2.89 Gbits/sec 215 431 KBytes [ 4] 2.00-3.00 sec 356 MBytes 2.99 Gbits/sec 78 361 KBytes [ 4] 3.00-4.00 sec 332 MBytes 2.79 Gbits/sec 57 318 KBytes [ 4] 4.00-5.00 sec 321 MBytes 2.68 Gbits/sec 69 393 KBytes [ 4] 5.00-6.00 sec 322 MBytes 2.71 Gbits/sec 134 501 KBytes [ 4] 6.00-7.00 sec 325 MBytes 2.73 Gbits/sec 42 498 KBytes [ 4] 7.00-8.00 sec 325 MBytes 2.73 Gbits/sec 113 452 KBytes [ 4] 8.00-9.00 sec 334 MBytes 2.80 Gbits/sec 82 458 KBytes [ 4] 9.00-10.00 sec 335 MBytes 2.81 Gbits/sec 26 540 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 3.21 GBytes 2.76 Gbits/sec 850 sender [ 4] 0.00-10.00 sec 3.21 GBytes 2.76 Gbits/sec receiver iperf Done. [root@dockersrv02 ~]# iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.10.0.254, port 57649 [ 5] local 10.10.1.32 port 5201 connected to 10.10.0.254 port 64408 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.04 GBytes 8.92 Gbits/sec [ 5] 1.00-2.00 sec 1.56 GBytes 13.4 Gbits/sec [ 5] 2.00-3.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 3.00-4.00 sec 1.65 GBytes 14.2 Gbits/sec [ 5] 4.00-5.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 5.00-6.00 sec 1.67 GBytes 14.3 Gbits/sec [ 5] 6.00-7.00 sec 1.64 GBytes 14.1 Gbits/sec [ 5] 7.00-8.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 8.00-9.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 9.00-10.00 sec 1.64 GBytes 14.1 Gbits/sec [ 5] 10.00-10.20 sec 333 MBytes 14.3 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.20 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.20 sec 16.1 GBytes 13.5 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated [root@dockersrv02 ~]# -
@notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
I'm not quite sure where I can make the improvements reuiqred in the incoming connections to pfsense where
We talked a lot here about this on the forum...
Always the configuration first, then the measurements (iPerf) can come.Yes this is different under Linux, as you can read in the description in the link.
I wouldn’t use hypervisor instead of bare metal for an important device like router + FW, but it’s a matter of taste, you don’t have redundancy.. (I hope you know that)
so i suggest you set your parameters first: loader.conf.local
such as:
- flowcontrol disable (dev.ixyz.0.fc=0)
- EEE disable (dev.ixyz.0.eee_disabled: value=1)
(these are explanatory values)
something like that, but implement it on your own hardware set:
+++edit:
test,.... can still be these:ifconfig -vma
dmesg | grep -i msi
vmstat -i
sysctl -a | grep num_queues -
@notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
@daddygo thanks for the links...
However I suppose I don't see the relationship to my iperf3 speeds in this case.
BTW, the ESXi box is very capable in terms of CPU horsepower IMO:
2x Xeon E5-2695 v2 @ 2.40GHz
Don't get me wrong, my current config is obviously capable of saturating my internet connection (at least downstream) with speed tests consistently around 1500-1600 Mbit/s. But ingress isn't always so stable (which I know is not so easy).
I just did a quick iperf3 test with the offloads disabled (current working config).
Interestingly enough the slow iperf3 speeds are only TO pfsense (pfsense being the ipef3 server). Going the other way (CentOs box being the iperf3 server) I'm seeing very good speeds as I would expect.
I'm not quite sure where I can make the improvements reuiqred in the incoming connections to pfsense where I'm seeing only ~2.8Gbit/s
See results (from the CentOS box):
Connecting to host 10.10.0.254, port 5201 [ 4] local 10.10.1.32 port 40738 connected to 10.10.0.254 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec 34 509 KBytes [ 4] 1.00-2.00 sec 345 MBytes 2.89 Gbits/sec 215 431 KBytes [ 4] 2.00-3.00 sec 356 MBytes 2.99 Gbits/sec 78 361 KBytes [ 4] 3.00-4.00 sec 332 MBytes 2.79 Gbits/sec 57 318 KBytes [ 4] 4.00-5.00 sec 321 MBytes 2.68 Gbits/sec 69 393 KBytes [ 4] 5.00-6.00 sec 322 MBytes 2.71 Gbits/sec 134 501 KBytes [ 4] 6.00-7.00 sec 325 MBytes 2.73 Gbits/sec 42 498 KBytes [ 4] 7.00-8.00 sec 325 MBytes 2.73 Gbits/sec 113 452 KBytes [ 4] 8.00-9.00 sec 334 MBytes 2.80 Gbits/sec 82 458 KBytes [ 4] 9.00-10.00 sec 335 MBytes 2.81 Gbits/sec 26 540 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 3.21 GBytes 2.76 Gbits/sec 850 sender [ 4] 0.00-10.00 sec 3.21 GBytes 2.76 Gbits/sec receiver iperf Done. [root@dockersrv02 ~]# iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.10.0.254, port 57649 [ 5] local 10.10.1.32 port 5201 connected to 10.10.0.254 port 64408 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.04 GBytes 8.92 Gbits/sec [ 5] 1.00-2.00 sec 1.56 GBytes 13.4 Gbits/sec [ 5] 2.00-3.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 3.00-4.00 sec 1.65 GBytes 14.2 Gbits/sec [ 5] 4.00-5.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 5.00-6.00 sec 1.67 GBytes 14.3 Gbits/sec [ 5] 6.00-7.00 sec 1.64 GBytes 14.1 Gbits/sec [ 5] 7.00-8.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 8.00-9.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 9.00-10.00 sec 1.64 GBytes 14.1 Gbits/sec [ 5] 10.00-10.20 sec 333 MBytes 14.3 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.20 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.20 sec 16.1 GBytes 13.5 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated [root@dockersrv02 ~]#@notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
@daddygo thanks for the links...
However I suppose I don't see the relationship to my iperf3 speeds in this case.
BTW, the ESXi box is very capable in terms of CPU horsepower IMO:
2x Xeon E5-2695 v2 @ 2.40GHz
Don't get me wrong, my current config is obviously capable of saturating my internet connection (at least downstream) with speed tests consistently around 1500-1600 Mbit/s. But ingress isn't always so stable (which I know is not so easy).
I just did a quick iperf3 test with the offloads disabled (current working config).
Interestingly enough the slow iperf3 speeds are only TO pfsense (pfsense being the ipef3 server). Going the other way (CentOs box being the iperf3 server) I'm seeing very good speeds as I would expect.
I'm not quite sure where I can make the improvements reuiqred in the incoming connections to pfsense where I'm seeing only ~2.8Gbit/s
See results (from the CentOS box):
Connecting to host 10.10.0.254, port 5201 [ 4] local 10.10.1.32 port 40738 connected to 10.10.0.254 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec 34 509 KBytes [ 4] 1.00-2.00 sec 345 MBytes 2.89 Gbits/sec 215 431 KBytes [ 4] 2.00-3.00 sec 356 MBytes 2.99 Gbits/sec 78 361 KBytes [ 4] 3.00-4.00 sec 332 MBytes 2.79 Gbits/sec 57 318 KBytes [ 4] 4.00-5.00 sec 321 MBytes 2.68 Gbits/sec 69 393 KBytes [ 4] 5.00-6.00 sec 322 MBytes 2.71 Gbits/sec 134 501 KBytes [ 4] 6.00-7.00 sec 325 MBytes 2.73 Gbits/sec 42 498 KBytes [ 4] 7.00-8.00 sec 325 MBytes 2.73 Gbits/sec 113 452 KBytes [ 4] 8.00-9.00 sec 334 MBytes 2.80 Gbits/sec 82 458 KBytes [ 4] 9.00-10.00 sec 335 MBytes 2.81 Gbits/sec 26 540 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 3.21 GBytes 2.76 Gbits/sec 850 sender [ 4] 0.00-10.00 sec 3.21 GBytes 2.76 Gbits/sec receiver iperf Done. [root@dockersrv02 ~]# iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 10.10.0.254, port 57649 [ 5] local 10.10.1.32 port 5201 connected to 10.10.0.254 port 64408 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.04 GBytes 8.92 Gbits/sec [ 5] 1.00-2.00 sec 1.56 GBytes 13.4 Gbits/sec [ 5] 2.00-3.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 3.00-4.00 sec 1.65 GBytes 14.2 Gbits/sec [ 5] 4.00-5.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 5.00-6.00 sec 1.67 GBytes 14.3 Gbits/sec [ 5] 6.00-7.00 sec 1.64 GBytes 14.1 Gbits/sec [ 5] 7.00-8.00 sec 1.66 GBytes 14.2 Gbits/sec [ 5] 8.00-9.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 9.00-10.00 sec 1.64 GBytes 14.1 Gbits/sec [ 5] 10.00-10.20 sec 333 MBytes 14.3 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.20 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.20 sec 16.1 GBytes 13.5 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- ^Ciperf3: interrupt - the server has terminated [root@dockersrv02 ~]#First golden rule if
iperf, do NOT put it on the pfSense box. That puts double-duty on the pfSense box and you get incorrect results. pfSense is a router, and thus is optimized for routing traffic through the box in one interface and out another. If it has to also take CPU time to be "iperf" as well, then things suffer. The pfSense developers have mentioned here many times to never put theiperfclient nor server on the pfSense box. Put theiperfpieces on other boxes, then route their traffic through pfSense (such as from one VLAN to another or one physical port to another). -
@bmeeks said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
First golden rule if iperf, do NOT put it on the pfSense box. That puts double-duty on the pfSense box and you get incorrect results. pfSense is a router, and thus is optimized for routing traffic through the box in one interface and out another. If it has to also take CPU time to be "iperf" as well, then things suffer. The pfSense developers have mentioned here many times to never put the iperf client nor server on the pfSense box. Put the iperf pieces on other boxes, then route their traffic through pfSense (such as from one VLAN to another or one physical port to another).
Well that makes zero sense to me. iperf (client and server) is installed by default in pfsense and is accessible via the diagnostic menu in the GUI.
Also we're not talking about some low power box, as you can see from the specs I listed above there is ZERO issues with horsepower here. ifperf isn't that resource hungry, I don't think the CPU even goes over 15% when I run the tests.
-
@notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
@bmeeks said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
First golden rule if iperf, do NOT put it on the pfSense box. That puts double-duty on the pfSense box and you get incorrect results. pfSense is a router, and thus is optimized for routing traffic through the box in one interface and out another. If it has to also take CPU time to be "iperf" as well, then things suffer. The pfSense developers have mentioned here many times to never put the iperf client nor server on the pfSense box. Put the iperf pieces on other boxes, then route their traffic through pfSense (such as from one VLAN to another or one physical port to another).
Well that makes zero sense to me. iperf (client and server) is installed by default in pfsense and is accessible via the diagnostic menu in the GUI.
Also we're not talking about some low power box, as you can see from the specs I listed above there is ZERO issues with horsepower here. ifperf isn't that resource hungry, I don't think the CPU even goes over 15% when I run the tests.
I'm simply repeating what the developers have posted here several times. There may well be more going on under the cover than the simple CPU usage stat shows. You said yourself, that putting the iperf server piece on a client (instead of pfSense) made a huge difference.
-
@bmeeks said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
I'm simply repeating what the developers have posted here several times. There may well be more going on under the cover than the simple CPU usage stat shows. You said yourself, that putting the iperf server piece on a client (instead of pfSense) made a huge difference.
I never said that?
I think you're misinformed here.
Thanks anyway.
-
@notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
@bmeeks said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
I'm simply repeating what the developers have posted here several times. There may well be more going on under the cover than the simple CPU usage stat shows. You said yourself, that putting the iperf server piece on a client (instead of pfSense) made a huge difference.
I never said that?
I think you're misinformed here.
Thanks anyway.
I don't want to argue, I was simply offering some advice. Here is what your post said --
Interestingly enough the slow iperf3 speeds are only TO pfsense (pfsense being the ipef3 server). Going the other way (CentOs box being the iperf3 server) I'm seeing very good speeds as I would expect.
-
@bmeeks said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
I don't want to argue, I was simply offering some advice. Here is what your post said --
So if you read that again - you will see that what I said is not what you think I said.
-
I would still like to see test through pfsense vs to or from pfsense.
This has been discussed many times. Unless you plan on moving files/data to or from actually pfsense.. Like running a proxy?
A more valid test to see if you are getting the speeds you should have is "through" pfsense..
-
@johnpoz said in iperf3 speeds confusing - fast = bad internet slow = good internet?:
I would still like to see test through pfsense vs to or from pfsense.
If you can give me an idea of how to perform that test I would be happy to do so.
Otherwise, yes I can get my full 1500/1000 (or slightly above on the downstream) WAN throughput consistently even with pfNG + Suricata running).
However just like the iperf tests show, the upload speeds are not quite as good as download/ingress speeds.
What I mean by that is - while I can generally get 1500-1600 Mbit/s down, the upload is rarely faster than 800-850 Mbit. I'm testing from a 10Gbe connected VM by the way. While understand there is overhead involved I couldn't help be curious about the iperf tests showing FAR slower network speeds TO pfsense compared to FROM pfsense.
I don't always put too much weight on internet speed testing as it's out of my control when it's past the gateway anyways.
