iperf bitrate differences... why?

gniting

setup:

192.168.7.1=pfsense box
192.168.7.2=local server running DNS and other stuff (hardwired into an unmanaged switch which is connected to pfsense)
192.168.7.3=macbook (wifi connection)

when I run iperf in server mode on local server, and connect to it from pfsense or macbook, I get expected results (~900mbps on pfsense and ~100mbps on macbook).

However, when I run iperf in server mode on pfsense, my bitrate drops to ~1mbps when connecting from either the local server or the macbook. No changes made to network config.

How can pfsense's outbound performance be so much better than when pfsense is "accepting" connections?

stephenw10

Are you seeing that result in both directions when running the iperf server on the local-server?

1Mbps is so low it can only really be explained by either deliberate traffic shaping or a low level connection issue, bad cable, bad NIC etc.
In it's normal mode (without the -R switch) iperf3 sends traffic from the client to the server. So the restriction you're seeing there would be upload speed against speedtest.net for example.

Steve

gniting

@stephenw10 said in iperf bitrate differences... why?:

Are you seeing that result in both directions when running the iperf server on the local-server?

1Mbps is so low it can only really be explained by either deliberate traffic shaping or a low level connection issue, bad cable, bad NIC etc.
In it's normal mode (without the -R switch) iperf3 sends traffic from the client to the server. So the restriction you're seeing there would be upload speed against speedtest.net for example.

Steve

I only see that result when iperf is running on pfsense in server mode. When pfsense acts as an iperf client, the numbers are good (~900mbps). I thought it was a bad cable too but then the same bad results surface when pfsense is running in server mode and I connect another system in client mode (MacBook over wifi).

johnpoz

We have been over this multiple times.. Testing to and from pfsense for iperf is not really good test.

But testing from server to client wouldn't even be going through pfsense at all, unless the client and server are on different vlans?

But your testing to and from pfsense should be way higher than 1mbps.. That seems like something really wrong, like a duplex mismatch or something.

If pfsense a vm? What hardware is it? Do you get normal speeds doing speedtest through pfsense?

$ iperf3.exe -c 192.168.9.253
Connecting to host 192.168.9.253, port 5201
[  5] local 192.168.9.100 port 62207 connected to 192.168.9.253 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  74.8 MBytes   627 Mbits/sec
[  5]   1.00-2.00   sec  77.0 MBytes   646 Mbits/sec
[  5]   2.00-3.00   sec  77.9 MBytes   653 Mbits/sec
[  5]   3.00-4.00   sec  76.5 MBytes   641 Mbits/sec
[  5]   4.00-5.00   sec  77.8 MBytes   652 Mbits/sec
[  5]   5.00-6.00   sec  79.2 MBytes   665 Mbits/sec
[  5]   6.00-7.00   sec  78.9 MBytes   662 Mbits/sec
[  5]   7.00-8.00   sec  79.1 MBytes   663 Mbits/sec
[  5]   8.00-9.00   sec  76.9 MBytes   645 Mbits/sec
[  5]   9.00-10.00  sec  79.1 MBytes   664 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   777 MBytes   652 Mbits/sec                  sender
[  5]   0.00-10.00  sec   777 MBytes   652 Mbits/sec                  receiver

$ iperf3.exe -c 192.168.9.253 -R
Connecting to host 192.168.9.253, port 5201
Reverse mode, remote host 192.168.9.253 is sending
[  5] local 192.168.9.100 port 62210 connected to 192.168.9.253 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  97.7 MBytes   819 Mbits/sec
[  5]   1.00-2.00   sec   104 MBytes   873 Mbits/sec
[  5]   2.00-3.00   sec   110 MBytes   925 Mbits/sec
[  5]   3.00-4.00   sec   114 MBytes   952 Mbits/sec
[  5]   4.00-5.00   sec   113 MBytes   949 Mbits/sec
[  5]   5.00-6.00   sec  80.3 MBytes   673 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec
[  5]   7.00-8.00   sec   113 MBytes   947 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   943 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   940 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.05 GBytes   898 Mbits/sec  1844             sender
[  5]   0.00-10.00  sec  1.04 GBytes   896 Mbits/sec                  receiver

The above is more like what should be seen with decent hardware.. Not 1mbps.. That is to and from my sg4860..

stephenw10

Yep testing to/from pfSense will always give a worse result that testing through it. But it can be a useful test to prove a link is not fundamentally bad.

However something else it may show up is bad hardware off-loading in your NIC. That will only affect connections to/from pfSense exactly as you are seeing.
So go to System > Adv > Networking and make sure all the hardware off-loading options are disabled.

Probably not an issue but note that running pfSense in iperf server mode with the client specifying the -R reverse switch so the server sends is not the same as running pfSense in client mode where it opens outbound connections.

Steve

gniting

Yes, it is official, I am stupid!

I use limiters and I had them also acting on my LAN interface! I've now updated the relevant firewall rule to only apply when "destination NOT LAN net." With that change, iperf is now back to normal.

Connecting to host 192.168.7.1, port 5201
[  5] local 192.168.7.2 port 58164 connected to 192.168.7.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  74.1 MBytes   622 Mbits/sec    0    840 KBytes
[  5]   1.00-2.00   sec  70.0 MBytes   587 Mbits/sec    0   1.53 MBytes
[  5]   2.00-3.00   sec  70.0 MBytes   587 Mbits/sec    0   1.70 MBytes
[  5]   3.00-4.00   sec  71.2 MBytes   598 Mbits/sec    1   1.24 MBytes
[  5]   4.00-5.00   sec  70.0 MBytes   587 Mbits/sec    0   1.37 MBytes
[  5]   5.00-6.00   sec  70.0 MBytes   587 Mbits/sec    0   1.47 MBytes
[  5]   6.00-7.00   sec  70.0 MBytes   587 Mbits/sec    0   1.55 MBytes
[  5]   7.00-8.00   sec  70.0 MBytes   587 Mbits/sec    0   1.61 MBytes
[  5]   8.00-9.00   sec  70.0 MBytes   587 Mbits/sec    1   1.18 MBytes
[  5]   9.00-10.00  sec  70.0 MBytes   587 Mbits/sec    0   1.26 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   705 MBytes   592 Mbits/sec    2             sender
[  5]   0.00-10.02  sec   703 MBytes   588 Mbits/sec                  receiver

iperf Done.

Thank you @johnpoz @stephenw10 for the hints and setting my mind on the right path.