Strange - Very slow routing performance while iperf shows gigabit speed
-
Hi there,
I ran into a really strange problem - Fresh pfSense installation, with no packages installed. It all seems working fine, except that speedtest shows something like 0.4 Mbits kind of speed (Internet speed is 200/200 and it is working fine). I did iperf to confirm that the network adapters are not likely the problem - My PC to/from the LAN interface reaches 1Gbps speed. The WAN interface (with a 192.168.x.x address, it is inside a LAN) to another PC also reaches 1Gbps speed. I have gone through the areas in pfSense that I thought might be relevant, but could not figure out anything unusual: Firewall, DNS, DHCP.
pfSense is installed as a VM on Proxmox. But since the interfaces are fine, I feel like VM is not the issue?
Any suggestions are welcome. Thanks much!
Wayne
-
What about an iperf test between a PC on the LAN and a PC on the WAN?
That level of throttling usually indicates some thing low level like a speed/duplex mismatch somewhere or a bad cable.
Steve
-
Thanks much, Steve.
I did some iperf tests. I am quite sure it is not a bad cable problem but some software setting issue. All speed tests are at the gigabit level, except if routed through pfSense. All NICs and switches are gigabit. So, what settings might be the issue?
A (WAN side PC): 10.10.10.100
B (pfSense box WAN interface): 10.10.10.10
C (pfSense box LAN interface): 192.168.1.1
D (LAN side PC): 192.168.1.2iperf3 test results are:
A < - > B: Gigabit speed.
C < - > D: Gigabit speed.
D --> A: miserable speed like 4~5 kDetailed iperf reports:
Case 1: A (server), B (client)
Connecting to host 10.10.10.100, port 5201
[ 5] local 10.10.10.10 port 1445 connected to 10.10.10.100 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 93.1 MBytes 780 Mbits/sec 0 449 KBytes
[ 5] 1.00-2.00 sec 100 MBytes 840 Mbits/sec 0 669 KBytes
[ 5] 2.00-3.00 sec 107 MBytes 899 Mbits/sec 0 831 KBytes
[ 5] 3.00-4.00 sec 108 MBytes 907 Mbits/sec 0 981 KBytes
[ 5] 4.00-5.00 sec 109 MBytes 911 Mbits/sec 0 1.08 MBytes
[ 5] 5.00-6.00 sec 96.5 MBytes 810 Mbits/sec 0 1.20 MBytes
[ 5] 6.00-7.00 sec 102 MBytes 852 Mbits/sec 0 1.30 MBytes
[ 5] 7.00-8.00 sec 110 MBytes 926 Mbits/sec 0 1.41 MBytes
[ 5] 8.00-9.00 sec 102 MBytes 853 Mbits/sec 132 460 KBytes
[ 5] 9.00-10.01 sec 107 MBytes 892 Mbits/sec 0 695 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 1.01 GBytes 867 Mbits/sec 132 sender
[ 5] 0.00-10.01 sec 1.01 GBytes 865 Mbits/sec receiverCase 2: A (client), B (server). I allowed “any” on the pfSense WAN interface.
Connecting to host 10.10.10.10, port 5201
[ 5] local 10.10.10.100 port 50379 connected to 10.10.10.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 105 MBytes 878 Mbits/sec 0 844 KBytes
[ 5] 1.00-2.00 sec 105 MBytes 881 Mbits/sec 0 1.61 MBytes
[ 5] 2.00-3.01 sec 103 MBytes 855 Mbits/sec 0 2.00 MBytes
[ 5] 3.01-4.00 sec 102 MBytes 867 Mbits/sec 0 2.00 MBytes
[ 5] 4.00-5.01 sec 106 MBytes 882 Mbits/sec 0 2.00 MBytes
[ 5] 5.01-6.00 sec 102 MBytes 864 Mbits/sec 0 2.00 MBytes
[ 5] 6.00-7.00 sec 103 MBytes 861 Mbits/sec 0 2.00 MBytes
[ 5] 7.00-8.00 sec 103 MBytes 868 Mbits/sec 0 2.00 MBytes
[ 5] 8.00-9.00 sec 89.8 MBytes 752 Mbits/sec 0 2.00 MBytes
[ 5] 9.00-10.00 sec 101 MBytes 845 Mbits/sec 0 2.00 MBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1020 MBytes 855 Mbits/sec 0 sender
[ 5] 0.00-10.34 sec 1020 MBytes 827 Mbits/sec receiverCase 3: C (server), D (client)
Connecting to host 192.168.1.1, port 5201
[ 4] local 192.168.1.2 port 2408 connected to 192.168.1.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 104 MBytes 876 Mbits/sec
[ 4] 1.00-2.00 sec 106 MBytes 888 Mbits/sec
[ 4] 2.00-3.00 sec 108 MBytes 903 Mbits/sec
[ 4] 3.00-4.00 sec 111 MBytes 930 Mbits/sec
[ 4] 4.00-5.00 sec 107 MBytes 898 Mbits/sec
[ 4] 5.00-6.00 sec 106 MBytes 892 Mbits/sec
[ 4] 6.00-7.00 sec 112 MBytes 936 Mbits/sec
[ 4] 7.00-8.00 sec 113 MBytes 949 Mbits/sec
[ 4] 8.00-9.00 sec 110 MBytes 926 Mbits/sec
[ 4] 9.00-10.00 sec 110 MBytes 925 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.06 GBytes 912 Mbits/sec sender
[ 4] 0.00-10.00 sec 1.06 GBytes 912 Mbits/sec receiverCase 5: C (client), D (server)
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 60417 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 111 MBytes 930 Mbits/sec 0 210 KBytes
[ 5] 1.00-2.00 sec 110 MBytes 921 Mbits/sec 0 210 KBytes
[ 5] 2.00-3.00 sec 104 MBytes 871 Mbits/sec 0 210 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 210 KBytes
[ 5] 4.00-5.00 sec 108 MBytes 910 Mbits/sec 0 210 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 936 Mbits/sec 0 210 KBytes
[ 5] 6.00-7.00 sec 107 MBytes 899 Mbits/sec 0 210 KBytes
[ 5] 7.00-8.00 sec 109 MBytes 911 Mbits/sec 0 210 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec 0 210 KBytes
[ 5] 9.00-10.00 sec 106 MBytes 893 Mbits/sec 0 210 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.06 GBytes 913 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.06 GBytes 913 Mbits/sec receiverCase 5: A (server), D (client)
Connecting to host 10.10.10.100, port 5201
[ 4] local 192.168.1.2 port 2416 connected to 10.10.10.100 port 5201
Connecting to host 10.10.10.100, port 5201
[ 4] local 192.168.1.2 port 2416 connected to 10.10.10.100 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 256 KBytes 2.08 Mbits/sec
[ 4] 1.01-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.01-3.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 3.00-4.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 4.01-5.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 5.01-6.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 6.01-7.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 7.01-8.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 8.01-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 9.01-10.02 sec 0.00 Bytes 0.00 bits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.02 sec 256 KBytes 209 Kbits/sec sender
[ 4] 0.00-10.02 sec 4.28 KBytes 3.50 Kbits/sec receiverThe report on the server side for the same test.
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.02 sec 1.43 KBytes 11.4 Kbits/sec
[ 5] 1.02-2.01 sec 1.43 KBytes 11.9 Kbits/sec
[ 5] 2.01-3.02 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.02-4.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.06-5.05 sec 1.43 KBytes 11.7 Kbits/sec
[ 5] 5.05-6.04 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.04-7.07 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.07-8.02 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.02-9.05 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.05-10.02 sec 0.00 Bytes 0.00 bits/sec
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.02 sec 4.28 KBytes 3.50 Kbits/sec receiverServer listening on 5201
-
@wxppro said in Strange - Very slow routing performance while iperf shows gigabit speed:
pfSense is installed as a VM on Proxmox.
There are some caveats to running on proxmox.. Look in the virtual section.. Pretty sure its a sticky..
Here
"https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-proxmox.html"did you do this?
Because the hardware checksum offload is not yet disabled, accessing pfSense webGUI might be sluggish. This is NORMAL and is fixed in the following step.To disable hardware checksum offload, navigate under System > Advanced and select Networking tab. Under Networking Interfaces section check the Disable hardware checksum offload and click save. Reboot will be required after this step.
-
Mmm, are you using VirtIO interfaces (vtnet)? Some sort of hardware pass-through?
I use Proxmox I have not seen that with the default settings. However I only test to an internal VM client so may not be hitting it.
Steve
-
@johnpoz said in Strange - Very slow routing performance while iperf shows gigabit speed:
@wxppro said in Strange - Very slow routing performance while iperf shows gigabit speed:
pfSense is installed as a VM on Proxmox.
There are some caveats to running on proxmox.. Look in the virtual section.. Pretty sure its a sticky..
Here
"https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-proxmox.html"did you do this?
Because the hardware checksum offload is not yet disabled, accessing pfSense webGUI might be sluggish. This is NORMAL and is fixed in the following step.To disable hardware checksum offload, navigate under System > Advanced and select Networking tab. Under Networking Interfaces section check the Disable hardware checksum offload and click save. Reboot will be required after this step.
That is the issue - hardware checksum offload.
I feel quite stupid. I have done pfSense virtualization in ESXi and Hyper-V before. Just recently started to work with proxmox. It looks quite easy to set up a VM. The process I did was identical to the guide.
But I did not know this hardware checksum offload can have such a huge influence. Besides, the NICs are Intel but not Realtek...Good to learn some new knowledge. Thanks much!
-
@stephenw10 said in Strange - Very slow routing performance while iperf shows gigabit speed:
Mmm, are you using VirtIO interfaces (vtnet)? Some sort of hardware pass-through?
I use Proxmox I have not seen that with the default settings. However I only test to an internal VM client so may not be hitting it.
Steve
Yes, Linux bridge, VirtIO (paravirtualized), and the interface ended up showing up as vtnet0 and vtnet 1 in pfSense.
-
So just clarification, you disabled checksum offload and your performance is where you would expect it now for a vm?
-
@johnpoz said in Strange - Very slow routing performance while iperf shows gigabit speed:
So just clarification, you disabled checksum offload and your performance is where you would expect it now for a vm?
Yes, gigabit speed now. Case 5 is similar to cases 1 - 4.
-
Nice.
