iperf3: Slower transfer speeds between VLANs vs same VLAN
-
Perhaps the OP can try an experiment with separate LANs vs VLANs, to see if it's a VLAN issue or routing.
While Ethernet is full duplex, there is more to it than just that. Normally, with TCP, you'd have data going one way and acks the other. So, with VLANs sharing the same wire, the data has to go out and back and likewise the acks. There also the pfSense performance to consider, etc. When on the same VLAN, pfSense is not at all involved in passing the traffic. I'm also assuming there's a switch in there somewhere. It's fine to speculate, but without actual tests, we don't know for sure. So, run some tests to see where the bottleneck is.
-
@JKnott said in iperf3: Slower transfer speeds between VLANs vs same VLAN:
The OP mentioned different VMs on different VLANs. Are those VMs running in the same box?
Yes I do have VMs the run all on the same box and are on different VLANs (ie; my fileserver and plexserver are on the VLAN 1 while the other two VMS are on VLAN 100), and yes I do have a managed switch (Netgear GS728TPv2).
I will eventually setup different LANs to test and see.
-
I just tried a simple experiment. I have a couple of VirtualBox VMs on a computer. I watched the Ethernet port with Wireshark and pinged one VM from the other. I did not see any packet captures of this, which shows that when connecting between VMs on the same box, the packets aren't even hitting the interface. This would result in much better performance than if the packets actually went over the wire and through the switch, let alone through pfSense. So, your comparing VM to VM in the same box is simply not a valid comparison with actually going out over the network.
-
@JKnott so I just tested iperf from my VM host (192.168.63.2) to a laptop on VLAN 10 (192.168.10.25)
The 192.168.63.10 address is OVS. Is there a way for iperf to listen/send on a specific interface (an interface that is not bridged)?
EDIT: Nvm, saw that there is a bind option. However the speeds are the same.
-
It looks like there's a 100 Mb connection somewhere, not Gb.
. -
@JKnott Well I do have a device that only operates @100Mb. Would that effect everything else on the switch tho? I would think the specific port the device is connected to would run at that speed while everything else uses 1Gb.
-
Correct, only that particular port would be running at 100 Mb, but your test (above) shows you're getting just under 100 Mb, and far lower than you'd expect from a Gb port. Is the device you're testing with that 100 Mb device? If so, it will be the limiting factor, even if your computer is capable of Gb.
-
@JKnott no I was not testing with that device. However now that I think about it, I think the laptop NIC I tested with only runs 100Mb. I will have to double-check and report back. I may have to use my Mac which does run 1Gb on the NIC.
-
That could very well be it. Throughput will always be determined by the slowest device. Imagine you had 2 computers connected by a switch. If any of those 3 runs at 100 Mb, then that's the best you will do.
-
@JKnott Ok just confirmed that the laptop I used only runs @100Mb. Just tested again on my MacBook which is on VLAN 10 and the server is on VLAN 1.
-
Now try with them on different interfaces.
-
@JKnott Yes now it's closer to Gb speed being on the same VLAN.
-
That is what I suspected. When you're using VLANs, the same wire is used twice, which limits throughput.
While you were doing that, I did an experiment of my own. I ran speedtest, as I frequently do. Here are my results. This is on a 75 Mb down/10 up connection. As you can see, pfSense isn't causing much of a slow down.
-
@JKnott Well it l sounds like I'll have to look into setting up inter-vlan routing on my switch. I've never set it up this device before so that will be a project I can look into doing. Wonder how this will effect vlan routing with my VMs that are using OVS....woohoo, new things to test out!
-
@simon_lefisch said in iperf3: Slower transfer speeds between VLANs vs same VLAN:
ell it l sounds like I'll have to look into setting up inter-vlan routing on my switch
You will have the same issue. It's not pfSense causing this, it's using VLANs to connect VMs on the same computer. When you do that, all the traffic has to go out to the router or L3 switch and then back again over the same wire. This was demonstrated when you got better performance with a different interface.
-
@JKnott so what about using dedicated interfaces for each LAN? I would still have to segment on the switch using VLANS, but if I ran each LAN on it's own interface and enabled inter-vlan routing on the switch, would that yield better transfer speeds between VLANs? I would think not since it won't be traveling down the same wire.
-
You have to decide what it is you're trying to do. Are you really worried about the performance between VMs in the same computer? That's not the way multiple VMs are normally used, but as long as you do that, you will not get best performance and it doesn't make much difference whether pfsense or a L3 switch is used to route between the VLANs.
-
@JKnott I'm not worried about speed between the VMs. It's the speed between the VMs and the other physical devices on the LAN/WLAN.
-
Then you should be OK.
-
@JKnott cool, thanks for your insight. I'll experiment when I have some time to bring my network down and set everything up.
