Slow Inter-VLAN Routing
-
Hi all,
I've been using a pfSense box as my main router for a few years now, and don't have any regrets whatsoever. It's been proven a very stable system.
About a month ago, I decided to go a step further as I had a VLAN-capable switch on my hands. I set up VLANS: among others, one for my servers and one for my home network, so I could more precisely control the communication between the two network.
All works fine, only are my inter-VLAN transferring speeds over the network limit to around 5-6 MB/sec, which is ofcourse painfully slow these days, especially when doing large backups or file transfers in general to my file server.
The hardware I use:
- Switch: TP-Link TL-SG2216
- pfSense Box
– CPU: Intel Core 2 Duo, 2,4 Ghz
-- RAM: 4GB DDR2
-- Samsung 120Gig SSD (in case that matters, which I highly doubt)
-- Intel 2-port PRO/1000 NIC, PCI-Express, 2 port (one LAN trunk port, one WAN)
If anyone out here could care to help me out, that would be great! ;) :)
Cheers,
Ruud -
Are you at 100mbps on your interface? Vlans cut bandwidth in half for intervlan traffic.. You do understand right? So if you were at 100mbps on the interface than yeah 5 to 6 MBps would seem about correct.
With a vlan on the same physical interface you hairpin all traffic, so yes you cut it in half the available bandwidth..
-
Hi Johnpoz,
Thank you for your response. No, I am on a gigabit network, 1000 Mbps, so no luck there.
I've noticed a new release of pfSense has come out, maybe upgrading to 2.3.4_1 will solve my problem.
Thanks,
Ruud -
Update: Upgrading to 2.3.4-RELEASE-p1 did not solve my problem. Sad face…
I'm transferring some files right now as a test case and I'm getting around 5,5 MB/sec throughput and a CPU usage of around 15%.
-
How are you testing?
Are you positive you negotiated at 1000-full?
-
At first, before I set up the VLANs, my network was running smoothly at 1000 Mbps, as all my network equipment is 1000 Mbps capable.
In the pfSense dashboard, I can see my interfaces and their advertised speeds: see attached image (LAN = no VLAN, the other two local networks are VLANs).
My switches all have gigabit connections to eachother, so does my macbook (Thunderbolt gigabit adapter) and my desktop computer (On-board Intel Gigabit NIC)-Ruud
-
"(LAN = no VLAN, the other two local networks are VLANs)."
And the Vlans are on what interface??
em1 (lan)
em1 (opt1 vlan 100)When lan talks to opt1 it is a hairpin and yes your available bandwidth is cut in half when lan talks to opt1
em1 (lan)
em2 (opt1)When you have this and lan and opt1 are talking to each other you do not have a hairpin and you get full bandwidth..
em1 (lan)
em2 (opt1)
em2 (opt2 vlan 100)when lan and opt1 talk to each other you have full, minus what opt2 is using since opt1 and opt2 share the same physical interface.
If opt1 is talking to opt2 you are hairpin and cut in half for available bandwidth… BTW your picture is not working get 502 bad gateway. Why do you not just attach any images direct to the post?
-
Hi Johnpoz,
Sorry for the bad link, I have modified the post and uploaded the image.
From what I can see, I should be in the third option you described. I've provided a screenshot of my interface assignments in attachment. Can you confirm?
Thanks,
Ruud
 -
So all your networks on on em0, so any traffic between any network would be hairpined and your available bandwidth cut. And then reduced even further by any traffic from the 3rd network say to the internet.
5 or 6 MBytes per second does seem really low.. But without knowing the full setup, maybe you have wifi involved? Maybe you have some uplink or port that is only at 100mbps that you are checking bandwidth between?
But with your current setup there are no devices that are on different networks lan, homelan or server lan that could talk to something in a different network where your bandwidth would not be limited by the vlans sharing the same physical interface. This is the nature of the beast and vlans.. And why normally uplinks between switches are lagged in some fashion or use a higher bandwidth link than clients. Because all clients on a switch are going to be sharing the uplink bandwidth, and when you have traffic that has to go through the uplink more than once you cut the available bandwidth in half, minus all other traffic on the link at the same time, etc.
Do you have any downstream switches where you might have 100mbps connections to your end devices? Or any devices that are wifi that your using for testing?
See attached so on the left where you have different uplinks to the router you would be able to get full speed on the left. Same could be said for middle type connection since your not sharing a path. But in the right most setup there are multiple places where you have hairpin connections and possible bottlenecks for speed.
Your devices talking to each other are hairpin on the router itself so this bandwidth is /2 you also have possible of speed issue on this connection maybe that is only 100 or maybe there is duplex issue that would drastically limit speed.. you also have an uplink that goes to a downstream switch so even if the router itself had 2 uplinks for the different vlans they are sharing the pipe on the uplink to the other switch and traffic is hairpinned. Or this link could be starved for bandwidth. Maybe you have device on the top switch in 10.0.1/24 as well talking to another device in 10.0.1 on the bottom switch eating up this pipe.
Either of your devices might be only connected at 100 or again duplex issue, etc. If you want to test the routing speed of pfsense you really need to take all the other variables out of the equation or fully understand them and make sure they do not come into play. The 2 left layouts are better suited for checking routing speed between 2 segments when you know that all the links have no issues.
Hope that helps shed some light on where we can look to see what is causing you your slowness.
-
Hi,
Thank you for you clear explanation! I do understand the things you say, but I don't think there's anything in the network itself that could cause this issue. I'm afraid this is going to be down to some hardware incompatibilities (even though I use Intel NICs…)
I'll give you a brief overview of my network.
pfSense <--- VLAN Trunk ---> TP-Link TL-SG2216 (Main switch) ---> ServerLAN (port 1-8) + HomeLAN (port 9-16)
Port 0 on the switch is the trunk port.
I do have another switch downstairs that is capable to VLAN networking, but I have done my tests connecting my (gigabite wired) laptop (reporting it is connected at gigabit speed) directly into the main TP-Link switch.Now ofcourse: If I hook my laptop up to the ServerLAN ports, I can do a full gigabit speed file transfer to my fileserver (which is ofcourse on ServerLAN), as no routing needs to be done by the pfSense box. If I however connect my laptop up to the HomeLAN ports on the switch, I have to use the trunk pipe and pfSense needs to intervene, leading to slow (5-6 MB/sec) transfers.
So my problem must lie somewhere in the heart of the network: the main switch and the pfsense box.
-
Hi Ruud
And did you solved your issue? I have same issue. Routing between subnets is very slow (199kbit measured with iperf3). NAT between WAN and LAN (all VLANs) is working very well. -
Hi,
Someone solved this issue? I have same problem. When I copy file from another vlan, the speed is about 30MB/ss then drop to 0 bytes/s immediately -
@tientun I have the same issue. Strangely i'm pretty confident that this did not occur on older releases (but have no proof). I have multiple vlans, and used to connect to a windows server on the "main" vlan without issue. I recently discovered that SMB became unusable.
Testing with iPerf I see performance basically start OK for a very short window then completely die to zero. I'll post logs soon. -
Please stop resurrecting year-old topics with completely different versions of pfsense and hardware but open a new topic, explain your problem, hardware and configuration/setup and ask for help that way. It's no need to reply to such old posts with "I have exactly the same problem" as you most certainly do not. It may seem like the same effects but I certainly doubt it's the same reason.
-
@Griffo said in Slow Inter-VLAN Routing:
@tientun I have the same issue. Strangely i'm pretty confident that this did not occur on older releases (but have no proof). I have multiple vlans, and used to connect to a windows server on the "main" vlan without issue. I recently discovered that SMB became unusable.
Testing with iPerf I see performance basically start OK for a very short window then completely die to zero. I'll post logs soon.What NIC do you use? I guess this problem is related to my realtek NIC.
