Inter vlan routing speed
-
I have some issues with throughput for inter vlan routing. Currently I get around 20 MBps from my 1 Gbps link but was hoping for more.
Everything works as it should but not sure if the low speed is due to my hardware or some configuration error in pfsense.
Everything is setup as "router on a stick" as i want each vlan to pass through the firewall.
2.4.5-RELEASE-p1 with below hardware,
Intel(R) Atom(TM) CPU E3845 @ 1.91GHz
Current: 498 MHz, Max: 1909 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)
82583V Gigabit Network ConnectionAll hardware offloading settings are stock.
Are these results expected or should my hardware be able to route at higher throughput?
-
One thing to bear in mind with "router on a stick" is you're cutting your bandwidth, between VLANs, in half as the packets travel over the same cable twice. Also is that actually MB/s or Mb/s?
-
@jknott
Interface says full duplex so shouldnt it still be able to handle 1Gbps between vlans?Yeah, its MB/s (megabytes). Inside the vlan i can fully saturate the link.
-
Yes, it is full duplex, but every packet has to be sent twice over the same link, once in each direction. Don't forget, a computer attached to a network would normally be able to send and receive at the same time. If it was a straight connection, the data wouldn't be sent twice.
Data rates are normally measured in b/s, while file transfers in B/s. 20 MB = 160 Mb then each packet has to be sent twice, which will cause a slowdown. Also, how fast is the hardware you're running pfsense on. I have a 500/20 connection from my ISP and with the previous computer I used, I would generally get low 500s down. When that computer died, I replaced it with the one described in my sig. Here is a recent speedtest. I saw an immediate performance increase, so the old computer was limiting what I could get.
Bottom line, a Gb NIC doesn't guarantee Gb throughput, if the rest of the hardware isn't up to it. -
@voxel Yeah, router on a stick leans towards security, but the trade-off is performance. You'll be very hard-pressed to see anything approaching wire-speed with a router on a stick design.
Without assessing any other variables, if you have a want (or need) to firewall your VLANS, the only option I see adding a significant performance boost is adding a 2nd NIC and moving any VLANs where you want inter-vlan performance boosted to the 2nd NIC.
A performance-based design change option would be moving your VLANs to an L3 switch. Your inter-vlan traffic would then be handled by the L3 switch @ wire speed, but then you lose the ability to firewall your VLANs. Personally, I always lean towards performance, but your priorities may be different.
Another observation that probably isn't the main issue, but may be a contributing factor is the fact that your CPU is running at 1/4th its speed... I would verify that PowerD is enabled and set to either Hiadaptive or Maximum.
-
@marvosa
I have more nics so I'll try that approach and see if it makes any difference.Yeah, I've considered using an L3 switch instead but as the network is small and the only reason i use vlans is from a security perspective i might as well just skip vlans entirely.
PowerD is enabled and set to Hiadaptive but will try maximu as well.
During transfer the CPU peaks at about 30% but not sure at what Hz. -
@jknott
Ok, i thought that theoretically full duplex would mean that "router on a stick" would still be able to route 1Gbps. Will try more nics and see if that increases performance.Harware is stated in original post. Quad core Intel(R) Atom(TM) CPU E3845 @ 1.91GHz.
-
Whatever others might claim: it's perfectly possible to get near 930MB/s inter-vlan traffic over a single gbe interface in a lab/home setup.
Source: I've been doing that on multiple sites for over a decadeIt's either the CPU that cannot keep up or buggy drivers or it needs nic tuning
-
Yeah, it depends how you're testing. In a UDP test you can send and receive at close to line rate and see that as throughput. In, much more typical, TCP test though all the ACK traffic and and any other traffic the other way removes available bandwidth for the test traffic.
I would expect to see significantly better than 160Mbps in an iperf test there with that CPU.
Check the output of
top -aSHwhilst testing. Make sure you are not pinning a single CPU core.Steve
-
@stephenw10
My tests have been just plain file copy. From within the vlan they perform near max but from any external vlan they drop to 160 Mbps.Will check cpu utilization and test if dedicated nics make any difference.
Is it possible I've made any configuration error with the vlans and the routing that could interfere?
-
Unlikely, it's just forwarding in and out between two directly connected subnets.
Some MTU mismatch could cause that sort of problem.
Steve
