Slow speed between VLANs
-
@pbnet
Yes i meant physical interfaces.What is your pfSense utilization when running the inter-vlan transfer ?
Adding another interface isn't going to do magic, if your CPU is congested./Bingo
-
@bingo600 CPU Usage when doing an IPERF3 from VLAN5 to VLAN5 gets me:
[SUM] 0.00-10.00 sec 8.85 GBytes 7.60 Gbits/sec sender
[SUM] 0.00-10.00 sec 8.83 GBytes 7.58 Gbits/sec receiverwith a very decent CPU usage:
-
Inter-VLAN CPU usage is about 71%
and this is the speed I get in IPFERF3:
[SUM] 0.00-10.00 sec 3.72 GBytes 3.20 Gbits/sec sender
[SUM] 0.00-10.00 sec 3.71 GBytes 3.19 Gbits/sec receiver -
@pbnet
Sorry .. You will have to inline/upload the graphs , if you want me to have a look.But i guess it's not needed ...
Same vlan xfer would be on L2 (handled by the switch) , and not passing pfSense at all.
Intervlan xfer is where all the packages have to pass pfsense , and load the pfSense cpu and interface(s) with 2 x 2Gb.
I have no idea what the performance level of your current hardware is.
You might have to ask Netgate if (71% cpu load) is "normal", in the given test scenario.Ps:
A pfSense based iperf would probably not give an optimal answer.
Always use iperf on the endpoints, as i suppose you do./Bingo
-
@pbnet said in Slow speed between VLANs:
Any idea what can I do to improve inter-VLAN bandwidth ?
One thing would be use different physical interfaces for the uplinks of these vlans. When you share the same physical interface for vlans that are talking to each other, you are hairpinning the traffic over the same physical interface it would be expected to not see full wire speed.
For optimal performance of intervlan traffic it is best to put these vlans that will be talking a lot between them on different physical interfaces. But seems your limited to the 1 interface via sfp+ so your kind of hindered in doing that.
Now that being said I would hope you would see more than what your seeing.. 3Gbps does seem low.. I would expect atleast 1/2 of physical interface speeds or atleast really freaking close or even above etc.. But the 7.x and 3.x something via the hairpin isn't all that out of wack depending..
71% cpu does seem a bit high as well for a 7100 (which is a beast).. Are you doing anything that could hinder the speed - say IPS or something? Or ntopng?
-
@johnpoz OK, so I opened a support case with Netgate. They asked me to remove the following packages: snort, darkstat, ntopng, bandwidthd, haproxy, squid
After removing them and doing a new test, I got: 4.18Gbps using IPERF3 and still a CPU load of 70%.
To be honest it's really far that what they advertise that the XG7100-1U can do.
I guess I also have to ask Netgate if adding a 4-Port SFP+ NIC will void my warranty.
I already purchased the device last year with an additional 4Port- 1Gbps NIC, so the raiser card should be already in there.Thanks.
-
@pbnet said in Slow speed between VLANs:
To be honest it's really far that what they advertise that the XG7100-1U can do.
Where did they say you would see wirespeed via a hairpin? I don't see any benchmarks for that.
Now if your talking the 9.85 in this benchmark
IPERF3 Traffic: 9.85 Gbps
If you have 2 different sfp+ at 10ge, and route between them what do you see.. Seems to me your doing vlans on same physical interface. Which is hairpin, and yes this would be lower.
From the iperf 7.6Gbps test you showed seems to be between 2 devices not even going through pfsense at all? So the rest of your network and test devices can not achieve wirespeed?
For a fair test I would think you would have to be using 2 sfp+ connections at 10ge and routing/firewalling between those 2 interface.. If you can only achieve 4 some gbps then I would be disappointed as well.
Can you run a test with that scenario?
-
@johnpoz The XG7100-1U I have only has 2 SFP+ - one is for WAN and one is for LAN.
The tests between 2 machines on the same VLAN (without going though PFSense) is about 7 to 9Gbps.
So, sadly, I cannot do more tests since I don't have more SFP+ ports on the XG7100-1U.
And the solution offered by netgate to disable all packet filtering is not something I'm comfortable of doing.
I get the feeling they are not really trying to help, but rather finding all sorts of workardounds.
They also stressed that the performance tests are based on the maximum memory configuration for the device: I have 24GB or RAM in the XG7100-1U out of which only 7% is used, yet they don't want to explain the high CPU usage.
Let's see what will they come with next. -
OK. Final statement from Netgate: 4.18Gbps is the max I can get on the device based on this article: https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/switch-overview.html
Also based on this article: https://info.netgate.com/hubfs/website-assets/netgate-hardware-comparison-doc.pdf they mentioned that the throughput measurements are based upon maximum bidirectional traffic across all available ports. As all tests were run by maximizing throughput across available ports on the base model physical ports (XG-7100 with 10 ports). I have the max model https://shop.netgate.com/products/7100-base-pfsense, but it doesn't seem to count.
Really disappointed by the product :( -
@pbnet said in Slow speed between VLANs:
has 2 SFP+ - one is for WAN and one is for LAN.
There you go - you clearly have 2 that you could "test" with..
-
@johnpoz True.. Just have to do it in week-ends, since I need to move the WAN to another interface.
-
@pbnet the switch would have a limit routing between vlans because it has a 5gbps uplink lag.. So lets see what you get for test when using 2 different 10ge interface that is not hairpin and not routed through the 5 gbps uplink lagg.
That doc you linked too goes over that..
But that should not be the case when going through 2 different independent interfaces. If that is the case - then yes I feel you would have a valid point that this should be pointed out in the docs that routing between 2 10ge interfaces is not capable of close to wirespeed.
-
OK, so I used IX0 for VLAN10.
and here are the results:
and the CPU usage:
I'll update also the Netgate ticket
