Slow speed between VLANs

pbnet

Hello,
I have a Netgate XG7100-1U on which I defined 3VLANs on interface ix1 (SFP+ port). alt text

The Netgate XG7100-1U connects to a Mikrotik switch via a fiber-optics cable. The Mikrotik Switch is also connected to a second Mikrtotik Switch (still via 10Gbps SFP+) and a PC + server are connected to the second switch.

Network Topology

Now comes the problem:

if I do a speedtest (IPERF3) between the PC (VLAN5) and a VM that is on VLAN5, I get about 9Gbps which is perfect
if I do a speedtest (IPERF3) between the PC (VLAN5) and a VM that is on VLAN 10 or VLAN 30, I only get about 2Gbps, which is not OK.

Any idea what can I do to improve inter-VLAN bandwidth ?

Thanks.

bingo600

@pbnet
Assuming this is not pfSense congestion.

Spread your vlans on multiple pfSense interfaces.
You if you are just using one pfSense interface it's a "On a stick" setup , where every packet going to another vlan must pass (and share) the same 10G interface.

There might be some pfSense "tuning parameters too" , but i don't have the knowledge to advice on those.

/Bingo

pbnet

@bingo600 I would do that, unfortunately, I only have 1 10Gbps SFP+ available. The only way to use multiple interfaces (I hope you're referring to physical ports) would be to purchase a multi-SFP+ port add-on card.

bingo600

@pbnet
Yes i meant physical interfaces.

What is your pfSense utilization when running the inter-vlan transfer ?
Adding another interface isn't going to do magic, if your CPU is congested.

/Bingo

pbnet

@bingo600 CPU Usage when doing an IPERF3 from VLAN5 to VLAN5 gets me:

[SUM] 0.00-10.00 sec 8.85 GBytes 7.60 Gbits/sec sender
[SUM] 0.00-10.00 sec 8.83 GBytes 7.58 Gbits/sec receiver

with a very decent CPU usage:

CPUUsage

pbnet

Inter-VLAN CPU usage is about 71%

InterVLN

and this is the speed I get in IPFERF3:

[SUM] 0.00-10.00 sec 3.72 GBytes 3.20 Gbits/sec sender
[SUM] 0.00-10.00 sec 3.71 GBytes 3.19 Gbits/sec receiver

bingo600

@pbnet
Sorry .. You will have to inline/upload the graphs , if you want me to have a look.

But i guess it's not needed ...

Same vlan xfer would be on L2 (handled by the switch) , and not passing pfSense at all.

Intervlan xfer is where all the packages have to pass pfsense , and load the pfSense cpu and interface(s) with 2 x 2Gb.

I have no idea what the performance level of your current hardware is.
You might have to ask Netgate if (71% cpu load) is "normal", in the given test scenario.

Ps:
A pfSense based iperf would probably not give an optimal answer.
Always use iperf on the endpoints, as i suppose you do.

/Bingo

johnpoz

@pbnet said in Slow speed between VLANs:

Any idea what can I do to improve inter-VLAN bandwidth ?

One thing would be use different physical interfaces for the uplinks of these vlans. When you share the same physical interface for vlans that are talking to each other, you are hairpinning the traffic over the same physical interface it would be expected to not see full wire speed.

For optimal performance of intervlan traffic it is best to put these vlans that will be talking a lot between them on different physical interfaces. But seems your limited to the 1 interface via sfp+ so your kind of hindered in doing that.

Now that being said I would hope you would see more than what your seeing.. 3Gbps does seem low.. I would expect atleast 1/2 of physical interface speeds or atleast really freaking close or even above etc.. But the 7.x and 3.x something via the hairpin isn't all that out of wack depending..

71% cpu does seem a bit high as well for a 7100 (which is a beast).. Are you doing anything that could hinder the speed - say IPS or something? Or ntopng?

pbnet

@johnpoz OK, so I opened a support case with Netgate. They asked me to remove the following packages: snort, darkstat, ntopng, bandwidthd, haproxy, squid

After removing them and doing a new test, I got: 4.18Gbps using IPERF3 and still a CPU load of 70%.
To be honest it's really far that what they advertise that the XG7100-1U can do.
I guess I also have to ask Netgate if adding a 4-Port SFP+ NIC will void my warranty.
I already purchased the device last year with an additional 4Port- 1Gbps NIC, so the raiser card should be already in there.

Thanks.

johnpoz

@pbnet said in Slow speed between VLANs:

To be honest it's really far that what they advertise that the XG7100-1U can do.

Where did they say you would see wirespeed via a hairpin? I don't see any benchmarks for that.

Now if your talking the 9.85 in this benchmark

IPERF3 Traffic: 9.85 Gbps

If you have 2 different sfp+ at 10ge, and route between them what do you see.. Seems to me your doing vlans on same physical interface. Which is hairpin, and yes this would be lower.

From the iperf 7.6Gbps test you showed seems to be between 2 devices not even going through pfsense at all? So the rest of your network and test devices can not achieve wirespeed?

For a fair test I would think you would have to be using 2 sfp+ connections at 10ge and routing/firewalling between those 2 interface.. If you can only achieve 4 some gbps then I would be disappointed as well.

Can you run a test with that scenario?

pbnet

@johnpoz The XG7100-1U I have only has 2 SFP+ - one is for WAN and one is for LAN.
The tests between 2 machines on the same VLAN (without going though PFSense) is about 7 to 9Gbps.
So, sadly, I cannot do more tests since I don't have more SFP+ ports on the XG7100-1U.
And the solution offered by netgate to disable all packet filtering is not something I'm comfortable of doing.
I get the feeling they are not really trying to help, but rather finding all sorts of workardounds.
They also stressed that the performance tests are based on the maximum memory configuration for the device: I have 24GB or RAM in the XG7100-1U out of which only 7% is used, yet they don't want to explain the high CPU usage.
Let's see what will they come with next.

pbnet

OK. Final statement from Netgate: 4.18Gbps is the max I can get on the device based on this article: https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/switch-overview.html
Also based on this article: https://info.netgate.com/hubfs/website-assets/netgate-hardware-comparison-doc.pdf they mentioned that the throughput measurements are based upon maximum bidirectional traffic across all available ports. As all tests were run by maximizing throughput across available ports on the base model physical ports (XG-7100 with 10 ports). I have the max model https://shop.netgate.com/products/7100-base-pfsense, but it doesn't seem to count.
Really disappointed by the product :(

johnpoz

@pbnet said in Slow speed between VLANs:

has 2 SFP+ - one is for WAN and one is for LAN.

There you go - you clearly have 2 that you could "test" with..

pbnet

@johnpoz True.. Just have to do it in week-ends, since I need to move the WAN to another interface.

johnpoz

@pbnet the switch would have a limit routing between vlans because it has a 5gbps uplink lag.. So lets see what you get for test when using 2 different 10ge interface that is not hairpin and not routed through the 5 gbps uplink lagg.

That doc you linked too goes over that..

But that should not be the case when going through 2 different independent interfaces. If that is the case - then yes I feel you would have a valid point that this should be pointed out in the docs that routing between 2 10ge interfaces is not capable of close to wirespeed.

pbnet

OK, so I used IX0 for VLAN10.

alt text

and here are the results:

alt text

and the CPU usage:
alt text

I'll update also the Netgate ticket