VLAN to VLAN slower than expected
-
@Jarhead yes I'm testing over 1g links, and I would expect to get 1g. However, I'm only getting half that.
-
@mouseskowitz The description suggests you should get about 1Gb. Are there any packages running that might slow it down? What is the CPU usage when you are testing?
-
@mouseskowitz said in VLAN to VLAN slower than expected:
@Jarhead yes I'm testing over 1g links, and I would expect to get 1g. However, I'm only getting half that.
Gotcha, this line:
Links on Switch C and pfSense say they are connected at 10 Gbe, but the traffic flow doesn't seem to back that up. Is there any possibility that the initial 1 Gbe bandwidth is being respected across all the connections? Or is something else possibly the issue?
Made it seem like you expected 10g because obviously the 1g would be respected across all links.
-
@mouseskowitz So we fully understand your setup... Could you draw out your connections?
When you say 2ge aggregate - you mean you have 2x1ge lacp connection?
So both vlans hairpin over the 10ge link to pfsense, or is there an additional lagg between switch C and pfsense that carries vlan B while the 10ge carries vlan A?
When you say aggregate, I take it your talking 2x1ge connection using lacp? But I am not really clear how many you have, and specific where. Your not meaning they are 2.5ge switch ports? I take it with the term aggregate that you have some sort of port channel, or lagg or lacp setup.. And these switch on both sides support whatever method your using. And your not setup in any sort of load sharing setup across the multiple links?
Does switch B have a 10ge link to switch C, or is this also a lagg? Does switch C only have the 1 10ge to pfsense where your vlans hairpin, or does it have also a 2x1 that carries the vlan B and the 10ge carries vlan A?
edit: also what exactly is pfsense running on? Its not VM is it?
-
When testing with iperf within the same VLAN, you could expect something between 930-970 Mbps. However you are doing it between 2 VLANs so it is going to be slower because routing is involved. How much slower? Well, it depends on your firewall rules and the processing power of your server. The chain of devices is also quite long and the same data flows through Switch C twice. I would run some tests on shorter device chains to understand that better.
-
@johnpoz your diagram was very close. pfSense is running on a dedicated server with and Intel Xeon D-1521 with 8GB of RAM.
-
I've been playing around with iperf3 all afternoon and have found some very strange behavior within the Unifi switches. If I have two servers connected to the 10G switch using the native VLAN 1 network, I can only get 1G speeds. If I simply switch both of them to a different VLAN, I can get 10G speeds. So, I'm thinking it might be an issue in the switch not in pfSense. We'll see what I can find out over in the Unifi forums.
-
@mouseskowitz yeah with even hairpinning on that 10ge connection I would expect you could see 900ish for sure..
Your saying if you have 2 devices connect at 10ge to the switch in vlan 1 (native default) they can't get 10ge? pfsense wouldn't even be part of that conversation..
Do you maybe have rate limiting set on the switch?
-
@johnpoz From what I can find my Unifi 16 XG is not supposed to be able to rate limit per VLAN, but that's what it feels like it's doing. I can get 10G speeds on VLAN10 and VLAN40, but only 1G on VLAN1 with the only change being the IP range/VLAN the hardware is set to. I might try plugging in a second 10G connection between the switch and pfSense and just run VLAN10 and VLAN40 on it. But that'll have to wait for another day.
-
@mouseskowitz are you running like their IDS on that vlan? Or have captive portal or guest mode on or something like that.. I have limited experience with their switches.. Not really a fan to be honest.
I use to have a USGP3 for a short time while I waited for my 4860, and if you had IDS turned on it could only do like 120Mbps, but with it off it handled my 500mbps internet connection fine.
So I wouldn't be surprised if they has some setting that dropped your performance into the dirt..
-
@johnpoz to use their IDS you need to be using one of their gateways. I think it's the same way with the captive portal. You can rate limit on the wifi, but that doesn't apply to wired connections.
I moved VLAN1 over to a dedicated 1G connection. I can get 1G speeds going from VLAN10 to VLAN1, but I only get half the speed going the other way. I can still only get 1/2 1G speeds going from VLAN10 to VLAN40.
-
It seems like this issue was hardware related. I swapped out the 10G card and the strange behavior has gone away.
