Very Poor Performance on VLAN Routing
-
I have been having an issue recently where the speeds across my VLAN's are very bad, as connection issues (not physical, but larges transfers are timing out or being interrupted). I have checked the Traffic Shaper and deleted any profiles on it. The system is a Protectli Vault 4 Port, Firewall Micro Appliance/Mini PC - Intel Quad Core, AES-NI, 8GB RAM, 120GB mSATA SSD running pfSense 2.4.5-RELEASE-p1. I am running VLANs and the layout is as follows:
igb0 - WAN
igb1 - Main, Gaming
igb2 - Management, Cameras, Switch Lab
igb3 - Guest, VoIPSystems specs are (which replaced my old system with lower specs and the same setup):
CPU: Intel Quad Core Celeron J3160, 64 bit, up to 2.2GHz, AES-NI hardware support
RAM: 8GB DDR3L RAM
SSD: 120GB mSATA
PORTS: 4x Intel i210 Gigabit Ethernet ports, 2x USB 3.0, 1x RJ-45 COM, 2x HDMII did have them all on igb1 (which worked in the past without issue), but recently spread them out to see if it would help with the speed (as I have seen people saying VLANs cut the throughput in pfSense). I have also tried a LAG setup with the 3 interfaces (didn't make a difference). All of the interfaces are reporting 1000base-T Full Duplex in the pfSense software and the switch agrees. In terms of the number of VLANS, I have not had an issue with this amount in the past (and even a couple more). It only recently everything started going bad (also before the 2.4.5-RELEASE-p1 as that was added to try and resolve the issue). The system has been rebooted recently (uptime of 3 days since posting this, and this has been going on since mid to late November I would say).
My switches are all Ubiquiti (4 total - 1 24 PoE & 1 16 with broken PoE for back bone with LAG link for 2GB, and 2 small 8 port in remote locations). I was using a used Cisco 2960G switch in place of the 16 with broken PoE, but removed it as I though it might have been failing. When I move the destination into the same network as the source, transfers are fast (what I would expect to see for a transfer on 1 GB Network) and without issue, it only seems to be when I cross VLANs (which is controlled by pfSense).
To give an idea of the speed issue, running a backup from the gaming system (with 1 GB Network Connection) in the Gaming Network, to a NAS in Main Network (the NAS is a Synology with 4x 1 GB bounded Network Adapters), and I am getting about 550 KB/s of transfer (yes, it says KB/s in the software). With my main system (1 GB Network Connection) in the Main Network (same location as the Synology), I am getting anywhere from 14 to 517 MB/s. The backup software is the same between the two systems (same brand and version), and the gaming system is newer hardware then the main system.
I know this is a lot of information to offload, but I am running out of ideas as to what could be causing the issue. Is there anything I can do to help narrow it down, or can provide to help with the troubleshooting?
-
If you have a need (or want) to firewall your VLANS, then certainly go down the troubleshooting route:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/low-throughput.html
https://docs.netgate.com/pfsense/en/latest/hardware/tune.htmlthen I'd start at layer 1 and work your way up the stack... e.g. check cabling, link lights, duplex, interface errors, etc... try different switches, try new PFsense hardware, etc, etc
However, that can take hours upon hours to track down... and you may not find anything glaring.
If performance is your priority, then I would move your VLANs to your switch, so inter-VLAN traffic is handled at line speed by the switch and doesn't have to traverse your firewall.
-
looks like a hardware, cabling or driver issue ....
i doubt you could accidently configure pfsense in a way to get performance that is THAT bad. -
@kdb9000 said in Very Poor Performance on VLAN Routing:
to 517 MB/s
Sorry but that is NOT possible with gig... The max transfer on a 1gig connection is about 113MBps.. Just not possible to see faster than that via 1 gig.. Doesn't matter if your nas has 1 interface or 20..
Where you could have a problem with traffic between main and gaming is that is a hairpin.. Your traffic is flowing over the same physical interface..
Do a transfer from say Main to Management.. Or flip your vlans so that main is on igb1 and gaming is on igb2..
To take disks out of the equation.. Run an iperf test between systems. On gig, you should see in the high 800's to say about mid 900s..
You can run iperf on your nas.. Here is a good place to download http://www.jadahl.com/iperf-arp-scan/
$ iperf3.exe -c 192.168.9.10 Connecting to host 192.168.9.10, port 5201 [ 5] local 192.168.9.100 port 1691 connected to 192.168.9.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 108 MBytes 900 Mbits/sec [ 5] 1.00-2.00 sec 113 MBytes 949 Mbits/sec [ 5] 2.00-3.00 sec 114 MBytes 958 Mbits/sec [ 5] 3.00-4.00 sec 113 MBytes 949 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 949 Mbits/sec [ 5] 5.00-6.00 sec 113 MBytes 949 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 949 Mbits/sec [ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec [ 5] 8.00-9.00 sec 113 MBytes 949 Mbits/sec [ 5] 9.00-10.00 sec 113 MBytes 949 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec sender [ 5] 0.00-10.02 sec 1.10 GBytes 943 Mbits/sec receiverThat is from my PC to my synology nas ds918+ on a 1 gig connection.. Here is it via 2.5gig connection.
$ iperf3.exe -c 192.168.10.10 Connecting to host 192.168.10.10, port 5201 [ 5] local 192.168.10.9 port 1703 connected to 192.168.10.10 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 270 MBytes 2.26 Gbits/sec [ 5] 1.00-2.00 sec 281 MBytes 2.36 Gbits/sec [ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 3.00-4.00 sec 281 MBytes 2.36 Gbits/sec [ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec [ 5] 5.00-6.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.00 sec 279 MBytes 2.35 Gbits/sec [ 5] 8.00-9.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 9.00-10.00 sec 281 MBytes 2.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec sender [ 5] 0.00-10.01 sec 2.73 GBytes 2.34 Gbits/sec receiver iperf Done.Via 2.5 gig connection max is about 280MBps.. for a file transfer..
But again its not possible to see 517MBps over 1 gig connection.. Its just not..
For a 1 gig connection best you could see is in the 113MBps area.. If you do smb3 multichannel and your nas and pc have say 2 gig connection, you can see double that 220ish MBps - I use to use that to my nas before I added the 2.5 gig connection.
-
@heper said in Very Poor Performance on VLAN Routing:
looks like a hardware, cabling or driver issue ....
i doubt you could accidently configure pfsense in a way to get performance that is THAT bad.A test I did before the changes that would show it isn't hardware, cabling, or driver issues (at least on the computers and/or NAS, can't speak for the pfSense box) is adding a VLAN on the Synology that was in the same network as one of the systems that had the slow speed. So at this point it was only working at Layer 2 when it did the backup and was at speeds I was expecting (in the MB range). That was before I changed anything hardware related (like removing the Cisco switch) or changing how the VLANs were setup on the pfSense. So if the transfer was done at Layer 2, everything was good. It was only bad when I got to Layer 3.
@johnpoz said in Very Poor Performance on VLAN Routing:
Sorry but that is NOT possible with gig... The max transfer on a 1gig connection is about 113MBps.. Just not possible to see faster than that via 1 gig.. Doesn't matter if your nas has 1 interface or 20..
The backup software is Veeam EndPoint Agent and I am just going by what it reported at the end of the transfer. It was an Incremental, so only a small amount of data is actually transferred (since it is only the difference from the full). After it runs for a while it will normally settle in the lower MB range I mentioned, but it normally starts off showing the higher number. I don't know how it is doing it's calculation, I am just providing it as information points that I have to work with and did for my test.
I will give the ipref a try, as well as trying the VLAN on a different interface (which never had that problem before).
-
Maybe its doing some serious compression or some math on how much data was backed up total even if just did a diff?
But from a network perspective that just not possible.
-
@johnpoz said in Very Poor Performance on VLAN Routing:
Maybe its doing some serious compression or some math on how much data was backed up total even if just did a diff?
But from a network perspective that just not possible.
Possibly. Like I said, when it starts off it is saying the speed is that, but after it runs for a while (like with a full) it gets down closer to the 14 MB/s.
I tried changing the interface of the VLAN and the system shows it was changed, but the computer can't go anywhere. I checked the Traffic in pfSense and it seems like it is still on igb1 (I moved it to igb3). I had restarted the computer, but that didn't make a difference. I haven't restarted pfSense yet (as I need my Internet to work from home, so I can restart it later today). I am guessing I might have been running into the same issue when I did the LAGG, since igb1 was in the LAGG I didn't notice any issues.
-
@kdb9000 said in Very Poor Performance on VLAN Routing:
I have also tried a LAG setup with the 3 interfaces (didn't make a difference).
Just out of curiosity, when you setup the lagg, did you also configure the corresponding port-channel (LACP) on the switch?
-
While hairpin (intervlan traffic on the same physical) interface should not be such a drastic hit in performance. It is always best to for vlans that want max possible performance between them not to hairpen. and possible if other vlans on these physical interface for them to be lower traffic sort of vlans.
Its possible maybe whatever is showing you your speed is just horrible at doing math ;)
I would suggest you do a iperf test and then just a simple file copy test... Timed - for example robocopy can show you exactly how long it took and what the speed was..
So for example here is test via 2.5gig connection
login-to-viewAnd then here is copy via 1gig both just copy paste in windows and then via robocopy..
You can always expect a bit of fluctuation in the math, etc. But 517MB is just not possible at all - so so something is way off there. If its way off on the high end, who says it not way off on the low end as well? Also if you doing lots of little file copies that can for sure slow the transfer.
So to see if your network is performing as it should... I would suggest the iperf test to take disks read/write or io issues out of it. And then a file copy of a single large file.. from known good disks - but any modern disk should be able to handle gig speeds of 110ish MBps.
2GB / 18 seconds works out to 111MBps which is pretty much maxed out for a 1 gig connection.
-
Just an idea
Could it be an "offloading" issue ?
Here is what i use for my Intel 210 nic's
System -> Advanced -> NetworkingWhen i installed pfSense
My Iperf tests showed around 980Mb/s TCP - To my Linux ServerDo you see any error/rejected frames on the switch ?
/Bingo
-
@bingo600 said in Very Poor Performance on VLAN Routing:
My Iperf tests showed around 980Mb/s TCP
Some funky math there as well since that is not possible to be honest.. Unless you were using jumbo frames? Not counting for overhead your prob at the 118MBps max - I think your going to really be around 940ish max moving any sort of data..
I quite often show 949 in my testing.. which I think is rounding errors or something to be honest.. Most calculations I do is like 940..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
@marvosa said in Very Poor Performance on VLAN Routing:
@kdb9000 said in Very Poor Performance on VLAN Routing:
I have also tried a LAG setup with the 3 interfaces (didn't make a difference).
Just out of curiosity, when you setup the lagg, did you also configure the corresponding port-channel (LACP) on the switch?
As far as I can tell with Ubiquiti. The ports were setup as an Aggregate across the 3 ports on the Switch.
@johnpoz
I tested transferring a VM Data file (with several files over 1 GB in size) from one system to the other (standard copy using Windows Explorer) across the VLAN and I get about 2 MB/s at max (it is bouncing between 2 MB/s and 1.8 MB/s, sometimes lower). If I transfer the same files (not at the same time) to the NAS (which is on the same network), I get anywhere from 80 MB/s to 60 MB/s (1 GB on my computer, 1 GB from the switch to the back bone switchs, 2x 1 GB from the 24 PoE to the 16 broken PoE, and then 4x 1 GB to the NAS). -
Your never going to be able to leverage those lagg connection from 1 device to another device.. Unless you were doing smb3 multichannel.
What does your iperf test show you? 80MBps is LOW for 1 gig.. You should be seeing in the low 100MBps if your network is working correctly..
For testing purposes I would really just turn off any lacp or lagg you have.. You should be able to saturate your 1 gig in the 940mbps range using iperf..
-
It could have been w. jumboframes , as i ran Jumbo for a short time.
Then i decided i didn't need Jumbo on my home network, due to many of my "home appliances" not supporting it. And disabled it site wide.I just reran an iperf to show the OP , that there isn't much difference between pure L2 , or L3 with pfSense as Vlan router.
Switches HP1820 Cat5e Linked
Linux server Deb10 - Realtec NIC - (iperf -s) : 192.168.x.y Linux WS Linux Mint - Intel 82579LM - (iperf -c) : 192.168.x.x or 10.x.x.x Client & Server On same Vlan (Pure L2) # iperf -t60 -i10 -c 192.168.x.y ------------------------------------------------------------ Client connecting to frodo, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.x.x port 58296 connected with 192.168.x.y port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.08 GBytes 931 Mbits/sec [ 3] 10.0-20.0 sec 1.08 GBytes 931 Mbits/sec [ 3] 20.0-30.0 sec 1.09 GBytes 933 Mbits/sec [ 3] 30.0-40.0 sec 1.09 GBytes 934 Mbits/sec [ 3] 40.0-50.0 sec 1.08 GBytes 929 Mbits/sec [ 3] 50.0-60.0 sec 1.08 GBytes 929 Mbits/sec [ 3] 0.0-60.0 sec 6.50 GBytes 931 Mbits/sec # Client & Server on different Vlans # iperf -t60 -i10 -c 192.168.x.y ------------------------------------------------------------ Client connecting to frodo, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.x.x.x port 33834 connected with 192.168.x.y port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.08 GBytes 930 Mbits/sec [ 3] 10.0-20.0 sec 1.08 GBytes 928 Mbits/sec [ 3] 20.0-30.0 sec 1.08 GBytes 927 Mbits/sec [ 3] 30.0-40.0 sec 1.08 GBytes 926 Mbits/sec [ 3] 40.0-50.0 sec 1.08 GBytes 929 Mbits/sec [ 3] 50.0-60.0 sec 1.08 GBytes 926 Mbits/sec [ 3] 0.0-60.0 sec 6.48 GBytes 927 Mbits/sec #No pfSense IGBx ethernet tuning at all.
Edit: pfSense CPU load during xfer 23..29%
/Bingo
-
@kdb9000 said in Very Poor Performance on VLAN Routing:
As far as I can tell with Ubiquiti. The ports were setup as an Aggregate across the 3 ports on the Switch.
Which mode was configured on the PFsense side? Which mode was configured on the Ubiquity switch?
-
My settings are the same as what you have in the image (disabled for the two offloading, checksum is the only offloader enabled).
I haven't gotten the iperf working on the Synology (not sure which one I need to install). I did use the Windows version between two computers on different VLANS.
Main > Gaming
Main VLAN>iperf3.exe -c 192.168.13.235 Connecting to host 192.168.13.235, port 5201 [ 4] local 192.168.10.60 port 57740 connected to 192.168.13.235 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 4.50 MBytes 37.7 Mbits/sec [ 4] 1.00-2.00 sec 38.0 MBytes 319 Mbits/sec [ 4] 2.00-3.00 sec 56.1 MBytes 470 Mbits/sec [ 4] 3.00-4.00 sec 56.0 MBytes 470 Mbits/sec [ 4] 4.00-5.00 sec 56.9 MBytes 477 Mbits/sec [ 4] 5.00-6.00 sec 55.1 MBytes 462 Mbits/sec [ 4] 6.00-7.00 sec 57.8 MBytes 484 Mbits/sec [ 4] 7.00-8.00 sec 58.4 MBytes 490 Mbits/sec [ 4] 8.00-9.00 sec 56.0 MBytes 470 Mbits/sec [ 4] 9.00-10.00 sec 55.4 MBytes 464 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 494 MBytes 414 Mbits/sec sender [ 4] 0.00-10.00 sec 494 MBytes 414 Mbits/sec receiver iperf Done. Gaming VLAN>iperf3.exe -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.10.60, port 57739 [ 5] local 192.168.13.235 port 5201 connected to 192.168.10.60 port 57740 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.01 sec 4.00 MBytes 33.2 Mbits/sec [ 5] 1.01-2.00 sec 32.3 MBytes 274 Mbits/sec [ 5] 2.00-3.00 sec 56.1 MBytes 471 Mbits/sec [ 5] 3.00-4.00 sec 56.0 MBytes 469 Mbits/sec [ 5] 4.00-5.00 sec 56.8 MBytes 476 Mbits/sec [ 5] 5.00-6.00 sec 55.1 MBytes 462 Mbits/sec [ 5] 6.00-7.00 sec 57.7 MBytes 484 Mbits/sec [ 5] 7.00-8.00 sec 58.5 MBytes 491 Mbits/sec [ 5] 8.00-9.00 sec 56.2 MBytes 471 Mbits/sec [ 5] 9.00-10.00 sec 55.4 MBytes 464 Mbits/sec [ 5] 10.00-10.11 sec 6.01 MBytes 475 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.11 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.11 sec 494 MBytes 410 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------And then from Gaming > Main
Gaming VLAN>iperf3.exe -c 192.168.10.60 Connecting to host 192.168.10.60, port 5201 [ 4] local 192.168.13.235 port 64557 connected to 192.168.10.60 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 34.2 MBytes 287 Mbits/sec [ 4] 1.00-2.01 sec 33.1 MBytes 277 Mbits/sec [ 4] 2.01-3.00 sec 33.6 MBytes 284 Mbits/sec [ 4] 3.00-4.00 sec 32.6 MBytes 273 Mbits/sec [ 4] 4.00-5.00 sec 29.5 MBytes 247 Mbits/sec [ 4] 5.00-6.01 sec 31.6 MBytes 265 Mbits/sec [ 4] 6.01-7.00 sec 33.0 MBytes 278 Mbits/sec [ 4] 7.00-8.00 sec 31.6 MBytes 265 Mbits/sec [ 4] 8.00-9.00 sec 33.0 MBytes 277 Mbits/sec [ 4] 9.00-10.00 sec 32.1 MBytes 269 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 324 MBytes 272 Mbits/sec sender [ 4] 0.00-10.00 sec 324 MBytes 272 Mbits/sec receiver iperf Done. Main VLAN>iperf3.exe -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.13.235, port 64556 [ 5] local 192.168.10.60 port 5201 connected to 192.168.13.235 port 64557 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 33.3 MBytes 280 Mbits/sec [ 5] 1.00-2.00 sec 32.5 MBytes 273 Mbits/sec [ 5] 2.00-3.00 sec 33.6 MBytes 282 Mbits/sec [ 5] 3.00-4.00 sec 32.6 MBytes 273 Mbits/sec [ 5] 4.00-5.00 sec 29.3 MBytes 246 Mbits/sec [ 5] 5.00-6.00 sec 31.7 MBytes 266 Mbits/sec [ 5] 6.00-7.00 sec 33.1 MBytes 278 Mbits/sec [ 5] 7.00-8.00 sec 31.5 MBytes 264 Mbits/sec [ 5] 8.00-9.00 sec 33.1 MBytes 277 Mbits/sec [ 5] 9.00-10.00 sec 32.1 MBytes 269 Mbits/sec [ 5] 10.00-10.04 sec 1.58 MBytes 304 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.04 sec 324 MBytes 271 Mbits/sec receiver ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------Nothing in the network as changed since I posted the initial thread, and the SMB traffic between the VLANs is still very poor compared to what the iperf test shows (which I have seen some people the iperf test isn't a very good test).
-
Iperf on pfsense is not a good test no.
But client to client through pfsense is good test.
So what do you see from client to client on the same network? Because those speeds are terrible for gig.. You should be seeing high 800's to low 900s for sure..
What specific model of nas do you have? And I can help you figure out which synology iperf you want. For example on my ds918 its the apollolake..
-
@marvosa said in Very Poor Performance on VLAN Routing:
@kdb9000 said in Very Poor Performance on VLAN Routing:
As far as I can tell with Ubiquiti. The ports were setup as an Aggregate across the 3 ports on the Switch.
Which mode was configured on the PFsense side? Which mode was configured on the Ubiquity switch?
Aggregate is what is it called on the Ubiquiti side, LAGG is what it is called on the pfSense side. On the pfSense side, the protocol was LACP. Ubiquiti doesn't have any other options to change for the Aggregate (aside from setting Link Speed and how many ports are in the Aggregate). At this time, I am not running LAGG on pfSense. Instead the 3 connections are individual with different VLAN's attached to them.
-
What does an L2 iperf report ?
I mean server & client on the same subnetAre you running hairpin / "On a Stick" when doing the inter Vlan xfers ?
I have divided my Vlans across two pfSense interfaces.
And made sure my Server and (cabled) Client vlans are on separate IGBx interfaces./Bingo
-
This post is deleted!
