Very Poor Performance on VLAN Routing


  • I have been having an issue recently where the speeds across my VLAN's are very bad, as connection issues (not physical, but larges transfers are timing out or being interrupted). I have checked the Traffic Shaper and deleted any profiles on it. The system is a Protectli Vault 4 Port, Firewall Micro Appliance/Mini PC - Intel Quad Core, AES-NI, 8GB RAM, 120GB mSATA SSD running pfSense 2.4.5-RELEASE-p1. I am running VLANs and the layout is as follows:

    igb0 - WAN
    igb1 - Main, Gaming
    igb2 - Management, Cameras, Switch Lab
    igb3 - Guest, VoIP

    Systems specs are (which replaced my old system with lower specs and the same setup):
    CPU: Intel Quad Core Celeron J3160, 64 bit, up to 2.2GHz, AES-NI hardware support
    RAM: 8GB DDR3L RAM
    SSD: 120GB mSATA
    PORTS: 4x Intel i210 Gigabit Ethernet ports, 2x USB 3.0, 1x RJ-45 COM, 2x HDMI

    I did have them all on igb1 (which worked in the past without issue), but recently spread them out to see if it would help with the speed (as I have seen people saying VLANs cut the throughput in pfSense). I have also tried a LAG setup with the 3 interfaces (didn't make a difference). All of the interfaces are reporting 1000base-T Full Duplex in the pfSense software and the switch agrees. In terms of the number of VLANS, I have not had an issue with this amount in the past (and even a couple more). It only recently everything started going bad (also before the 2.4.5-RELEASE-p1 as that was added to try and resolve the issue). The system has been rebooted recently (uptime of 3 days since posting this, and this has been going on since mid to late November I would say).

    My switches are all Ubiquiti (4 total - 1 24 PoE & 1 16 with broken PoE for back bone with LAG link for 2GB, and 2 small 8 port in remote locations). I was using a used Cisco 2960G switch in place of the 16 with broken PoE, but removed it as I though it might have been failing. When I move the destination into the same network as the source, transfers are fast (what I would expect to see for a transfer on 1 GB Network) and without issue, it only seems to be when I cross VLANs (which is controlled by pfSense).

    To give an idea of the speed issue, running a backup from the gaming system (with 1 GB Network Connection) in the Gaming Network, to a NAS in Main Network (the NAS is a Synology with 4x 1 GB bounded Network Adapters), and I am getting about 550 KB/s of transfer (yes, it says KB/s in the software). With my main system (1 GB Network Connection) in the Main Network (same location as the Synology), I am getting anywhere from 14 to 517 MB/s. The backup software is the same between the two systems (same brand and version), and the gaming system is newer hardware then the main system.

    I know this is a lot of information to offload, but I am running out of ideas as to what could be causing the issue. Is there anything I can do to help narrow it down, or can provide to help with the troubleshooting?


  • If you have a need (or want) to firewall your VLANS, then certainly go down the troubleshooting route:

    https://docs.netgate.com/pfsense/en/latest/troubleshooting/low-throughput.html
    https://docs.netgate.com/pfsense/en/latest/hardware/tune.html

    then I'd start at layer 1 and work your way up the stack... e.g. check cabling, link lights, duplex, interface errors, etc... try different switches, try new PFsense hardware, etc, etc

    However, that can take hours upon hours to track down... and you may not find anything glaring.

    If performance is your priority, then I would move your VLANs to your switch, so inter-VLAN traffic is handled at line speed by the switch and doesn't have to traverse your firewall.


  • looks like a hardware, cabling or driver issue ....
    i doubt you could accidently configure pfsense in a way to get performance that is THAT bad.

  • LAYER 8 Global Moderator

    @kdb9000 said in Very Poor Performance on VLAN Routing:

    to 517 MB/s

    Sorry but that is NOT possible with gig... The max transfer on a 1gig connection is about 113MBps.. Just not possible to see faster than that via 1 gig.. Doesn't matter if your nas has 1 interface or 20..

    Where you could have a problem with traffic between main and gaming is that is a hairpin.. Your traffic is flowing over the same physical interface..

    Do a transfer from say Main to Management.. Or flip your vlans so that main is on igb1 and gaming is on igb2..

    To take disks out of the equation.. Run an iperf test between systems. On gig, you should see in the high 800's to say about mid 900s..

    You can run iperf on your nas.. Here is a good place to download http://www.jadahl.com/iperf-arp-scan/

    $ iperf3.exe -c 192.168.9.10
    Connecting to host 192.168.9.10, port 5201
    [  5] local 192.168.9.100 port 1691 connected to 192.168.9.10 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec   108 MBytes   900 Mbits/sec
    [  5]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
    [  5]   2.00-3.00   sec   114 MBytes   958 Mbits/sec
    [  5]   3.00-4.00   sec   113 MBytes   949 Mbits/sec
    [  5]   4.00-5.00   sec   113 MBytes   949 Mbits/sec
    [  5]   5.00-6.00   sec   113 MBytes   949 Mbits/sec
    [  5]   6.00-7.00   sec   113 MBytes   949 Mbits/sec
    [  5]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
    [  5]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
    [  5]   9.00-10.00  sec   113 MBytes   949 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec  1.10 GBytes   945 Mbits/sec                  sender
    [  5]   0.00-10.02  sec  1.10 GBytes   943 Mbits/sec                  receiver
    

    That is from my PC to my synology nas ds918+ on a 1 gig connection.. Here is it via 2.5gig connection.

    $ iperf3.exe -c 192.168.10.10                                                          
    Connecting to host 192.168.10.10, port 5201                                            
    [  5] local 192.168.10.9 port 1703 connected to 192.168.10.10 port 5201                
    [ ID] Interval           Transfer     Bitrate                                          
    [  5]   0.00-1.00   sec   270 MBytes  2.26 Gbits/sec                                   
    [  5]   1.00-2.00   sec   281 MBytes  2.36 Gbits/sec                                   
    [  5]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec                                   
    [  5]   3.00-4.00   sec   281 MBytes  2.36 Gbits/sec                                   
    [  5]   4.00-5.00   sec   281 MBytes  2.36 Gbits/sec                                   
    [  5]   5.00-6.00   sec   281 MBytes  2.35 Gbits/sec                                   
    [  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec                                   
    [  5]   7.00-8.00   sec   279 MBytes  2.35 Gbits/sec                                   
    [  5]   8.00-9.00   sec   281 MBytes  2.35 Gbits/sec                                   
    [  5]   9.00-10.00  sec   281 MBytes  2.36 Gbits/sec                                   
    - - - - - - - - - - - - - - - - - - - - - - - - -                                      
    [ ID] Interval           Transfer     Bitrate                                          
    [  5]   0.00-10.00  sec  2.73 GBytes  2.34 Gbits/sec                  sender           
    [  5]   0.00-10.01  sec  2.73 GBytes  2.34 Gbits/sec                  receiver         
                                                                                           
    iperf Done.                                                                            
    

    Via 2.5 gig connection max is about 280MBps.. for a file transfer..

    But again its not possible to see 517MBps over 1 gig connection.. Its just not..

    For a 1 gig connection best you could see is in the 113MBps area.. If you do smb3 multichannel and your nas and pc have say 2 gig connection, you can see double that 220ish MBps - I use to use that to my nas before I added the 2.5 gig connection.


  • @heper said in Very Poor Performance on VLAN Routing:

    looks like a hardware, cabling or driver issue ....
    i doubt you could accidently configure pfsense in a way to get performance that is THAT bad.

    A test I did before the changes that would show it isn't hardware, cabling, or driver issues (at least on the computers and/or NAS, can't speak for the pfSense box) is adding a VLAN on the Synology that was in the same network as one of the systems that had the slow speed. So at this point it was only working at Layer 2 when it did the backup and was at speeds I was expecting (in the MB range). That was before I changed anything hardware related (like removing the Cisco switch) or changing how the VLANs were setup on the pfSense. So if the transfer was done at Layer 2, everything was good. It was only bad when I got to Layer 3.

    @johnpoz said in Very Poor Performance on VLAN Routing:

    Sorry but that is NOT possible with gig... The max transfer on a 1gig connection is about 113MBps.. Just not possible to see faster than that via 1 gig.. Doesn't matter if your nas has 1 interface or 20..

    The backup software is Veeam EndPoint Agent and I am just going by what it reported at the end of the transfer. It was an Incremental, so only a small amount of data is actually transferred (since it is only the difference from the full). After it runs for a while it will normally settle in the lower MB range I mentioned, but it normally starts off showing the higher number. I don't know how it is doing it's calculation, I am just providing it as information points that I have to work with and did for my test.

    I will give the ipref a try, as well as trying the VLAN on a different interface (which never had that problem before).

  • LAYER 8 Global Moderator

    Maybe its doing some serious compression or some math on how much data was backed up total even if just did a diff?

    But from a network perspective that just not possible.


  • @johnpoz said in Very Poor Performance on VLAN Routing:

    Maybe its doing some serious compression or some math on how much data was backed up total even if just did a diff?

    But from a network perspective that just not possible.

    Possibly. Like I said, when it starts off it is saying the speed is that, but after it runs for a while (like with a full) it gets down closer to the 14 MB/s.

    I tried changing the interface of the VLAN and the system shows it was changed, but the computer can't go anywhere. I checked the Traffic in pfSense and it seems like it is still on igb1 (I moved it to igb3). I had restarted the computer, but that didn't make a difference. I haven't restarted pfSense yet (as I need my Internet to work from home, so I can restart it later today). I am guessing I might have been running into the same issue when I did the LAGG, since igb1 was in the LAGG I didn't notice any issues.


  • @kdb9000 said in Very Poor Performance on VLAN Routing:

    I have also tried a LAG setup with the 3 interfaces (didn't make a difference).

    Just out of curiosity, when you setup the lagg, did you also configure the corresponding port-channel (LACP) on the switch?

  • LAYER 8 Global Moderator

    While hairpin (intervlan traffic on the same physical) interface should not be such a drastic hit in performance. It is always best to for vlans that want max possible performance between them not to hairpen. and possible if other vlans on these physical interface for them to be lower traffic sort of vlans.

    Its possible maybe whatever is showing you your speed is just horrible at doing math ;)

    I would suggest you do a iperf test and then just a simple file copy test... Timed - for example robocopy can show you exactly how long it took and what the speed was..

    So for example here is test via 2.5gig connection
    filecopy.png

    And then here is copy via 1gig both just copy paste in windows and then via robocopy..

    robocopy.png

    You can always expect a bit of fluctuation in the math, etc. But 517MB is just not possible at all - so so something is way off there. If its way off on the high end, who says it not way off on the low end as well? Also if you doing lots of little file copies that can for sure slow the transfer.

    So to see if your network is performing as it should... I would suggest the iperf test to take disks read/write or io issues out of it. And then a file copy of a single large file.. from known good disks - but any modern disk should be able to handle gig speeds of 110ish MBps.

    2GB / 18 seconds works out to 111MBps which is pretty much maxed out for a 1 gig connection.


  • Just an idea

    Could it be an "offloading" issue ?

    Here is what i use for my Intel 210 nic's

    System -> Advanced -> Networking
    

    a10bb150-511e-4042-a854-9baac9432c28-image.png

    When i installed pfSense
    My Iperf tests showed around 980Mb/s TCP - To my Linux Server

    Do you see any error/rejected frames on the switch ?

    /Bingo

  • LAYER 8 Global Moderator

    @bingo600 said in Very Poor Performance on VLAN Routing:

    My Iperf tests showed around 980Mb/s TCP

    Some funky math there as well since that is not possible to be honest.. Unless you were using jumbo frames? Not counting for overhead your prob at the 118MBps max - I think your going to really be around 940ish max moving any sort of data..

    I quite often show 949 in my testing.. which I think is rounding errors or something to be honest.. Most calculations I do is like 940..


  • @marvosa said in Very Poor Performance on VLAN Routing:

    @kdb9000 said in Very Poor Performance on VLAN Routing:

    I have also tried a LAG setup with the 3 interfaces (didn't make a difference).

    Just out of curiosity, when you setup the lagg, did you also configure the corresponding port-channel (LACP) on the switch?

    As far as I can tell with Ubiquiti. The ports were setup as an Aggregate across the 3 ports on the Switch.

    @johnpoz
    I tested transferring a VM Data file (with several files over 1 GB in size) from one system to the other (standard copy using Windows Explorer) across the VLAN and I get about 2 MB/s at max (it is bouncing between 2 MB/s and 1.8 MB/s, sometimes lower). If I transfer the same files (not at the same time) to the NAS (which is on the same network), I get anywhere from 80 MB/s to 60 MB/s (1 GB on my computer, 1 GB from the switch to the back bone switchs, 2x 1 GB from the 24 PoE to the 16 broken PoE, and then 4x 1 GB to the NAS).

  • LAYER 8 Global Moderator

    Your never going to be able to leverage those lagg connection from 1 device to another device.. Unless you were doing smb3 multichannel.

    What does your iperf test show you? 80MBps is LOW for 1 gig.. You should be seeing in the low 100MBps if your network is working correctly..

    For testing purposes I would really just turn off any lacp or lagg you have.. You should be able to saturate your 1 gig in the 940mbps range using iperf..


  • @johnpoz

    It could have been w. jumboframes , as i ran Jumbo for a short time.
    Then i decided i didn't need Jumbo on my home network, due to many of my "home appliances" not supporting it. And disabled it site wide.

    I just reran an iperf to show the OP , that there isn't much difference between pure L2 , or L3 with pfSense as Vlan router.

    Switches HP1820 Cat5e Linked

    Linux server Deb10      - Realtec NIC    - (iperf -s) : 192.168.x.y
    Linux WS     Linux Mint - Intel 82579LM  - (iperf -c) : 192.168.x.x  or 10.x.x.x
    
    
    Client & Server On same Vlan (Pure L2)
    
    # iperf  -t60 -i10 -c 192.168.x.y
    ------------------------------------------------------------
    Client connecting to frodo, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.x.x port 58296 connected with 192.168.x.y port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  1.08 GBytes   931 Mbits/sec
    [  3] 10.0-20.0 sec  1.08 GBytes   931 Mbits/sec
    [  3] 20.0-30.0 sec  1.09 GBytes   933 Mbits/sec
    [  3] 30.0-40.0 sec  1.09 GBytes   934 Mbits/sec
    [  3] 40.0-50.0 sec  1.08 GBytes   929 Mbits/sec
    [  3] 50.0-60.0 sec  1.08 GBytes   929 Mbits/sec
    [  3]  0.0-60.0 sec  6.50 GBytes   931 Mbits/sec
    #
    
    
    Client & Server on different Vlans
    
    # iperf  -t60 -i10 -c 192.168.x.y
    ------------------------------------------------------------
    Client connecting to frodo, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 10.x.x.x port 33834 connected with 192.168.x.y port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  1.08 GBytes   930 Mbits/sec
    [  3] 10.0-20.0 sec  1.08 GBytes   928 Mbits/sec
    [  3] 20.0-30.0 sec  1.08 GBytes   927 Mbits/sec
    [  3] 30.0-40.0 sec  1.08 GBytes   926 Mbits/sec
    [  3] 40.0-50.0 sec  1.08 GBytes   929 Mbits/sec
    [  3] 50.0-60.0 sec  1.08 GBytes   926 Mbits/sec
    [  3]  0.0-60.0 sec  6.48 GBytes   927 Mbits/sec
    #
    

    No pfSense IGBx ethernet tuning at all.

    Edit: pfSense CPU load during xfer 23..29%

    /Bingo


  • @kdb9000 said in Very Poor Performance on VLAN Routing:

    As far as I can tell with Ubiquiti. The ports were setup as an Aggregate across the 3 ports on the Switch.

    Which mode was configured on the PFsense side? Which mode was configured on the Ubiquity switch?


  • @bingo600

    My settings are the same as what you have in the image (disabled for the two offloading, checksum is the only offloader enabled).

    @johnpoz

    I haven't gotten the iperf working on the Synology (not sure which one I need to install). I did use the Windows version between two computers on different VLANS.

    Main > Gaming

    Main VLAN>iperf3.exe -c 192.168.13.235
    Connecting to host 192.168.13.235, port 5201
    [  4] local 192.168.10.60 port 57740 connected to 192.168.13.235 port 5201
    [ ID] Interval           Transfer     Bandwidth
    [  4]   0.00-1.00   sec  4.50 MBytes  37.7 Mbits/sec
    [  4]   1.00-2.00   sec  38.0 MBytes   319 Mbits/sec
    [  4]   2.00-3.00   sec  56.1 MBytes   470 Mbits/sec
    [  4]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec
    [  4]   4.00-5.00   sec  56.9 MBytes   477 Mbits/sec
    [  4]   5.00-6.00   sec  55.1 MBytes   462 Mbits/sec
    [  4]   6.00-7.00   sec  57.8 MBytes   484 Mbits/sec
    [  4]   7.00-8.00   sec  58.4 MBytes   490 Mbits/sec
    [  4]   8.00-9.00   sec  56.0 MBytes   470 Mbits/sec
    [  4]   9.00-10.00  sec  55.4 MBytes   464 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth
    [  4]   0.00-10.00  sec   494 MBytes   414 Mbits/sec                  sender
    [  4]   0.00-10.00  sec   494 MBytes   414 Mbits/sec                  receiver
    
    iperf Done.
    
    Gaming VLAN>iperf3.exe -s
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    Accepted connection from 192.168.10.60, port 57739
    [  5] local 192.168.13.235 port 5201 connected to 192.168.10.60 port 57740
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-1.01   sec  4.00 MBytes  33.2 Mbits/sec
    [  5]   1.01-2.00   sec  32.3 MBytes   274 Mbits/sec
    [  5]   2.00-3.00   sec  56.1 MBytes   471 Mbits/sec
    [  5]   3.00-4.00   sec  56.0 MBytes   469 Mbits/sec
    [  5]   4.00-5.00   sec  56.8 MBytes   476 Mbits/sec
    [  5]   5.00-6.00   sec  55.1 MBytes   462 Mbits/sec
    [  5]   6.00-7.00   sec  57.7 MBytes   484 Mbits/sec
    [  5]   7.00-8.00   sec  58.5 MBytes   491 Mbits/sec
    [  5]   8.00-9.00   sec  56.2 MBytes   471 Mbits/sec
    [  5]   9.00-10.00  sec  55.4 MBytes   464 Mbits/sec
    [  5]  10.00-10.11  sec  6.01 MBytes   475 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-10.11  sec  0.00 Bytes  0.00 bits/sec                  sender
    [  5]   0.00-10.11  sec   494 MBytes   410 Mbits/sec                  receiver
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    

    And then from Gaming > Main

    Gaming VLAN>iperf3.exe -c 192.168.10.60
    Connecting to host 192.168.10.60, port 5201
    [  4] local 192.168.13.235 port 64557 connected to 192.168.10.60 port 5201
    [ ID] Interval           Transfer     Bandwidth
    [  4]   0.00-1.00   sec  34.2 MBytes   287 Mbits/sec
    [  4]   1.00-2.01   sec  33.1 MBytes   277 Mbits/sec
    [  4]   2.01-3.00   sec  33.6 MBytes   284 Mbits/sec
    [  4]   3.00-4.00   sec  32.6 MBytes   273 Mbits/sec
    [  4]   4.00-5.00   sec  29.5 MBytes   247 Mbits/sec
    [  4]   5.00-6.01   sec  31.6 MBytes   265 Mbits/sec
    [  4]   6.01-7.00   sec  33.0 MBytes   278 Mbits/sec
    [  4]   7.00-8.00   sec  31.6 MBytes   265 Mbits/sec
    [  4]   8.00-9.00   sec  33.0 MBytes   277 Mbits/sec
    [  4]   9.00-10.00  sec  32.1 MBytes   269 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth
    [  4]   0.00-10.00  sec   324 MBytes   272 Mbits/sec                  sender
    [  4]   0.00-10.00  sec   324 MBytes   272 Mbits/sec                  receiver
    
    iperf Done.
    
    Main VLAN>iperf3.exe -s
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    Accepted connection from 192.168.13.235, port 64556
    [  5] local 192.168.10.60 port 5201 connected to 192.168.13.235 port 64557
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-1.00   sec  33.3 MBytes   280 Mbits/sec
    [  5]   1.00-2.00   sec  32.5 MBytes   273 Mbits/sec
    [  5]   2.00-3.00   sec  33.6 MBytes   282 Mbits/sec
    [  5]   3.00-4.00   sec  32.6 MBytes   273 Mbits/sec
    [  5]   4.00-5.00   sec  29.3 MBytes   246 Mbits/sec
    [  5]   5.00-6.00   sec  31.7 MBytes   266 Mbits/sec
    [  5]   6.00-7.00   sec  33.1 MBytes   278 Mbits/sec
    [  5]   7.00-8.00   sec  31.5 MBytes   264 Mbits/sec
    [  5]   8.00-9.00   sec  33.1 MBytes   277 Mbits/sec
    [  5]   9.00-10.00  sec  32.1 MBytes   269 Mbits/sec
    [  5]  10.00-10.04  sec  1.58 MBytes   304 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
    [  5]   0.00-10.04  sec   324 MBytes   271 Mbits/sec                  receiver
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    

    Nothing in the network as changed since I posted the initial thread, and the SMB traffic between the VLANs is still very poor compared to what the iperf test shows (which I have seen some people the iperf test isn't a very good test).

  • LAYER 8 Global Moderator

    Iperf on pfsense is not a good test no.

    But client to client through pfsense is good test.

    So what do you see from client to client on the same network? Because those speeds are terrible for gig.. You should be seeing high 800's to low 900s for sure..

    What specific model of nas do you have? And I can help you figure out which synology iperf you want. For example on my ds918 its the apollolake..


  • @marvosa said in Very Poor Performance on VLAN Routing:

    @kdb9000 said in Very Poor Performance on VLAN Routing:

    As far as I can tell with Ubiquiti. The ports were setup as an Aggregate across the 3 ports on the Switch.

    Which mode was configured on the PFsense side? Which mode was configured on the Ubiquity switch?

    Aggregate is what is it called on the Ubiquiti side, LAGG is what it is called on the pfSense side. On the pfSense side, the protocol was LACP. Ubiquiti doesn't have any other options to change for the Aggregate (aside from setting Link Speed and how many ports are in the Aggregate). At this time, I am not running LAGG on pfSense. Instead the 3 connections are individual with different VLAN's attached to them.


  • @kdb9000

    What does an L2 iperf report ?
    I mean server & client on the same subnet

    Are you running hairpin / "On a Stick" when doing the inter Vlan xfers ?

    I have divided my Vlans across two pfSense interfaces.
    And made sure my Server and (cabled) Client vlans are on separate IGBx interfaces.

    /Bingo


  • This post is deleted!

  • @johnpoz said in Very Poor Performance on VLAN Routing:

    Iperf on pfsense is not a good test no.

    But client to client through pfsense is good test.

    So what do you see from client to client on the same network? Because those speeds are terrible for gig.. You should be seeing high 800's to low 900s for sure..

    What specific model of nas do you have? And I can help you figure out which synology iperf you want. For example on my ds918 its the apollolake..

    I haven't been able to test that yet. Having issues getting iperf on the Synology. I will say, when the Synology was on a different network (had one called Server before I moved the Synology) I had a lot of issues with transferring files to it and even using OwnCloud (which is hosted on the Synology). Backup using Veeam was also an issue (similar to what I am seeing with the on in the Gaming VLAN). Once it was moved to the Main VLAN, all of those issues went away (so going from Layer 3 to Layer 2). The number of hops and the setup of the Synology (other then the IP) has not changed.


  • @bingo600 said in Very Poor Performance on VLAN Routing:

    @kdb9000

    What does an L2 iperf report ?
    I mean server & client on the same subnet

    Are you running hairpin / "On a Stick" when doing the inter Vlan xfers ?

    I have divided my Vlans across two pfSense interfaces.
    And made sure my Server and (cabled) Client vlans are on separate IGBx interfaces.

    /Bingo

    It was "On a Stick" and worked without issue for a long time. It was only recently it started acting up. I have since spread out the VLANs onto the other interfaces, although Main and Gaming at on the same interface. When I tried to move it, pfSense was having issues with the routing (it still said it was on the one interface when I had moved it to another interface) and was blocking the traffic (at least outbound from the VLAN, inbound to the VLAN worked fine).

  • LAYER 8 Global Moderator

    @kdb9000 which specific nas do you have - can lookup up which version of the software you need. I have ds918 which is the apollolake software..


  • @johnpoz said in Very Poor Performance on VLAN Routing:

    @kdb9000 which specific nas do you have - can lookup up which version of the software you need. I have ds918 which is the apollolake software..

    DS1817+


  • @kdb9000 said in Very Poor Performance on VLAN Routing:

    Nothing in the network as changed since I posted the initial thread, and the SMB traffic between the VLANs is still very poor compared to what the iperf test shows (which I have seen some people the iperf test isn't a very good test).

    I hate it when people are using SMB as ANY kind of network performance test.
    SMB performance depends on the Server CPU load , and disk load at the exact moment.

    Then i end up having people blaming the network , for their lousy overcomitted Virtual server, with mechanical disks 😠

    /Bingo


  • @kdb9000 said in Very Poor Performance on VLAN Routing:

    although Main and Gaming at on the same interface. When I tried to move it, pfSense was having issues with the routing (it still said it was on the one interface when I had moved it to another interface) and was blocking the traffic (at least outbound from the VLAN, inbound to the VLAN worked fine).

    So the above iperfs you showed , are using "On a stick" (same) interface.
    As it's Main -> Gaming , and reverse ?

    /Bingo


  • @bingo600 said in Very Poor Performance on VLAN Routing:

    @kdb9000 said in Very Poor Performance on VLAN Routing:

    Nothing in the network as changed since I posted the initial thread, and the SMB traffic between the VLANs is still very poor compared to what the iperf test shows (which I have seen some people the iperf test isn't a very good test).

    I hate it when people are using SMB as ANY kind of network performance test.
    SMB performance depends on the Server CPU load , and disk load at the exact moment.

    Then i end up having people blaming the network , for their lousy overcomitted Virtual server, with mechanical disks 😠

    /Bingo

    I monitored the Synology system, it was basically idle when I tried doing the backup and/or file transfers. OwnCloud doesn't use SMB (at least when uploading through the web page and I do not believe so with the Windows Client), and was also problematic. The Veeam backup does use SMB, and while monitoring the performance of the computer and the storage there wasn't anything that would cause the transfer to be slow. To also add to that, when I did the VLAN setup on the Synology (one VLAN in the same as the system backing up, the other required me to go to pfSense) it would preform at the 500 KB/s going across the VLAN compared to the MB/s I see when I went to it directly on Layer 2. These test were done one right after the other (not at the same time).

    I will also point out that I do not have any Virtual Servers (unless you want to count what I am running in Docker as a Virtual Server) in play with this setup. If you want to blame Docker for the OwnCloud part, I had slow transfers, interrupted transfers, and issues when using it over Layer 3. I did not have any issues when I switch it to Layer 2.


  • @bingo600 said in Very Poor Performance on VLAN Routing:

    @kdb9000 said in Very Poor Performance on VLAN Routing:

    although Main and Gaming at on the same interface. When I tried to move it, pfSense was having issues with the routing (it still said it was on the one interface when I had moved it to another interface) and was blocking the traffic (at least outbound from the VLAN, inbound to the VLAN worked fine).

    So the above iperfs you showed , are using "On a stick" (same) interface.
    As it's Main -> Gaming , and reverse ?

    /Bingo

    Yes, until I can get pfSense to correctly move the VLAN to another interface. I am Working from Home, so it isn't very easy to reset my Router/Firewall at this time.


  • @kdb9000

    My SMB "rant" was not meant for you, in particular.
    It was gathered , from many job debug situations , where 95% of the SMB tests , were proven wrong by iperf. But it takes a lot of convincing to get a M$ Admin to accept that iperf is the way to go, when testing network performance.


  • @bingo600 said in Very Poor Performance on VLAN Routing:

    @kdb9000

    My SMB "rant" was not meant for you, in particular.
    It was gathered , from many job debug situations , where 95% of the SMB tests , were proven wrong by iperf. But it takes a lot of convincing to get a M$ Admin to accept that iperf is the way to go, when testing network performance.

    I wasn't sure, but I know some people might pick it up and run with it. And I know what you mean, we have to fight with out Database people about the Storage system (they keep blaming performance issues on Storage when we do not see any issues related to it). We did find issues with the Databases that we brought to their attention, and after that most of the issues stopped or it wasn't as bad as it was.



  • @bingo600 said in Very Poor Performance on VLAN Routing:

    @kdb9000

    Did you see this one
    https://www.reddit.com/r/synology/comments/f2s6nv/only_getting_1gbit_transfer_speeds_rather_than/

    Something about SMB signing

    SMB Signing is related to Domains (as far as I can tell and something that was mentioned in the Reddit you sent). One of the Synology troubleshooting guides mentioned SMB Signing as well, but it was only if the Synology was part of a Domain setup.


  • @kdb9000

    Re iperf install

    https://www.synology.com/en-global/knowledgebase/DSM/tutorial/Compatibility_Peripherals/What_kind_of_CPU_does_my_NAS_have

    DS1817+ Intel Atom C2538 Quad Core 4 Yes Avoton DDR3 2/8 GB

    Do you see any Avoton arch - iperf packages

  • LAYER 8 Global Moderator

    @bingo600 said in Very Poor Performance on VLAN Routing:

    DS1817+ Intel Atom C2538 Quad Core 4 Yes Avoton DDR3 2/8 GB

    http://www.jadahl.com/iperf-arp-scan/DSM_6.2/iperf_avoton-6.2_3.7-1.spk

    Install that spk, then ssh to your nas, then just run iperf3 -s

    Then hit your nas IP from linux or windows client also running iperf.. I compile iperf3 for windows myself..

    Here is that
    iperf3.9_64.zip


  • @bingo600 said in Very Poor Performance on VLAN Routing:

    @kdb9000

    Re iperf install

    https://www.synology.com/en-global/knowledgebase/DSM/tutorial/Compatibility_Peripherals/What_kind_of_CPU_does_my_NAS_have

    DS1817+ Intel Atom C2538 Quad Core 4 Yes Avoton DDR3 2/8 GB

    Do you see any Avoton arch - iperf packages

    @johnpoz said in Very Poor Performance on VLAN Routing:

    @bingo600 said in Very Poor Performance on VLAN Routing:

    DS1817+ Intel Atom C2538 Quad Core 4 Yes Avoton DDR3 2/8 GB

    http://www.jadahl.com/iperf-arp-scan/DSM_6.2/iperf_avoton-6.2_3.7-1.spk

    That one worked. Is there a specific way to turn on the server part (do I need to do it from the command line or is it just always running)?

  • LAYER 8 Global Moderator

    see my post above yours ;) I edited it.

    from the client you would run iperf3 -c ipaddressofnas

    edit: To @bingo600 rant on smb tests.. I really agree there... While slow smb transfers of can be a symptom of network problems. It doesn't prove that there is a network issue..

    If you do an iperf and its slow - then that can explain your slow smb. If your your iperf test shows the network is fine for speed, then your slow smb problem is something else..

    The iperf you showed are not normal for a gig network... So something is clearly not right there..

    But lets see test from device A to B on the same network - this should be 940ish maxed out if everything is perfect.. But as long as your in the 800's - could just be driver issue, etc.. Maybe you need to do some tweaking of the driver settings or something. But if you were seeing 800ish on your network... You should be seeing more than 80MBps -- should be 100 or so.. And you sure shouldn't be seeing wild fluctuations in the transfer


  • @johnpoz said in Very Poor Performance on VLAN Routing:

    see my post above yours ;) I edited it.

    from the client you would run iperf3 -c ipaddressofnas

    Not sure which post you are referring to. When I try and run the test to the Synology, I get connection refused (I have it installed and it says it is running).

  • LAYER 8 Global Moderator

    And are you running firewall on your NAS?


  • @johnpoz said in Very Poor Performance on VLAN Routing:

    And are you running firewall on your NAS?

    No, it is disabled.


  • @kdb9000 said in Very Poor Performance on VLAN Routing:

    @johnpoz said in Very Poor Performance on VLAN Routing:

    see my post above yours ;) I edited it.

    from the client you would run iperf3 -c ipaddressofnas

    Not sure which post you are referring to. When I try and run the test to the Synology, I get connection refused (I have it installed and it says it is running).

    Now .. Your pfSense or your NAS isn't blocking the iperf packages ?
    I usually "cheat" and run iperf as root ... Not that that would help if iptables is blocking.

    /Bingo