Slow Web GUI with many VLAN Interfaces - 300$



  • Hi,

    we have pfSense system with about 350 VLAN interfaces.
    As stated here: https://forum.pfsense.org/index.php?topic=102607.0 : and here: https://forum.pfsense.org/index.php?topic=101448 the GUI becomes very slow and assign_interfaces.php becomes unusable.

    There is already a bug report: https://redmine.pfsense.org/issues/6400 where this problem is described, but the Fix got moved to pfSense version 2.4.1.

    We will pay 300$ if this can be fixed within the next two weeks, since we will add another pfSense system with that number of interfaces.
    At the moment we add,edit or remove interfaces via the developer shell, but this is very annoying.

    Some more information about why we need so many interfaces:

    We manage a modern student dorm with about 300 apartments. Each student has its own separated ip subnet (via VLAN). The VLAN Tag is either assigned via dynamic RADIUS VLAN assignment (WLAN) or via the Switch (untagged LAN Port at the apartment).

    After loggin into the WLAN, students can easily stream content from their smartphones/tablet to a connected AppleTV/Smart TV etc.
    There is also KNX Smart Home technology added so that students can control their lighting, heating, jalousie, sound etc. via Smartphone/Tablet/Voice.

    pfSense hardware:
    Xeon E3-1240v6 4x 3.70GHz
    16GB DDR4-2400 ECC RAM
    MB Supermicro X11SSH-LN4F
    SSD 240GB RAID 1
    Chelsio 10Gbit T520-SO-CR
    WAN: 1 Gbit/s Up/Down Business Fiber Connection

    If there is more information needed I’m glad to help.
    Thanks in advance!



  • Hi,

    Try these files (back up your existing ones first!) On my testing with 100 VLANs, the page load time is about 6-7 seconds vs 14-15 seconds with the default pages.

    All functionality should still be present and working.

    Files: /etc/inc/interfaces.inc
            /usr/local/www/interfaces_assign.php

    EDIT: updated archive with bug fix below.

    EDIT2: another bug fix

    [slow ui.zip](/public/imported_attachments/1/slow ui.zip)



  • Has this helped at all?



  • I have to do a little bit more testing the next days but on the first look it’s great!
    I can open interfaces_assign.php again now and it’s a lot faster.
    After more testing I’ll pm you  🙂

    EDIT:
    When I add a new VLAN it doesn’t show up on availible network ports to add a new interface.



  • bug squashed 🙂

    files: /usr/local/www/interfaces_assign.php

    interfaces_assign.zip



  • Everything working as it should?



  • Another bug squashed, seems ppp stuff got broken, so here’s fix:

    restore the backed up interfaces.inc in /etc/inc and then use the attached files:

    files: /etc/inc/interfaces_fast.inc
            /usr/local/www/interfaces_assign.php

    [slow ui.zip](/public/imported_attachments/1/slow ui.zip)



  • Guys this is great and all but if you’re going to post these publicly I think you might as well work these patches via github so they can be diffed and merged along with the rest of the project, gets more eyes on them as well…



  • I don’t use git and this particular ‘solution’ is a placeholder because it’s already been fixed officially, the powers that be just aren’t ready to release the official fix yet, and probably won’t be in the near future.



  • @loonylion:

    this particular ‘solution’ is a placeholder because it’s already been fixed officially, the powers that be just aren’t ready to release the official fix yet, and probably won’t be in the near future.

    ?? could you point us to the redmine that claims it is fixed ?



  • @posto587:

    but the Fix got moved to pfSense version 2.4.1.

    I read that as a fix exists but isn’t going to be made public yet.



  • Bad news: pfSense 2.4 is even worse  😞

    good news: I fixed it  🙂

    Weird news: Exactly the same edits that shaved 7 seconds off before now shave 2:20 off  ::)

    Files: /etc/inc/interfaces_fast.inc
            /usr/local/www/interfaces_assign.php
            /usr/local/www/interfaces_vlan.php

    [slow ui 2-4.zip](/public/imported_attachments/1/slow ui 2-4.zip)



  • Has this been fixed in 2.4.1?



  • @xciter327:

    Has this been fixed in 2.4.1?

    not officially, my fixes should work.


  • Administrator

    @loonylion:

    @xciter327:

    Has this been fixed in 2.4.1?

    not officially, my fixes should work.

    Can you please submit a Pull Request on https://github.com/pfsense/pfsense so we can review and merge the fixes?



  • I did some testing with multiple vlan interfaces using standard pfSense 2.4.1 to get a better view of the problem.

    Boot time with 001 vlans: 0 min, 45 seconds
    Boot time with 050 vlans: 0 min, 49 seconds
    Boot time with 100 vlans: 0 min, 59 seconds
    Boot time with 200 vlans: 1 min, 52 seconds
    Boot time with 300 vlans: 4 min, 19 seconds
    Boot time with 400 vlans: 9 min, 16 seconds
    Boot time with 500 vlans: 13 min, 19 seconds

    As you can see the boot time is not linear. Maybe this helps with finding and implementing a fix. Because a fix needs to address this non-linear groth as well. When pfSense is finshed with booting with 500 interfaces the web GUI just gives a 504 after some time. Using 400 vlan interfaces and less the web GUI is slow. Then I tried with loonylion patch which he posted October 19 in this tread using 300 vlans.

    Boot time with 300 vlans, loonylion path: 3 min, 51 seconds

    To GUI is slow with 300 interfaces. Even with the patch from loonylion. I didn’t notice a difference.

    Test where done using a Intel® Xeon® CPU E3-1585 v5 @ 3.50GHz, 16GB of memory.

    Hope this helps.



  • my patch will not affect boot time in the slightest, it’s a GUI modification only. There must be something else at play to get those results.



  • @Renato:

    @loonylion:

    @xciter327:

    Has this been fixed in 2.4.1?

    not officially, my fixes should work.

    Can you please submit a Pull Request on https://github.com/pfsense/pfsense so we can review and merge the fixes?

    I’ll tidy up the code and try to figure out how to do this; I’ve never used git before.



  • I know this doesn’t really solve the issue but isn’t having 300+ interfaces off a firewall kind of crazy? I probably would virtualize your PfSense and have several PfSense vm’s running in the same box and try to get the job done like that. Are there switches out there that can handle 300 vlans? Again I know this doesn’t solve the root issue just seems like a engineering issue.



  • I’m in the process of tidying the code up and making sure it adheres to the pfSense coding guidelines. I also decided to do a bit more profiling of it, and I came up with the attached graph. At 500 VLANs the page load time with my patches is 43 seconds as measured by FireFox. The original code times out as mentioned above. Also noted above, the time doesn’t increase linearly with the original code, and my graph shows that.

    The graph shows page generation time, actual load time as experienced by the user will be a bit longer. These times are for viewing the interface_assign.php page; for adding an interface add ~3 seconds to page generation and for deleting add ~2 seconds.

    ![page generation.png_thumb](/public/imported_attachments/1/page generation.png_thumb)
    ![page generation.png](/public/imported_attachments/1/page generation.png)



  • cleaned up and seems to follow coding guidelines as far as I can see, hopefully final version attached.

    I think I managed to work the pull request stuff out, would be nice if someone with the relevant knowledge/access could confirm I’ve got it right, because as I said, I’ve never used git before.

    it says 2.4 but I can confirm it works on 2.4.1

    EDIT: further improvements, mainly on page load time.
    EDIT2: bugfix

    [slow ui 2-4.zip](/public/imported_attachments/1/slow ui 2-4.zip)



  • @loonylion:

    cleaned up and seems to follow coding guidelines as far as I can see, hopefully final version attached.

    I think I managed to work the pull request stuff out, would be nice if someone with the relevant knowledge/access could confirm I’ve got it right, because as I said, I’ve never used git before.

    it says 2.4 but I can confirm it works on 2.4.1

    if you’ve never used git before, i’d suggest you use the github webgui editor todo the work for you

    1)basically you go to the pfsense github page & select the master branch.
    2)then you find the file you wish to edit. You make your changes & click ‘propose change’ (fill topic/comments to explain your commit)
    3)github will now fork the repo & you will have your own version of the pfsense code.
    4)adjust other files in the same way, but this time be sure to edit them in your forked version (for example goto: github.com/loonylion/tree/patch-1)
    5)adjust adjust adjust
    6)click the ‘new pull request’ button & if you are certain, send it
    7)you probably need to sign a CLA before they can accept your code (unless that changed recently)



  • updated archive with further improvements, it’s now under 30 seconds from request to complete page load with 500 VLANs. Also added to pull request.
    just tested with 1001 VLANs, takes about 1 minute for the page to load fully.



  • Thanks loonylion for your work and the pull request. Hopefully it will be reviewed and added by pfSense.
    Any idea on how to lower the boottime with this amount of interfaces? Having to wait for about halve a hour after a reboot is a bit stressful 🙂



  • I haven’t looked at the boot process but I dont think mine takes anywhere near that long even with 300-500 vlans.



  • Hi there,

    have to confirm. Issue is still present with 2.4.2-p1.

    interfaces_assign.php does not load any more, get a 504 error.

    I have about 150 VLANs, but only 50 VLAN interfaces used atm.

    But Issue starts here already with ~ 5 VLAN interfaces (interfaces_assign.php is very slow then)

    Do not see the boot issue.

    With the fix from loonylion everything works as expected. Thank you

    Btw: There is also an issue with the dashboard if there are so many VLAN interfaces used, dashboards loads much slower, too, but does not break totally…

    Cheers
    Martin



  • @mpcore:

    have to confirm. Issue is still present with 2.4.2-p1.

    interfaces_assign.php does not load any more, get a 504 error.

    I have about 150 VLANs, but only 50 VLAN interfaces used atm.

    But Issue starts here already with ~ 5 VLAN interfaces (interfaces_assign.php is very slow then)

    I’m on 2.4.2-p1 , and have 17 Vlans.
    I have never experienced any problems or 504 timeouts.

    So it’s not all >= 5 vlan installations , that are affected.

    /Bingo



  • the primary cause of this bug is that essentially because of how the code is/was written, both the page generation time and the size of the output HTML increase exponentially as more VLANS are added. The patch I’ve submitted removes the exponential increase part from the page generation time, and reduces the base HTML output size. Even so, with 1k VLANs the output HTML weighs in at a hefty 64MB.

    There are two solutions to this side of the problem, as far as I can see: 1) redesign the page, which I don’t believe is within my authority to do, or 2) add all the select boxes (by far the most significant contributor to the bloat) via javascript after the page has loaded (so that you’re only sending a single select box rather than 1+(1*VLANs) select boxes.) My javascript skills are pretty ropey so I’m not sure that its within my ability to achieve.



  • I would second the option to just virtualize many firewalls. I have a cloud solution for clients running on vmware and I have my internet pipes vlan’d on the network so I can just spool up a pfsense per client.

    The downside, you would need much more than an E3. I almost went the “super-firewall” route using a server with dual E5-2630v4 and 64GB of RAM with 8x 240GB SSDs in a RAID 10. But then decided to just use smaller virtual firewalls on my main ESXi servers.

    A managed switch, even if it is just a “smart” switch that can handle vlans on the internet side as what I call a “dirty switch”. VLan your internet pipes, lets VLAN150 and VLAN151. Then I would route that to dual servers, single E5-2620v4 with at least 16GB of ram or a dual E5-2620v4 with 32 GB minimum. Then load ESXi and set 6 total firewalls with 50 VLANs each.

    You can use something like pfmonitor to manage all of those virtual firewalls.

    You could conceivably even have 300 virtual firewalls, I would have more powerful servers. Maybe a stack of 3 dual-proc servers running full vmotion and such, like ESX Essentials Plus.

    Or at that point, just do straight up L3 routing with a dedicated external IP per ethernet port dorm. Then let the kids put their own firewalls and wireless networks in. Sure it causes congestion, but if it works in high-rises in NYC. Heck I live in a suburban neighborhood with 53 other houses on my street and I can clearly see a dozen or more wireless networks.


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy