Slow Web GUI with many VLAN Interfaces - 300$
-
but the Fix got moved to pfSense version 2.4.1.
I read that as a fix exists but isn't going to be made public yet.
-
Bad news: pfSense 2.4 is even worse :(
good news: I fixed it :)
Weird news: Exactly the same edits that shaved 7 seconds off before now shave 2:20 off ::)
Files: /etc/inc/interfaces_fast.inc
/usr/local/www/interfaces_assign.php
/usr/local/www/interfaces_vlan.php[slow ui 2-4.zip](/public/imported_attachments/1/slow ui 2-4.zip)
-
Has this been fixed in 2.4.1?
-
-
Has this been fixed in 2.4.1?
not officially, my fixes should work.
Can you please submit a Pull Request on https://github.com/pfsense/pfsense so we can review and merge the fixes?
-
I did some testing with multiple vlan interfaces using standard pfSense 2.4.1 to get a better view of the problem.
Boot time with 001 vlans: 0 min, 45 seconds
Boot time with 050 vlans: 0 min, 49 seconds
Boot time with 100 vlans: 0 min, 59 seconds
Boot time with 200 vlans: 1 min, 52 seconds
Boot time with 300 vlans: 4 min, 19 seconds
Boot time with 400 vlans: 9 min, 16 seconds
Boot time with 500 vlans: 13 min, 19 secondsAs you can see the boot time is not linear. Maybe this helps with finding and implementing a fix. Because a fix needs to address this non-linear groth as well. When pfSense is finshed with booting with 500 interfaces the web GUI just gives a 504 after some time. Using 400 vlan interfaces and less the web GUI is slow. Then I tried with loonylion patch which he posted October 19 in this tread using 300 vlans.
Boot time with 300 vlans, loonylion path: 3 min, 51 seconds
To GUI is slow with 300 interfaces. Even with the patch from loonylion. I didn't notice a difference.
Test where done using a Intel(R) Xeon(R) CPU E3-1585 v5 @ 3.50GHz, 16GB of memory.
Hope this helps.
-
my patch will not affect boot time in the slightest, it's a GUI modification only. There must be something else at play to get those results.
-
Has this been fixed in 2.4.1?
not officially, my fixes should work.
Can you please submit a Pull Request on https://github.com/pfsense/pfsense so we can review and merge the fixes?
I'll tidy up the code and try to figure out how to do this; I've never used git before.
-
I know this doesn't really solve the issue but isn't having 300+ interfaces off a firewall kind of crazy? I probably would virtualize your PfSense and have several PfSense vm's running in the same box and try to get the job done like that. Are there switches out there that can handle 300 vlans? Again I know this doesn't solve the root issue just seems like a engineering issue.
-
I'm in the process of tidying the code up and making sure it adheres to the pfSense coding guidelines. I also decided to do a bit more profiling of it, and I came up with the attached graph. At 500 VLANs the page load time with my patches is 43 seconds as measured by FireFox. The original code times out as mentioned above. Also noted above, the time doesn't increase linearly with the original code, and my graph shows that.
The graph shows page generation time, actual load time as experienced by the user will be a bit longer. These times are for viewing the interface_assign.php page; for adding an interface add ~3 seconds to page generation and for deleting add ~2 seconds.
 -
cleaned up and seems to follow coding guidelines as far as I can see, hopefully final version attached.
I think I managed to work the pull request stuff out, would be nice if someone with the relevant knowledge/access could confirm I've got it right, because as I said, I've never used git before.
it says 2.4 but I can confirm it works on 2.4.1
EDIT: further improvements, mainly on page load time.
EDIT2: bugfix[slow ui 2-4.zip](/public/imported_attachments/1/slow ui 2-4.zip)
-
cleaned up and seems to follow coding guidelines as far as I can see, hopefully final version attached.
I think I managed to work the pull request stuff out, would be nice if someone with the relevant knowledge/access could confirm I've got it right, because as I said, I've never used git before.
it says 2.4 but I can confirm it works on 2.4.1
if you've never used git before, i'd suggest you use the github webgui editor todo the work for you
1)basically you go to the pfsense github page & select the master branch.
2)then you find the file you wish to edit. You make your changes & click 'propose change' (fill topic/comments to explain your commit)
3)github will now fork the repo & you will have your own version of the pfsense code.
4)adjust other files in the same way, but this time be sure to edit them in your forked version (for example goto: github.com/loonylion/tree/patch-1)
5)adjust adjust adjust
6)click the 'new pull request' button & if you are certain, send it
7)you probably need to sign a CLA before they can accept your code (unless that changed recently) -
updated archive with further improvements, it's now under 30 seconds from request to complete page load with 500 VLANs. Also added to pull request.
just tested with 1001 VLANs, takes about 1 minute for the page to load fully. -
Thanks loonylion for your work and the pull request. Hopefully it will be reviewed and added by pfSense.
Any idea on how to lower the boottime with this amount of interfaces? Having to wait for about halve a hour after a reboot is a bit stressful :) -
I haven't looked at the boot process but I dont think mine takes anywhere near that long even with 300-500 vlans.
-
Hi there,
have to confirm. Issue is still present with 2.4.2-p1.
interfaces_assign.php does not load any more, get a 504 error.
I have about 150 VLANs, but only 50 VLAN interfaces used atm.
But Issue starts here already with ~ 5 VLAN interfaces (interfaces_assign.php is very slow then)
Do not see the boot issue.
With the fix from loonylion everything works as expected. Thank you
Btw: There is also an issue with the dashboard if there are so many VLAN interfaces used, dashboards loads much slower, too, but does not break totally…
Cheers
Martin -
have to confirm. Issue is still present with 2.4.2-p1.
interfaces_assign.php does not load any more, get a 504 error.
I have about 150 VLANs, but only 50 VLAN interfaces used atm.
But Issue starts here already with ~ 5 VLAN interfaces (interfaces_assign.php is very slow then)
I'm on 2.4.2-p1 , and have 17 Vlans.
I have never experienced any problems or 504 timeouts.So it's not all >= 5 vlan installations , that are affected.
/Bingo
-
the primary cause of this bug is that essentially because of how the code is/was written, both the page generation time and the size of the output HTML increase exponentially as more VLANS are added. The patch I've submitted removes the exponential increase part from the page generation time, and reduces the base HTML output size. Even so, with 1k VLANs the output HTML weighs in at a hefty 64MB.
There are two solutions to this side of the problem, as far as I can see: 1) redesign the page, which I don't believe is within my authority to do, or 2) add all the select boxes (by far the most significant contributor to the bloat) via javascript after the page has loaded (so that you're only sending a single select box rather than 1+(1*VLANs) select boxes.) My javascript skills are pretty ropey so I'm not sure that its within my ability to achieve.
-
I would second the option to just virtualize many firewalls. I have a cloud solution for clients running on vmware and I have my internet pipes vlan'd on the network so I can just spool up a pfsense per client.
The downside, you would need much more than an E3. I almost went the "super-firewall" route using a server with dual E5-2630v4 and 64GB of RAM with 8x 240GB SSDs in a RAID 10. But then decided to just use smaller virtual firewalls on my main ESXi servers.
A managed switch, even if it is just a "smart" switch that can handle vlans on the internet side as what I call a "dirty switch". VLan your internet pipes, lets VLAN150 and VLAN151. Then I would route that to dual servers, single E5-2620v4 with at least 16GB of ram or a dual E5-2620v4 with 32 GB minimum. Then load ESXi and set 6 total firewalls with 50 VLANs each.
You can use something like pfmonitor to manage all of those virtual firewalls.
You could conceivably even have 300 virtual firewalls, I would have more powerful servers. Maybe a stack of 3 dual-proc servers running full vmotion and such, like ESX Essentials Plus.
Or at that point, just do straight up L3 routing with a dedicated external IP per ethernet port dorm. Then let the kids put their own firewalls and wireless networks in. Sure it causes congestion, but if it works in high-rises in NYC. Heck I live in a suburban neighborhood with 53 other houses on my street and I can clearly see a dozen or more wireless networks.
