Performance issues while using many vlans

[bfr]

Hi,

we have installed pfSense 2.2.4 on a server with a E3-1220 @3GHz and 16GB memory. After creating 500 VLANs with interfaces for a special setup, pfSense becomes almost unusable. When logging into the webinterface, php-fpm uses one core at 100% for 5-10 minutes.  Every click e.g. on Firewall -> Rules results in a similar behaviour.

We tried also 2.3-alpha hoping the new webinterface will solve this problem.

Is it possible to fix this issue?

Regards
bfr

[johnpoz]

500 vlans???  Over how many physical interfaces?? 1??  Come on

[firewalluser]

The use of vlans will increase the cpu demand by a bit, this then has a knock on effect elsewhere in any system.

My cpu usage is constantly at 50% + since adding 10 vlans to a single nic device running as my firewall.

Better nics with their own onboard processing help offload the load placed on cpu(s) when using vlans.

Edit. I should add as a possible workaround to the waiting times, if you have a number of rules which are applied across a number of interface groups https://doc.pfsense.org/index.php/Interface_Groups
and because the scripts will have to work in some cases checking and applying changes across all vlans which explains the waiting time, a different approach would be to have a number of pfsense instances/devices in a tree like setup, which will break down the waiting time when making rule changes. So the top tier pfsense instance/device could be used just for routing but lower level pfsense instances/devices could actually handle the rules for x number of vlans.
Ideally separate devices would be best for physical isolation, but easily achievable if running virtual pfsense instances but the risk then shifts to the virtualisation host device itself as bugs are potential backdoors.

[awebster]

I think there are two separate issues here.

First, while, to me, it seems far fetched to create 500 VLANs, the web configurator has to manage all this data for 500+ interfaces.  I'd imagine that unless it is running on a pretty hefty box, this is going to create the problem described, whenever any access is occurring to pages where interfaces are referenced.    php-fpm does not appear to be multi-threaded, so it will impact once CPU core, but should not impact the ability of the box to pass traffic.

Second, FreeBSD per-se does not have any limit on the number of VLANs, but hardware offload will certainly help, but only in situations where there is traffic. 
500 VLANs with light traffic should not put any load on the system if there is no web configurator session.

Of course the project owners could choose to put some arbitrary hard limits to the number of VLANs, interfaces, objects, whatever resource, so that it works well in all situations, but then it would probably be a closed source proprietary project.  One of the nice things about open source projects is that many things are left as open-ended, meaning if you have the resources to run it, then go for it, just don't expect it to work in every extreme situation imaginable.

[Guest]

500 vlans???  Over how many physical interfaces?? 1??  Come on

??? It takes me real 2 minutes to stop laughing about!  ;D

pfSense comes with many features, options and a rich feature set, but activating them all together
means also that you should be know what exactly and how hefty it is narrowing down the entire
system (hardware).

1st:
(More Routing points in the network)
Why not using Layer3 switches that are stacked together and if needed not the cheap SMB (KMU)
switches for less money, but more the $2000 - $3000 (cut ´n through) models that are capable to
handle this with ease and the firewall is then powerful enough to do the WAN - LAN and WAN - DMZ
routing only? And this might be not related to the number of employees of this company but more
tended then on the needs this company have.

After creating 500 VLANs with interfaces for a special setup,

I hope this are 10 GBit/s interfaces and then more then only one of them.

Perhaps you might have a look for other network hardware that is also taking many load from the
pfSense machine and together with stronger based hardware the pfSense will do the job more with ease!?

2nd:
(The right hardware playing also right together)
Taking other hardware together which will have many more benefits as you might be seeing on the
first look is often the goal that can be reached by assembling it wise together.

• Chelsio-T520-CR are often able to offload totally the NAT and VLAN process!
  That means two of the greater adapters would be meaning one is offloading the entire NAT load
  and the second one is offloading the entire VLAN part.
• Lanner FW-889x Series is powerfull enough to handle many more connections as an Intel Xeon E3 system
  The FW-8894, FW-8895 or FW-8896 appliances are very strong and can be sorted right with different
  NIC modules, that will be also a fine choice later to bring Intels QuickAssist in the game!
• We are using the following switches without any failure and problems!
  – Netgear XSM7224S (M7300) with Layer3 license as Core Switch or with Layer2+ for servers, NAS/SANs, ect..
  -- Netgear XCM Series (M6100) if the installation is small with and mixed set up as Access Switches
  -- Netgear GSM7352S (M5300) Layer3 stacked also as Access Switches

This is also not Cisco, Avaya, Allied Telesis, Brocade or Juniper for sure and this are also store and
forward switches, but they are fast enough for all our network needs and not the cheaper $200 SMB ones.

3rd:
(Your network topology)
Bringing it clearly to the point, you are able to stretch your network by inserting more devices
and setting up more network layers.

As now or today you have your firewall and then connect directly to this your "Access Switches"
(Access Layer) but you can also do the following:
Firewall - Core Layer - Distributed Layer - Access Layer

This might be more expensive but the whole network load would be spreading over more Switch chips
by using more switches as well and so you might speed up the entire network to a point where you will
bring it back to act smooth and liquid likes before. But this could or better should not be only pointed to
the firewall, as in days before.

[bfr]

Hi,

thanks for all the answers, especially the one which pointed out the php-fpm issue. The "high" number of vlans is related to requirements of a customer setup, so I did not want to start a discussion about the network topology or if an appliance would fit better.

The main issue is that the webinterface tries to  get all the information of interfaces etc. on every access which only scales when using a few interfaces.

Regards
bfr

[Derelict]

"A few" being dozens. I have no problem with more than a few VLANs.  < 500.

There has to be a better way (Private VLANs in your switches maybe).

[David_W]

@Derelict:

There has to be a better way (Private VLANs in your switches maybe).

I was thinking the same thing. I doubt that the customer mandates a dedicated VLAN per client device - what they are likely after is client devices isolated from each other. Most modern managed switches allow you to have a VLAN where ports intended for client devices are isolated from each other. Derelict suggests "private VLAN". I have seen the feature called "isolation".

If the original poster gives details of the switches in use then another solution might emerge.

[Guest]

Derelict suggests "private VLAN". I have seen the feature called "isolation".

Mostly together with WiFi clients and guest and private VLANs for them to separate this both
groups you can surely build two VLANs one for guest and one for private usage and then activating
the "client isolation" so that the smart phones from the guests inside the guests VLANs are not able to
see the all the other guests and trying up to connect with them. Also often used in school WLAN that
they are not able to share all the crap over the wireless network and this is then slowing down extremely
the entire throughput.

[Derelict]

Isolation across multiple APs especially across multiple switches on the same VLAN is non-trivial. The easiest way is for the feature to be built in to the wireless system or controller. But that doesn't help with guest wired ports.

[robi]

Well, the Ethernet standard says that abut 4000 VLANs can be configured on a networking device. Low end, cheap VLAN-switches handle up to 500 VLANs. Medium category to enterprise class equipment handle 4000.
So expecting pfSense to handle 500 VLANs should not be something outragious.

There may be some certain bussiness cases when this is required, that's all. I can imagine pretty complicated industrial environments where traffic itself is very low, but network has to be designed like that. Atfer all, the standard allows 4000 VLANs in a sinlge trunk, having 500 now is only just a bit more than 10%…

If pfSense fails to handle this properly because of the user interface, but othewise it could, it's simply a shame.
But that doesn't make it a worse product.

[Derelict]

Switching 500 VLANs in hardware and a router with 500 interfaces are two completely different things.

I wouldn't think a decent stack of switches would break a sweat with 500 VLANs.  Though 500 spanning tree instances might cause it some grief.

[CaptainElmo]

500 VLANS makes perfect sense for something like a WISP, where they have 500 customers each coming in on their own virtual interface.

[Guest]

So expecting pfSense to handle 500 VLANs should not be something outragious.

Yes for sure you are right, but then please also please on adequate sorted or strong enough hardware
that is able to drive this VLANs. And in the last time I see here in the forum more and more peoples
they let the router do really the switch jobs on top of all other things. If the need is there and for sure
also the traffic it must be a stronger router playing together with more powerful switches and often the
SMB (KMU) mid-ranged ones will be in the game, but not the really powerful ones for more money, but
pfSense should be then even the evil, has failures, produces problems and so on.

I like the way @Firewalluser was suggesting as a fast solution to get more headroom, building
Interface groups should be a really good point. And perhaps a Chelsio NIC from the pfSense store
that is able to full offload the VLAN part would be also a thing that could help a bit out here.

But that doesn't help with guest wired ports.

Yes for sure this is right.

500 VLANS makes perfect sense for something like a WISP, where they have 500 customers

There fore I was thinking perhaps the client isolation would do a good job, to prevent from the many VLANs.