50x Slower VLAN Enumeration on pfSense 2.8.1

seb69300

Hi,

After upgrading from pfSense 2.7.x to 2.8.1, we're experiencing severe performance degradation when pfSense enumerates interfaces.

Our environment contains a large number of VLANs , which makes the issue particularly noticeable

2.7.0 :
pfs-2.7.0_ifconfig2.png

2.8.1 :
pfs-2.8.1_ifconfig2.png

This causes significant delays in the dashboard and interface-related pages.

It is exactly the same hardware and the same configuration, yet the difference in execution time when listing interfaces/VLANs with ifconfig -v between pfSense 2.7.0 and 2.8.1 is simply staggering.

Are you aware of any major changes introduced in pfSense 2.8.x or FreeBSD 15 that could explain such a significant performance regression?

Thanks ,

Best regards

johnpoz

@seb69300 50x?

.747 to 1.425 isn't even 2x, .017 to .080 is like 4.7x

Not sure how your coming up with 50x?

What is that 100% vs the 3.3% and 8.6% - is that related to cpu used/average?

Been really long time since used time command.

seb69300

The 50x figure comes from the elapsed time, not the CPU time.

On pfSense 2.7.0:

0.110u 0.747s 0:00.85 100.0%

On pfSense 2.8.1:

0.075u 1.425s 0:44.61 3.3%

I agree that the kernel CPU time only increased from 0.747s to 1.425s (less than 2x).

However, the elapsed time increased from 0.85s to 44.61s (~52x slower), while CPU usage dropped to only 3.3%.

seb69300

In addition, I performed a local test on a Hyper-V VM using the same pfSense version and the same configuration, and I do not experience the issue. Interface enumeration is very fast.

Could this be related to the Broadcom NICs and the bnxt driver under FreeBSD 15?

vendor = 'Broadcom Inc. and subsidiaries'
device = 'BCM57502 NetXtreme-E 10Gb/25Gb/40Gb/50Gb Ethernet'

Thanks

johnpoz

@seb69300 great follow up info - the man that I would ask about anything like this would be @stephenw10 - lets see if he chimes in now that I have mentioned him ;)

stephenw10

Yes it could. If you run an ifconfig command against that is there anything it's noticeably waiting for?

If I recall there was at least one other driver that was hanging trying to read SFP module info if the module was not present. That caused all sorts of other issues.

Just how many VLANs do you have?

seb69300

@stephenw10 We have 102 VLANs, and a simple ifconfig is not excessively slow, taking about 1.69 seconds.

However, for widgets or from the Interface Status page, it looks like pfSense enumerates or queries each interface individually. Since querying a single interface takes a significant amount of time, this likely explains why these pages become very slow when a large number of VLANs are configured.

johnpoz

@seb69300 said in 50x Slower VLAN Enumeration on pfSense 2.8.1:

We have 102 VLANs

Yup I would call that is a lot of vlans ;) I would have to double check some our sites at work - but can tell you for sure its no where near that number. Maybe 20-30 tops at our larger locations - and pretty sure those could be trimmed - quite often when we acquire a new location/company and we integrate their network into ours - clean up of old vlans is not high on priority list ;)

And now my curiosity cat is meowing at me - why would you have so many?

stephenw10

@seb69300 said in 50x Slower VLAN Enumeration on pfSense 2.8.1:

a simple ifconfig is not excessively slow, taking about 1.69 seconds.

For all interfaces?