WebGUI performance related to password hash strength - improved WebGUI speed by lowering hash. Why?

[rgijsen]

We have quite a bunch of 1100 and 2100 appliances on smaller offices. They would never be really quick of course, but lately I noticed some of them being about twice as slow than others in the WebGUI, to the point where it's a huge nuicance to use them. I tried to narrow down for hours and hours between a slow and a fast unit, until the last thing different in the config was the user account. It couldn't be that right? Wrong.
It seems to be related to https://redmine.pfsense.org/issues/10298, which is about password hashing complexity, and was closed 4 years ago. I'm not sure when it finally got default in pfSense+.

Basically a users password previously got hashed with BCrypt with cost-factor 10, and now with cost-factor 12. It turns out this has great influence on the GUI speed. I'm no web-developer by any means, but I don't see why, after authenticating, this should have any influence when using session tokens, correct me if I'm wrong. On bigger (read: AMD64) units, this will probably not be noticable but on the small guys it is.

Of course, it's very important to have strong security on your accounts. At the same time, 1100 and 2100 units are becoming so tedious to work with, we decided to go with the old encryption. We made sure the WebUI is only accessible from one management server, and lowered the encryption cost-factor again. This made them SO much faster!

Original 'slow' config.xml user:

<user>
<name>admin</name>
..other info..
<bcrypt-hash>$2y$12$your hash here</bcrypt-hash>
</user>

Fast config.xml:

<user>
<name>admin</name>
..other info..
<bcrypt-hash>$2y$10$your hash here</bcrypt-hash>
</user>

The difference is in the $2**$10** and $2**$12** in the password hash. You can generate your own hash online (not adviced) or with a local tool, and put that in the config. My advice is to first create another admin user, and change that first. You could of course change your primary admin account but you could get locked out if you mess up, needing the console to recover.

Of course, only do this on systems where it's feasible. Lock down the WebGUI, and certainly don't lower security if you allow the webGUI to be accessed from whole your office IP-space. Having said that, it tremendously increased speed on the WebGUI for us, to a point where it's actually usable again.

I've not checked how thich would influence RADIUS accounts, as currently we don't use that. I suppose though they default to $2$12 as well.

Question for Netgate people; is there a reasoning why after authenticating, the password hash mechanism could or should have this much influence?

[luckman212]

This piqued my curiosity, so I decided to check my 6100 (running 26.03 beta, with an old config that's been upgraded over many years) and, lo and behold, I had the bcrypt-10 style hash for my admin password. As a test, I changed my password and re-saved it, confirming that in the config.xml it was now stored with cost factor 12.

After that, I experienced a noticable slowdown in page loads across the board (not just log/dashboard, but basically every page load was (unscientifically measured) about 2x slower.

I'm now digging into this, looking at auth.inc, phpsessionmanager.inc etc.

edit: made a patch which fixes this for now, until 26.03 is released when it should hopefully no longer be needed.

[fabnavigator]

@luckman212 I tested this on my 2100 Max. Here are the results. Of course my reaction time with the stopwatch makes the numbers slight higher than they actually are. My test was clicking back and forth between WAN and LAN in my firewall rules.

cost-factor : time (seconds)
12 : 4.7
10 : 4.0
4 : 3.2

So, I need someone to tell me why using 4 is a very bad idea.

[rgijsen]

Using SHA512 is about as fast, or maybe a tad faster than BCrypt-10. Now one should never ever use SHA for passwords, but there you go. As stated before, good encryption is important for your passwords. Having said that, if you force https on your webgui, or use SSH, the data is encrypted already anyway, and sniffing would be harder. In combination with allowing management access from a dedicated IP or maybe a management subnet, I guess there'd not be much harm in lowering the password hash. Still, it's totally dependent of the type of network the Netgate runs in. In a hospital I'd not lower security, but they won't be running an measly 2100 either

I didn't expect an 6100 to be impacted that much, as it has quite a bit more CPU power.

My main question is, why the whole WebGUI needs to be slower after logging in. I can imagine some E2E encryption running, but when HTTPS is in use already, for most environments I'd say that suffices? Especially with a cost-factor 4 hash, even the 1100 and 2100 get quite responsive and quite a bit more pleasant to use. Anyone with some insights?

[luckman212]

I spent the day getting Xdebug built for 26.03 dev. It wasn't the easiest thing I've done. Spun up a Proxmox VM with FreeBSD-16.0-CURRENT-amd64-zfs-20260216-e4bcfe4ebf03-283952 which was the closest to the current public beta snap build. Had to fiddle to get the qcow2 image imported and converted to the right disk format.

Then I was able to jimmy the pkg repos and certs in there in order to install php85-8.5.2 and compile the xdebug.so module. It works! @jimp FYI (last word on this in 2022 was a no-go, but it seems that's been sorted!)

Never used this before so it was a lot of fooling around with the ini files and VSCode extension, but I'm making progress... it's doing the stuff now...

[johnpoz]

Following this thread.. I too am very curious why the hash for the password would have impact on loading pages once authed..

I would think once your authed, it shouldn't have to deal with the password hash again? Should it?

[stephenw10]

Are you seeing this only in Plus? Only with Nexus enabled/disabled perhaps?

[fabnavigator]

@stephenw10 I only have Plus. I don't use Nexus.

[stephenw10]

Mmm, OK we are digging into this. Some great info provided here already makes it easier.

[ltctech]

@stephenw10

Seems check_current_password is being called once or twice on every page load via head.inc to check for a poor password? Which then calls password_verify($passwd, $user['bcrypt-hash'] in auth.inc?

https://github.com/pfsense/pfsense/blob/master/src/usr/local/www/head.inc#L574
https://github.com/pfsense/pfsense/blob/master/src/etc/inc/auth.inc#L699
https://github.com/pfsense/pfsense/blob/master/src/etc/inc/auth.inc#L449

[jwt]

this was done to comply with various (California, etc) laws requiring password security.

it's being investigated for rework for 26.03

[luckman212]

I'm trying to wrap my head around the QCachegrind GUI that parses these Xdebug traces. I chose the System Patches page for testing (not sure why, but it seemed like a fairly simple page, and loads very quickly on an identical 6100 with a smaller level-10 bcrypt password).

So far it looks like this slowdown is caused by some unnecessary or at least redundant calls to PHP's internal password_verify() function. That seems to be where most of the time is being spent. That seems to be called 4 times for just this single page load... it's called by pfSense's check_current_password() function inside auth.inc 2 times during this exec (looks like once by head.inc and once by authgui.inc). Not sure why.

[luckman212]

I've been looking at the code in head.inc and it really seems overly cautious. I don't know which California laws specifically @jwt was referring to requires the password to be hashed 2-4 times on every page load... but it seems like a bad trade to make everyone in the world suffer with slow pageloads just to appease what must be a low-single-digit percentage of the overall user base.

As a test, I changed the check_current_password() function in /etc/inc/auth.inc from

function check_current_password($username) {
	if (local_backed($username, $username)) {
		return sprintf(gettext('Password is currently set to the username (%s).'), $username);
	}
	if (local_backed($username, g_get('factory_shipped_password'))) {
		return sprintf(gettext('Password is currently set to the default value (%s).'), g_get('factory_shipped_password'));
	}
	return null;
}

to

function check_current_password($username) {
	return null;
	if (local_backed($username, $username)) {
		return sprintf(gettext('Password is currently set to the username (%s).'), $username);
	}
	if (local_backed($username, g_get('factory_shipped_password'))) {
		return sprintf(gettext('Password is currently set to the default value (%s).'), g_get('factory_shipped_password'));
	}
	return null;
}

BAM instantly this fixed the problem, pages are snappy again!

How about a new checkbox:

I don't live in California (or don't care)

[jwt]

@luckman212 said in WebGUI performance related to password hash strength - improved WebGUI speed by lowering hash. Why?:

BAM instantly this fixed the problem, pages are snappy again!

How about a new checkbox:

I don't live in California (or don't care)

unfortunately, that's not how the various laws (example: California's SB-327) read. If you think that's ugly, take a look at EU's CRA

Even.More.Red.Tape.

In any case, what I said applies, we'll address this in 26.03 (which was near RC ...)

[rgijsen]

@luckman212 That's some great investigation there. As per test, I made this modification on a 2100 and virtualized CE instance, the difference especially on the 2100 is astounding. Looking forward to 26.03 :-)

[luckman212]

For anyone following along who wants to speed up their pageloads without hand-editing any files, I made a patch that can be applied right from System Patches. It only makes that 1 small change. Tested on 25.07.1, 25.11.1, and 26.03.b.20260219.2016.

Just go to System → Patches → Add New Patch

• Description: make check_current_password() a noop
• Commit ID: f63fa2ff5e814af0aa835a48e4250939df8debfd
• Patch Contents: (leave empty)
• Leave everything else as-is
• Click Save

Then click Fetch, then Apply. It should look like this:

[Mission-Ghost]

@luckman212 wow! Thank you for this.

The r&d into this problem is impressive and highlights the value of the “open” in open source.

Makes me wonder…did this module ever receive a code review or sqa in its life before this?

[Gertjan]

@luckman212

Wait ...
I'll start with a
and I love the

You forked pfSense into your own github.
Applied a patch and you have the commit ID : Commit ID: f63fa2ff5e814af0aa835a48e4250939df8debfd

When I feed this "Commit ID: f63fa2ff5e814af0aa835a48e4250939df8debfd" into the pfSense patcher, it found it and loaded it.
From your github, not the 'official one'
I get it : I see what it did, :

but this also means I could fork pfSense, write pure BS patches, publish the commits and wait for the sky to fall.
Did I miss something or is it that easy ?
Humm, to much freedom is nice, and a scary thing.

[johnpoz]

@Gertjan said in WebGUI performance related to password hash strength - improved WebGUI speed by lowering hash. Why?:

publish the commits

Yeah that seems a bit odd to me too, while sure the freedom of pulling patch from 3rd party is a feature if you ask me. I would hope the admin would have to make a conscious decision to do so.. Like enable 3rd party patches checkbox or something.. Or I would hope the patch manager would be limited to patches actually published by netgate.

I would of thought some sort of extra step would be required to use some patch from somewhere other than netgate other then just putting in a commit ID.

I put this in by hand last night to test, and on my 4860 I didn't really notice any difference.. I looked at my admin account and its using the older bcrypt-hash>$2y$10 on my admin account, new user I create does use bcrypt-hash>$2y$12

[Gertjan]

@johnpoz said in WebGUI performance related to password hash strength - improved WebGUI speed by lowering hash. Why?:

Yeah that seems a bit odd to me too

Thanks for that confirmation. I really stated to think I 'missed' something.

@johnpoz said in WebGUI performance related to password hash strength - improved WebGUI speed by lowering hash. Why?:

my 4860 I didn't really notice any difference..

On my 4100 neither. The dashboard still takes 3 seconds, (most of )all the other : snappy.
Anyway, it's a bit silly to compare a "1100" with a "4100".

@luckman212 : nothing against you of course