Improving slow boot with large number of accounts?

[guitarpicker]

I am setting up pfSense 2.2.4 (AMD64) as a captive portal for an Internet connection.  It is running on a Dell Precision workstation with a couple of non-SSD hard drives in a GEOM mirror.  We have over 700 user accounts, but the actual usage is only a handful at a time.  I imported the accounts by uploading an edited config.xml file that had all the accounts in them.  The import ran very quickly, and the accounts were working.  However, every time I have rebooted since importing, it takes about 20 minutes longer to boot.

During the delay, pfSense is busy "Synchronizing user settings" with the hard drive light indicating heavy usage.  No traffic is able to pass during this delay.  I perused the source code, and it's basically obliterating the groups and users and adding them back from the config file.

I find it very odd that the rebuild during reboot takes significantly longer than the initial import did.  Does anyone know of any way to tweak the drive cache or delay processing until after the firewall has started?

I had considered using the FreeRADIUS module, which probably has less boot-time overhead, but I didn't like that it stores the clear text passwords in the config file, so I went with the standard accounts.  We would rather live with the delay than risk exposing our user's passwords if the machine were compromised.

[heper]

i've never used freeradius before, so i have no clue if theres a way to circumvent the clear-text issue.

i generally have a active-directory running on sites with >300 users … its pretty easy to get CP authenticating with active-directory.

[guitarpicker]

Unfortunately, this is on an isolated network segment, and everything needs to run on the same machine.

[cmb]

You're in uncharted territory there AFAIK, everyone I'm aware of with >100 users uses RADIUS or LDAP because that's easier to manage, and often already exists (MS AD or similar).

It's not the most efficient thing in the world it's doing there. It should really be idempotent rather than redoing everything, and that will happen at some point in the future.

How'd you do the "initial import"? The process that syncs accounts should be the same regardless of whether or not you're booting.

[guitarpicker]

I guess I get to chart a little bit of this territory, then. :D
As far as being idempotent (thanks for the new word), I have to chuckle a little, pfSense's method is very similar to how I imported users on the last captive portal I had set up.  It was always much easier to bulk delete and bulk import than it was to only update the changed items.  I can appreciate the difficulty involved in improving this.

I would love to go with LDAP or FreeRADIUS if there is a straightforward way to run them on the same machine and not store plain text passwords.  We're trying to keep complexity low while practicing good security.  Basically, we are hosting an Internet hot spot for registered visitors to our location.  I'm making a best effort to protect their data, since I'm sure that many of them probably use the same user name and password for more important systems than ours.

The initial import was done by exporting the config.xml, inserting the appropriate User and Group lines, and reimporting the System portion of the configuration.  It didn't take a noticeable amount of time for the import to complete, and for the new accounts to be available.

The XML code was generated from a CSV to XML PHP script I found at https://github.com/Daxx92/pfsense-user-xml-generator.  The output looked sane compared to users created through the GUI, and the UID started at the right number and incremented properly.

 <user><scope>user</scope>
 <password>$1$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</password>
 <md5-hash>yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy</md5-hash>
 <nt-hash>zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz</nt-hash>
 <name>someuser</name>

 <expires><authorizedkeys><ipsecpsk><uid>2090</uid></ipsecpsk></authorizedkeys></expires></user> 
...

I also added group entries for each imported user in the appropriate group.

 <group><name>StandardUsers</name>

  <gid>2000</gid>
  <member>2002</member>
  <member>2003</member>
  <member>2004</member>
  <member>2005</member>
  <member>2006</member>
...</group> 

I also updated <nextuid>to the appropriate number.  I did just notice that the "<scope>system</scope>" line was missing from my group, so I added it now.  I haven't had a chance to reboot and see if this improves anything.

Due to the results, I am considering some combination of the following possibilities:

• The boot-time code runs slower than the web interface, perhaps due to some sort of caching or optimization not being loaded yet.

• The web interface code actually takes the same amount of time to run, but it does so in the background, and is transparent to the administrator since the user accounts show up in the list quickly and are usable.

• The boot-time import code is doing more than the web interface code.</nextuid>

[bullet92]

I had considered using the FreeRADIUS module, which probably has less boot-time overhead, but I didn't like that it stores the clear text passwords in the config file, so I went with the standard accounts

From what i knwown freeradius supports encrypted users passwords. It's sufficient to select md5-password or what else instead of clear text password. Am I missing anything?

[guitarpicker]

@bullet92:

From what i knwown freeradius supports encrypted users passwords. It's sufficient to select md5-password or what else instead of clear text password. Am I missing anything?

FreeRADIUS does store them encrypted in the FreeRADIUS files when you choose the appropriate encryption.  However, pfSense stores the unencrypted one in the config.xml file, presumably so it can build the encrypted FreeRADIUS's tables on boot.

It's discussed in the FAQ, and I can attest that it is correct.
https://doc.pfsense.org/index.php/Why_are_some_passwords_stored_in_plaintext_in_config.xml

Sample:

		 <freeradius><config><sortable><varusersusername>importantdude</varusersusername>
				<varuserspassword>SuperSecretPassword</varuserspassword>
				<varuserspasswordencryption>MD5-Password</varuserspasswordencryption>
				...</sortable></config></freeradius> 

[bullet92]

I did not know this behavior  :P
Is possible an upgrade to SSD disk?
Sounds crazy a script that save users in a custom file, at reboot shutdown delete users from pfsense config (i think is unnecessary considering that u are adding this users manually is necessary because u will add further users from GUI, I presume) and then import after boot then from custom file? :o

[guitarpicker]

SSD disks are not a viable option for us, but would probably help a good bit.  This is a low-volume system which does not justify high-end hardware.

The custom import was a one-time action, and does the same thing as if the users were entered via the GUI.  After this import, all new users are being added via the GUI.  I am not running any custom scripts

The deletion happens at every boot is just how pfSense (and upstream m0n0wall) work - not by any sort of customization that I have done.  You can see for yourself in the source code for the local_sync_accounts function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L378) which gets called at boot up.  The general flow as seen in the comments is:

• Delete local users

• Delete local groups

• Sync (import) all local users

• Sync all local groups

This simple and rather foolproof method of synchronizing the user accounts with the configuration file could be optimized for performance, but the code would be immensely more complex to do so.  I think the lion's share of the delay is due to calling the local_user_set function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L450) for each user on each boot, which has a laundry list of things to do when setting up a new user.

It sounds like this isn't a high-demand feature.  I would much rather have an option to use FreeRADIUS without storing the plain text passwords than to spend a lot of time optimizing the local account sync process.  I don't think this is likely to happen either, since the whole reason they store the passwords is so that you can change the encryption type in FreeRADIUS without losing all the accounts.

Fortunately, pfSense has been rock solid and I haven't needed to reboot much.  I schedule it to reboot at night when I need to, so that the delay doesn't affect our users.

[NOYB]

On a very basic implementation with only a few accounts (less than 6) I randomly see delay at "Synchronizing user settings" during boot up.  Sometimes it takes maybe a minute, other times only a second or so.  It is inconsistent from one boot to the next.  USB Flash Drive Full Install with /tmp and /var ramdisk.

[bullet92]

@guitarpicker:

The custom import was a one-time action, and does the same thing as if the users were entered via the GUI.  After this import, all new users are being added via the GUI.  I am not running any custom scripts

Sorry i wasn't clear. I meant: what about a custom script that save user somwhere, download it at boot and then readd it (maybe in import-like mode)?
Firewall would be reacheble even without users and then wil readd it again. But u should be able to:
1. prevent pfsense to save itself the users
2. store them (local persistent HD or remote)
3. import them at startup (maybe the simpler things to do)
@guitarpicker:

The deletion happens at every boot is just how pfSense (and upstream m0n0wall) work - not by any sort of customization that I have done.  You can see for yourself in the source code for the local_sync_accounts function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L378) which gets called at boot up.  The general flow as seen in the comments is:

• Delete local users

• Delete local groups

• Sync (import) all local users

• Sync all local groups

This simple and rather foolproof method of synchronizing the user accounts with the configuration file could be optimized for performance, but the code would be immensely more complex to do so.  I think the lion's share of the delay is due to calling the local_user_set function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L450) for each user on each boot, which has a laundry list of things to do when setting up a new user.

It sounds like this isn't a high-demand feature.  I would much rather have an option to use FreeRADIUS without storing the plain text passwords than to spend a lot of time optimizing the local account sync process.  I don't think this is likely to happen either, since the whole reason they store the passwords is so that you can change the encryption type in FreeRADIUS without losing all the accounts.

Fortunately, pfSense has been rock solid and I haven't needed to reboot much.  I schedule it to reboot at night when I need to, so that the delay doesn't affect our users.

Looking at code it call system binary file to read and write users correctly(and set them the password):

$user_op = "useradd -m -k /etc/skel -o";

$cmd = "/usr/sbin/pw {$user_op} -q -u {$user_uid} -n {$user_name}".
" -g {$user_group} -s {$user_shell} -d {$user_home}".
" -c ".escapeshellarg($comment)." -H 0 2>&1";

Write users differently implicate a function that write "X" users directly to user file being careful to not corrupt this file.. it seems risky :D