Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Improving slow boot with large number of accounts?

    Scheduled Pinned Locked Moved General pfSense Questions
    11 Posts 5 Posters 2.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      guitarpicker
      last edited by

      I am setting up pfSense 2.2.4 (AMD64) as a captive portal for an Internet connection.  It is running on a Dell Precision workstation with a couple of non-SSD hard drives in a GEOM mirror.  We have over 700 user accounts, but the actual usage is only a handful at a time.  I imported the accounts by uploading an edited config.xml file that had all the accounts in them.  The import ran very quickly, and the accounts were working.  However, every time I have rebooted since importing, it takes about 20 minutes longer to boot.

      During the delay, pfSense is busy "Synchronizing user settings" with the hard drive light indicating heavy usage.  No traffic is able to pass during this delay.  I perused the source code, and it's basically obliterating the groups and users and adding them back from the config file.

      I find it very odd that the rebuild during reboot takes significantly longer than the initial import did.  Does anyone know of any way to tweak the drive cache or delay processing until after the firewall has started?

      I had considered using the FreeRADIUS module, which probably has less boot-time overhead, but I didn't like that it stores the clear text passwords in the config file, so I went with the standard accounts.  We would rather live with the delay than risk exposing our user's passwords if the machine were compromised.

      1 Reply Last reply Reply Quote 0
      • H
        heper
        last edited by

        i've never used freeradius before, so i have no clue if theres a way to circumvent the clear-text issue.

        i generally have a active-directory running on sites with >300 users … its pretty easy to get CP authenticating with active-directory.

        1 Reply Last reply Reply Quote 0
        • G
          guitarpicker
          last edited by

          Unfortunately, this is on an isolated network segment, and everything needs to run on the same machine.

          1 Reply Last reply Reply Quote 0
          • C
            cmb
            last edited by

            You're in uncharted territory there AFAIK, everyone I'm aware of with >100 users uses RADIUS or LDAP because that's easier to manage, and often already exists (MS AD or similar).

            It's not the most efficient thing in the world it's doing there. It should really be idempotent rather than redoing everything, and that will happen at some point in the future.

            How'd you do the "initial import"? The process that syncs accounts should be the same regardless of whether or not you're booting.

            1 Reply Last reply Reply Quote 0
            • G
              guitarpicker
              last edited by

              I guess I get to chart a little bit of this territory, then. :D
              As far as being idempotent (thanks for the new word), I have to chuckle a little, pfSense's method is very similar to how I imported users on the last captive portal I had set up.  It was always much easier to bulk delete and bulk import than it was to only update the changed items.  I can appreciate the difficulty involved in improving this.

              I would love to go with LDAP or FreeRADIUS if there is a straightforward way to run them on the same machine and not store plain text passwords.  We're trying to keep complexity low while practicing good security.  Basically, we are hosting an Internet hot spot for registered visitors to our location.  I'm making a best effort to protect their data, since I'm sure that many of them probably use the same user name and password for more important systems than ours.

              The initial import was done by exporting the config.xml, inserting the appropriate User and Group lines, and reimporting the System portion of the configuration.  It didn't take a noticeable amount of time for the import to complete, and for the new accounts to be available.

              The XML code was generated from a CSV to XML PHP script I found at https://github.com/Daxx92/pfsense-user-xml-generator.  The output looked sane compared to users created through the GUI, and the UID started at the right number and incremented properly.

              
               <user><scope>user</scope>
               <password>$1$xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</password>
               <md5-hash>yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy</md5-hash>
               <nt-hash>zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz</nt-hash>
               <name>someuser</name>
              
               <expires><authorizedkeys><ipsecpsk><uid>2090</uid></ipsecpsk></authorizedkeys></expires></user> 
              ...
              
              

              I also added group entries for each imported user in the appropriate group.

              
               <group><name>StandardUsers</name>
              
                <gid>2000</gid>
                <member>2002</member>
                <member>2003</member>
                <member>2004</member>
                <member>2005</member>
                <member>2006</member>
              ...</group> 
              

              I also updated <nextuid>to the appropriate number.  I did just notice that the "<scope>system</scope>" line was missing from my group, so I added it now.  I haven't had a chance to reboot and see if this improves anything.

              Due to the results, I am considering some combination of the following possibilities:

              • The boot-time code runs slower than the web interface, perhaps due to some sort of caching or optimization not being loaded yet.

              • The web interface code actually takes the same amount of time to run, but it does so in the background, and is transparent to the administrator since the user accounts show up in the list quickly and are usable.

              • The boot-time import code is doing more than the web interface code.</nextuid>

              1 Reply Last reply Reply Quote 0
              • B
                bullet92
                last edited by

                I had considered using the FreeRADIUS module, which probably has less boot-time overhead, but I didn't like that it stores the clear text passwords in the config file, so I went with the standard accounts

                From what i knwown freeradius supports encrypted users passwords. It's sufficient to select md5-password or what else instead of clear text password. Am I missing anything?

                1 Reply Last reply Reply Quote 0
                • G
                  guitarpicker
                  last edited by

                  @bullet92:

                  From what i knwown freeradius supports encrypted users passwords. It's sufficient to select md5-password or what else instead of clear text password. Am I missing anything?

                  FreeRADIUS does store them encrypted in the FreeRADIUS files when you choose the appropriate encryption.  However, pfSense stores the unencrypted one in the config.xml file, presumably so it can build the encrypted FreeRADIUS's tables on boot.

                  It's discussed in the FAQ, and I can attest that it is correct.
                  https://doc.pfsense.org/index.php/Why_are_some_passwords_stored_in_plaintext_in_config.xml

                  Sample:

                  		 <freeradius><config><sortable><varusersusername>importantdude</varusersusername>
                  				<varuserspassword>SuperSecretPassword</varuserspassword>
                  				<varuserspasswordencryption>MD5-Password</varuserspasswordencryption>
                  				...</sortable></config></freeradius> 
                  
                  
                  1 Reply Last reply Reply Quote 0
                  • B
                    bullet92
                    last edited by

                    I did not know this behavior  :P
                    Is possible an upgrade to SSD disk?
                    Sounds crazy a script that save users in a custom file, at reboot shutdown delete users from pfsense config (i think is unnecessary considering that u are adding this users manually is necessary because u will add further users from GUI, I presume) and then import after boot then from custom file? :o

                    1 Reply Last reply Reply Quote 0
                    • G
                      guitarpicker
                      last edited by

                      SSD disks are not a viable option for us, but would probably help a good bit.  This is a low-volume system which does not justify high-end hardware.

                      The custom import was a one-time action, and does the same thing as if the users were entered via the GUI.  After this import, all new users are being added via the GUI.  I am not running any custom scripts

                      The deletion happens at every boot is just how pfSense (and upstream m0n0wall) work - not by any sort of customization that I have done.  You can see for yourself in the source code for the local_sync_accounts function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L378) which gets called at boot up.  The general flow as seen in the comments is:

                      • Delete local users

                      • Delete local groups

                      • Sync (import) all local users

                      • Sync all local groups

                      This simple and rather foolproof method of synchronizing the user accounts with the configuration file could be optimized for performance, but the code would be immensely more complex to do so.  I think the lion's share of the delay is due to calling the local_user_set function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L450) for each user on each boot, which has a laundry list of things to do when setting up a new user.

                      It sounds like this isn't a high-demand feature.  I would much rather have an option to use FreeRADIUS without storing the plain text passwords than to spend a lot of time optimizing the local account sync process.  I don't think this is likely to happen either, since the whole reason they store the passwords is so that you can change the encryption type in FreeRADIUS without losing all the accounts.

                      Fortunately, pfSense has been rock solid and I haven't needed to reboot much.  I schedule it to reboot at night when I need to, so that the delay doesn't affect our users.

                      1 Reply Last reply Reply Quote 0
                      • N
                        NOYB
                        last edited by

                        On a very basic implementation with only a few accounts (less than 6) I randomly see delay at "Synchronizing user settings" during boot up.  Sometimes it takes maybe a minute, other times only a second or so.  It is inconsistent from one boot to the next.  USB Flash Drive Full Install with /tmp and /var ramdisk.

                        1 Reply Last reply Reply Quote 0
                        • B
                          bullet92
                          last edited by

                          @guitarpicker:

                          The custom import was a one-time action, and does the same thing as if the users were entered via the GUI.  After this import, all new users are being added via the GUI.  I am not running any custom scripts

                          Sorry i wasn't clear. I meant: what about a custom script that save user somwhere, download it at boot and then readd it (maybe in import-like mode)?
                          Firewall would be reacheble even without users and then wil readd it again. But u should be able to:
                          1. prevent pfsense to save itself the users
                          2. store them (local persistent HD or remote)
                          3. import them at startup (maybe the simpler things to do)
                          @guitarpicker:

                          The deletion happens at every boot is just how pfSense (and upstream m0n0wall) work - not by any sort of customization that I have done.  You can see for yourself in the source code for the local_sync_accounts function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L378) which gets called at boot up.  The general flow as seen in the comments is:

                          • Delete local users

                          • Delete local groups

                          • Sync (import) all local users

                          • Sync all local groups

                          This simple and rather foolproof method of synchronizing the user accounts with the configuration file could be optimized for performance, but the code would be immensely more complex to do so.  I think the lion's share of the delay is due to calling the local_user_set function (https://github.com/pfsense/pfsense/blob/f1551428c4fe708232fc80239ec207640b058a28/src/etc/inc/auth.inc#L450) for each user on each boot, which has a laundry list of things to do when setting up a new user.

                          It sounds like this isn't a high-demand feature.  I would much rather have an option to use FreeRADIUS without storing the plain text passwords than to spend a lot of time optimizing the local account sync process.  I don't think this is likely to happen either, since the whole reason they store the passwords is so that you can change the encryption type in FreeRADIUS without losing all the accounts.

                          Fortunately, pfSense has been rock solid and I haven't needed to reboot much.  I schedule it to reboot at night when I need to, so that the delay doesn't affect our users.

                          Looking at code it call system binary file to read and write users correctly(and set them the password):

                          $user_op = "useradd -m -k /etc/skel -o";

                          $cmd = "/usr/sbin/pw {$user_op} -q -u {$user_uid} -n {$user_name}".
                          " -g {$user_group} -s {$user_shell} -d {$user_home}".
                          " -c ".escapeshellarg($comment)." -H 0 2>&1";

                          Write users differently implicate a function that write "X" users directly to user file being careful to not corrupt this file.. it seems risky :D

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.