24.03-RC install long delays
-
Repliacted it and opened a bug:
-
@stephenw10 said in 24.03-RC install long delays:
Repliacted it and opened a bug:
Thanks, Steve! This explains it perfectly.
-
I have said it before and will say it again: Those updates for blocklists, DNSBL Feeds, Rule Sets, in short everything pulling from outside sources shouldn't be part of the upgrade process to begin with.
-
@Bob-Dig said in 24.03-RC install long delays:
shouldn't be part of the upgrade process to begin with
I've read somewhere in the past : "Before a pfSense upgrade, remove packages".
I'm not doing that. Most of us don't do that, I guess.
But it would accelerate upgrading for sure. -
That's the safest way to be sure. But it shouldn't normally be required.
-
@Bob-Dig The fetching of remote content are a step during the installation of the package. During an upgrade, the old package versions must be removed and the new package versions installed. This is because the old package binaries are linked against an older set of libraries that might not be present on the upgraded system. For example when the FreeBSD major version is updated as part of the pfSense upgrade.
The pfSense upgrade process isn't explicitly requesting the blocklists, DNSBL Feeds, Rule Sets, etc. Its just that when each package is installed with the prior configuration, it attempts to fetch the remote content required by the configuration.
Leaving the prior packages installed isn't an option. The question is when should packages be re-installed after a pfSense upgrade and the prior configuration is applied. It used to be that pfSense would upgrade, bring the network interfaces online and then begin to apply packages. The downside of this approach is that during the window after the network interfaces are active and the packages are fully re-installed the security posture of the router is incomplete. During that time there is an opportunity for unwanted traffic to pass.
The new Boot Environment upgrade process installs the packages before the network interfaces are configured. The theoretical benefit is that the router won't start with missing functionality. However, when the packages require fetching content as part of the installation this fails. If a security package such as snort, pfblockerng, suricata, etc is installed but doesn't obtain the required remote data, then not only will it fail to function properly when the network interfaces are brought up but its unclear how long it will be before the required data is fetched. For example I believe snort updates every six hours by default.
Perhaps there is a middle ground. @stephenw10, can the WAN interfaces be configured and brought online without the LAN interfaces and then perform the package installation? Doing so would permit the remote content to be obtained without allowing traffic to pass through the router until all of the packages are fully configured.
-
@jaltman Very good points and I like your last suggestion. In my case, I did see that there was a Suricata package update available right before upgrading from the latest 24.03 beta to the RC, but I know it's recommended not to upgrade the packages until the main OS is updated.
From this point forward, if I see that there is a package update and a pfSense update, I'll probably perform that upgrade from the console so I can see the entire upgrade process which is normally done in just a few minutes. I was freaking out when I couldn't get in via SSH or HTTPS after 10 minutes.
-
@jaltman said in 24.03-RC install long delays:
it attempts to fetch the remote content required by the configuration.
And that is problematic, not even related to the problem in this thread.
-
Yes it's an issue anyway beyond the delays noted here, I agree.
-
@stephenw10 said in 24.03-RC install long delays:
Yes it's an issue anyway beyond the delays noted here, I agree.
For instance, ISP "Deutsche Telekom" (known in the US as "T-Mobile", there they are playing the good guy) was blocking some AWS services. And by blocking I mean, they let you start a download but never finish it, it will go on forever.
Now I was a new customer of them, not knowing what they are up to, and was upgrading my pfSense installation. And it always failed or better never finished.
Took me some days to figure this out. I think I connected to my neighbors WiFi to get around this...
Conclusion, only upgrade from content that netgate is hosting or has somewhat under their control and do not pull from everywhere else where some "feed" is hosted.In hindsight I could just upgraded without the packages but I never needed that before so ...
-
@stephenw10 Here is the output from the update to 24.03.r.20240416.0005. The timeout for each download operation is approximately 45 seconds.
Welcome to Netgate pfSense Plus 24.03-RC... Checking dump device /dev/nda0p3 for crash dumps ... no crash dumps on /dev/nda0p3. ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/loca l/lib/ipsec /usr/local/lib/perl5/5.36/mach/CORE 32-bit compatibility ldconfig path: done. 3450 External config loader 1.0 is now starting... nda0p1 nda0p2 nda0p4 Launching the init system...Updating CPU Microcode... CPU: Intel(R) Atom(TM) CPU C3338R @ 1.80GHz (1800.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x506f1 Family=0x6 Model=0x5f Stepping=1 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2 ,SS,HTT,TM,PBE> Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT, TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA> Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> IA32_ARCH_CAPS=0xc69<RDCL_NO,SKIP_L1DFL_VME,MDS_NO> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics Done. done. Initializing.................... done. Starting device manager (devd)...ichsmb0: <Intel Denverton SMBus controller> port 0x2000-0x201f mem 0x816d8000-0x816d80ff at dev ice 31.4 on pci0 smbus0: <System Management Bus> on ichsmb0 done. Loading configuration....done. Updating configuration...done. Loading cryptographic accelerator drivers...qat0: <Intel c3xxx QuickAssist> mem 0x81500000-0x8153ffff,0x81540000-0x8157ffff at d evice 0.0 on pci1 qat0: qat_dev0 started 6 acceleration engines qat0: FW version: 4.18.0 qat0: Excessive clock measure delay qat_ocf0: <QAT engine> done. Setting up extended sysctls...done. Executing deferred package installation scripts...Running last steps of acme installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()...done. Menu items... done. Writing configuration... done. Running last steps of Avahi installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_resync_config_command()...done. Menu items... done. Services... done. Writing configuration... done. Running last steps of aws-wizard installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Custom commands... Menu items... done. Writing configuration... done. Running last steps of iperf installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Custom commands... Menu items... done. Services... done. Writing configuration... done. Running last steps of ipsec-profile-wizard installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Custom commands... Menu items... done. Writing configuration... done. Running last steps of mailreport installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Custom commands... Menu items... done. Writing configuration... done. Running last steps of Netgate_Firmware_Upgrade installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Menu items... done. Writing configuration... done. Running last steps of pfBlockerNG installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()... Rebuilding GeoIP tabs... done. Creating Firewall filter service... done. Renew Firewall filter executables... done. Starting Firewall filter Service... done. Creating DNSBL service... done. Renew DNSBL lighttpd executable... done. Creating DNSBL web server config ... done. Creating DNSBL Certificate... done. Starting DNSBL Service... done. Upgrading previous settings: Adv. Inbound firewall rule settings... no changes required ... done. OpenVPN/IPSec interface selections... no changes required ... done. Proofpoint/ET IQRisk settings... no changes required ... done. General Tab -> IP Tab settings... no changes required ... done. pfBlockerNGSuppress Alias -> IPv4 Suppression Customlist... no changes required ... done. Upgrading previous EasyLists to new format... no changes required ... done. Upgrading previous Firefox DoH to new format... no changes required ... done. MaxMind License Key configuration setting... no changes required ... done. Validating Widget cron settings... no changes required ... done. Upgrading... done Custom commands completed ... done. Executing custom_php_resync_config_command()...igc0: link state changed to UP igc3: link state changed to UP igc0: link state changed to DOWN 2024-04-16T04:55:16.268790-04:00 - php-fpm 575 - - /rc.linkup: Ignoring link event during boot sequence. 2024-04-16T04:55:16.272607-04:00 - php-fpm 574 - - /rc.linkup: DHCP Client not running on wan (igc3), reconfiguring dhclient. igc3: link state changed to DOWN 2024-04-16T04:55:16.371586-04:00 - php-fpm 61180 - - /rc.linkup: Ignoring link event during boot sequence. done. Menu items... done. Services... done. Writing configuration... done. Running last steps of snort installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()...Saved settings detected. Migrating settings to new configuration... done. Downloading configured rule sets. This may take some time... 2024-04-16T04:55:17.358347-04:00 - php-fpm 575 - - /rc.linkup: DHCP Client not running on wan (igc3), reconfiguring dhclient. 2024-04-16T04:55:17.368212-04:00 - php-fpm 575 - - /rc.linkup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var /run/dhclient.igc3.pid igc3 > /tmp/igc3_output 2> /tmp/igc3_error_output' returned exit code '1', the output was '' igc0: link state changed to UP 2024-04-16T04:55:20.210415-04:00 - php-fpm 575 - - /rc.linkup: Ignoring link event during boot sequence. igc3: link state changed to UP Downloading Snort Subscriber rules md5 file... FAILED! Snort Subscriber rules md5 error ... Server returned error code 0 ... Snort Subscriber rules will not be updated. Server returned error code 0. Downloading Snort OpenAppID detectors md5 file... FAILED! Snort OpenAppID detectors md5 error ... Server returned error code 0 ... Snort OpenAppID detectors will not be updated. Server returned error code 0. Downloading Snort AppID Open Text Rules md5 file... FAILED! Snort AppID Open Text Rules md5 error ... Server returned error code 0 ... Snort AppID Open Text Rules will not be updated. Server returned error code 0. Downloading Snort GPLv2 Community Rules md5 file... FAILED! Snort GPLv2 Community Rules md5 error ... Server returned error code 0 ... Snort GPLv2 Community Rules will not be updated. Server returned error code 0. Downloading Emerging Threats Open rules md5 file... FAILED! Emerging Threats Open rules md5 error ... Server returned error code 0 ... Emerging Threats Open rules will not be updated. Server returned error code 0. Downloading Feodo Tracker Botnet C2 IP rules file...Feodo Tracker Botnet C2 IP rules file download failed! Cleaning up temp dirs and files... done. The Rules update has finished. Generating snort.conf configuration file from saved settings. Generating configuration for WAN...route: route has not been found route: route has not been found done. Generating snort.sh script in /usr/local/etc/rc.d/... done. Finished rebuilding Snort configuration files. done. Executing custom_php_resync_config_command()...route: route has not been found route: route has not been found done. Menu items... done. Services... done. Writing configuration... done. Please visit Services - Snort - Interfaces tab first and select your desired rules. Afterwards visit the Updates tab to download your configured rulesets.Running last steps of System_Patches installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()...done. Menu items... done. Writing configuration... done. done. Executing early shell commands...done. coretemp0: <CPU On-Die Thermal Sensors> on cpu0 Setting timezone...done. Configuring looplo0: link state changed to UP back interface...done. Starting syslog...done. Setting up interfaces microcode...done. Configuring loopback interface...done. Configuring LAN interface...igc0: link state changed to DOWN done. Configuring WAN interface...done. Configuring CARP settings...done. Syncing OpenVPN settings...done. Configuring firewall......done. Starting PFLOG...done. Setting up gateway monitors...done. Setting up static routes...route: message indicates error: Invalid argument done. Setting up DNSs... Starting DNS Resolver...done. Synchronizing user settings...done. Configuring CRON...done. Bootstrapping clock...done. Starting NTP Server...done. Starting webConfigurator...done. Starting DHCP service...done. Starting DHCPv6 service...done. Configuring firewall......done. Generating RRD graphs...done. Starting UPnP service... done. Starting syslog...done. Starting CRON... done. Starting package AWS VPC Wizard...done. Starting package IPsec Profile Wizard...done. Starting package acme...done. Starting package iperf...done. Starting package Avahi...done. Starting package System Patches...done. Starting package Netgate Firmware Upgrade...done. Starting package pfBlockerNG...done. Starting package mailreport...done. Starting package snort...done. Starting /usr/local/etc/rc.d/pfb_dnsbl.sh...done. Starting /usr/local/etc/rc.d/pfb_filter.sh...done. Netgate pfSense Plus 24.03-RC amd64 20240416-0005 Bootup complete Performing automatic boot verification...done. -
During upgrading :
your igc3, your WAN interface (?) starts going up-down-up-.....
IMHO : No so perfect timingAnd, AFAIK, because pfSense is in the installing+booting sequence, interface events are ignored (as shown) ....
so packages that need to reach outside for extra info will time out = making the upgrade/boot way longer.It's probably an idea : before the next upgrade, de activate packages that need the access like snort.
Also : was it pfSense rebuilding the igc3 = WAN ? Or the device connected to the other end ?
-
My 6100 MAX had a similar timeout again on today's build about 3 hours ago. I've installed every single build throughput the entire 24.03 beta cycle and this issue did not begin until the first RC last week. No packages were upgraded this time either.
-
It's something we're looking at for 24.07 at this point. Since it doesn't actually prevent upgrading and 24.03 is overdue.
-
@stephenw10 said in 24.03-RC install long delays:
It's something we're looking at for 24.07 at this point. Since it doesn't actually prevent upgrading and 24.03 is overdue.
Understood. I wonder what changed between the last beta and the first RC build?
-
@stephenw10 I request that the possibility of long delays be documented as part of the upgrade notes. At least with snort and pfBlockerNG, users should verify the state of the feeds after the pfSense upgrade has completed.
Thanks.
-
@jaltman This is what is confusing. These long delays did not begin for me until the first RC last week. All of the other 24.03 development builds had less than a 2 min reboot time during the upgrade process (with the same packages installed).
-
Hmm, probably because it's a different branch which forces a pkg reinstall. Though that would imply you should not see it coming from last weeks RC.
-
@stephenw10 said in 24.03-RC install long delays:
Hmm, probably because it's a different branch which forces a pkg reinstall. Though that would imply you should not see it coming from last weeks RC.
There is no package re-install indicated in my console output from this morning. However, it does "Executing deferred package installation scripts" and when it gets to snort it:
- overwrites the previous configuration
- executes custom_php_install_command()
- notices that the settings were saved
- migrates the settings to the new configuration
- which triggers the downloading of rule sets
All of this occurs before the interfaces have been configured.
A related question is why does it take so long for the https connections to time out. Why aren't the underlying tcp/ip connections failing immediately when there is no route?
-
Because in the new ZFS/BE upgrade system all the packages are installed/updated into the new BE before rebooting. That reduces the down time significantly for most installs. But it does mean that packages with installs scripts end up being run differently as we've seen. There's a good chance this can be patched before 24.07 IMO.
