24.03-RC install long delays
-
@jaltman said in 24.03-RC install long delays:
Upgrading from the prior beta to 24.03-RC on a 6100 MAX resulted in delays of close to 8 minutes when installing from the UI. Eventually I went downstairs to connect a terminal to the serial console and caught messages from failed attempts to download snort and pfBlockerNG updates.
The downloads are occurring before the WAN is configured so that isn't going to work.
I literally said the same EXACT thing about three or four hours ago on the RC thread. Same device too. As soon as I brought my laptop down with the console cable, I could see that Suricata was still updating…. Whew!
-
I'm used to the packages being reinstalled after the pfSense UI is available. In this case the packages were reinstalled much earlier. Here is the output I was able to capture on the console.
FAILED! Emerging Threats Open rules md5 error ... Server returned error code 0 ... Emerging Threats Open rules will not be updated. Server returned error code 0. Downloading Feodo Tracker Botnet C2 IP rules file... Feodo Tracker Botnet C2 IP rules file download failed! Cleaning up temp dirs and files... done. The Rules update has finished. Generating snort.conf configuration file from saved settings. Generating configuration for WAN...route: route has not been found route: route has not been found done. Generating snort.sh script in /usr/local/etc/rc.d/... done. Finished rebuilding Snort configuration files. done. Executing custom_php_resync_config_command()...route: route has not been found route: route has not been found done. Menu items... done. Services... done. Writing configuration... done. Please visit Services - Snort - Interfaces tab first and select your desired rules. Afterwards visit the Updates tab to download your configured rulesets.Running last steps of System_Patches installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()...done. Menu items... done. Writing configuration... done. done. Executing early shell commands...done. coretemp0: <CPU On-Die Thermal Sensors> on cpu0 Setting timezone...done. Configuring looplo0: link state changed to UP back interface...done. Starting syslog...done. Setting up interfaces microcode...done. Configuring loopback interface...done. Configuring LAN interface...igc0: link state changed to DOWN done. Configuring WAN interface...done. Configuring CARP settings...done. Syncing OpenVPN settings...done. Configuring firewall......done. Starting PFLOG...done. Setting up gateway monitors...done. Setting up static routes...route: message indicates error: Invalid argument done. Setting up DNSs... Starting DNS Resolver...done. Synchronizing user settings...done. Configuring CRON...done. Bootstrapping clock...done. Starting NTP Server...done. Starting webConfigurator...done. Starting DHCP service...done. Starting DHCPv6 service...done. Configuring firewall......done. Generating RRD graphs...done. Starting UPnP service... done. Starting syslog...done. Starting CRON... done. Starting package AWS VPC Wizard...done. Starting package IPsec Profile Wizard...done. Starting package acme...done. Starting package iperf...done. Starting package Avahi...done. Starting package System Patches...done. Starting package Netgate Firmware Upgrade...done. Starting package pfBlockerNG...done. Starting package mailreport...done. Starting package snort...done. Starting /usr/local/etc/rc.d/pfb_dnsbl.sh...done. Starting /usr/local/etc/rc.d/pfb_filter.sh...done. Netgate pfSense Plus 24.03-RC amd64 20240410-1729 Bootup complete Performing automatic boot verification...done. -
@jaltman said in 24.03-RC install long delays:
I'm used to the packages being reinstalled after the pfSense UI is available. In this case the packages were reinstalled much earlier. Here is the output I was able to capture on the console.
FAILED! Emerging Threats Open rules md5 error ... Server returned error code 0 ... Emerging Threats Open rules will not be updated. Server returned error code 0. Downloading Feodo Tracker Botnet C2 IP rules file... Feodo Tracker Botnet C2 IP rules file download failed! Cleaning up temp dirs and files... done. The Rules update has finished. Generating snort.conf configuration file from saved settings. Generating configuration for WAN...route: route has not been found route: route has not been found done. Generating snort.sh script in /usr/local/etc/rc.d/... done. Finished rebuilding Snort configuration files. done. Executing custom_php_resync_config_command()...route: route has not been found route: route has not been found done. Menu items... done. Services... done. Writing configuration... done. Please visit Services - Snort - Interfaces tab first and select your desired rules. Afterwards visit the Updates tab to download your configured rulesets.Running last steps of System_Patches installation. Saving updated package information... overwrite! Loading package configuration... done. Configuring package components... Loading package instructions... Custom commands... Executing custom_php_install_command()...done. Menu items... done. Writing configuration... done. done. Executing early shell commands...done. coretemp0: <CPU On-Die Thermal Sensors> on cpu0 Setting timezone...done. Configuring looplo0: link state changed to UP back interface...done. Starting syslog...done. Setting up interfaces microcode...done. Configuring loopback interface...done. Configuring LAN interface...igc0: link state changed to DOWN done. Configuring WAN interface...done. Configuring CARP settings...done. Syncing OpenVPN settings...done. Configuring firewall......done. Starting PFLOG...done. Setting up gateway monitors...done. Setting up static routes...route: message indicates error: Invalid argument done. Setting up DNSs... Starting DNS Resolver...done. Synchronizing user settings...done. Configuring CRON...done. Bootstrapping clock...done. Starting NTP Server...done. Starting webConfigurator...done. Starting DHCP service...done. Starting DHCPv6 service...done. Configuring firewall......done. Generating RRD graphs...done. Starting UPnP service... done. Starting syslog...done. Starting CRON... done. Starting package AWS VPC Wizard...done. Starting package IPsec Profile Wizard...done. Starting package acme...done. Starting package iperf...done. Starting package Avahi...done. Starting package System Patches...done. Starting package Netgate Firmware Upgrade...done. Starting package pfBlockerNG...done. Starting package mailreport...done. Starting package snort...done. Starting /usr/local/etc/rc.d/pfb_dnsbl.sh...done. Starting /usr/local/etc/rc.d/pfb_filter.sh...done. Netgate pfSense Plus 24.03-RC amd64 20240410-1729 Bootup complete Performing automatic boot verification...done.I only caught the very end of the console as it was spinning up the interfaces by the time I plugged into my 6100 MAX. Each 24.03 build up until today only took a few minutes. Interestingly enough, I didn’t get a boot verification prompt either which well beyond the 300 seconds I have defined unless I somehow did it via the console connection as I was tapping the enter key trying to establish a connection.. Everything is working perfectly fine now though.
-
D DefenderLLC referenced this topic on
-
Yeah this requires investigation. We had seen delays from those packages but not like that. Digging....
-
@stephenw10 Would it help if I switched back to the last beta boot environment and re-ran the upgrade but this time from the console?
-
Yes a full console log showing exactly where it is triggered should narrow things down.
We should be able to replicate that here though.
-
@stephenw10 If you haven't replicated it by late this evening EDT, I can repeat the process then.
-
@jaltman said in 24.03-RC install long delays:
@stephenw10 If you haven't replicated it by late this evening EDT, I can repeat the process then.
Did you get the boot verification post-upgrade? Mine never prompted me despite every previous beta install. My 300 seconds was definitely exceeded by the time it took for this RC install.
-
@DefenderLLC The last three lines of my console output were
Netgate pfSense Plus 24.03-RC amd64 20240410-1729 Bootup complete Performing automatic boot verification...done. -
@jaltman said in 24.03-RC install long delays:
@DefenderLLC The last three lines of my console output were
Netgate pfSense Plus 24.03-RC amd64 20240410-1729 Bootup complete Performing automatic boot verification...done.I believe that's what mine said too. Just wondering if you were ever prompted in the GUI post-upgrade. I'm guessing that you weren't.
I was not prompted his time.
-
@DefenderLLC I was not prompted
-
To be clear you would only expect to see it if manual verification is set and it isn't by default.
-
@stephenw10 I do not have manual verification enabled
-
@stephenw10 Mine was already enabled for 300 seconds and I was never prompted in the GUI unlike the other 24.03 betas. As I mentioned on the other thread, I could have inadvertently verified it via console (if that's even possible) when I was frantically trying to establish a USB connection.
-
@DefenderLLC I appreciate that you have a different problem than I observed. I would appreciate it if discussion of that problem were held in a separate topic. Thanks.
-
@jaltman said in 24.03-RC install long delays:
@DefenderLLC I appreciate that you have a different problem than I observed. I would appreciate it if discussion of that problem were held in a separate topic. Thanks.
For the record, I also had the same problem as you. 10 minutes install on my 6100 MAX due to package failures. The main difference between our issues is Snort vs Suricata. They both appeared to be exhibiting similar issues during the upgrade.
-
@DefenderLLC Discussion of the delay problem is on topic. Discussion of manual boot verification is not. They are unrelated and hijacking my topic will make it more difficult for Netgate to obtain the necessary details to identify and fix the ordering of the package update process during a pfSense upgrade.
Please create a separate topic for "24.03-RC Manual Boot Verification failure" or something.
Thanks.
-
@jaltman But they MAY be related. I only brought that up to see if you were also not prompted for verification which is probably related to the unusually long install time. The last statement in console log support this theory.
Didn't mean to upset you, but we both have the same exact device experiencing the same exact issues.
-
Repliacted it and opened a bug:
-
@stephenw10 said in 24.03-RC install long delays:
Repliacted it and opened a bug:
Thanks, Steve! This explains it perfectly.
