Don't upgrade to x86 CE build 2.4.0.r.20171008.0625
-
Updating to build 2.4.0.r.20171009.1758 worked like a charme.
pfSense is now running about 12 hours without showing any problems so I'd say "Good work and thank you" to the pfSense Team.
-
I'm also running good & stable on 2.4.1.a.20171009.2023 since last night. Thank you guys 8)
-
@kpa:
Edit: I realize now that the real problem was a bad dhcpd binary and the package system didn't include an updated dhcpd package, it actually did but since the version number hadn't changed pkg didn't offer it as an update and the only way to get the fixed dhcpd package was to use pkg install/upgrade -f.
Errr… Uh. So, WTH is the deal here?
$ cat /etc/version.buildtime Mon Oct 09 18:53:43 CDT 2017Enter an option: 13 >>> Updating repositories metadata... Updating Unofficial repository catalogue... Fetching meta.txz: . done Fetching packagesite.txz: . done Processing entries: .. done Unofficial repository update completed. 11 packages processed. Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Your packages are up to date# pkg upgrade -f Updating Unofficial repository catalogue... Fetching meta.txz: 100% 260 B 0.3kB/s 00:01 Fetching packagesite.txz: 100% 5 KiB 4.7kB/s 00:01 Processing entries: 100% Unofficial repository update completed. 11 packages processed. Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Checking for upgrades (236 candidates): 100% Processing candidates (236 candidates): 100% The following 236 package(s) will be affected (of 0 checked): Installed packages to be REINSTALLED: pkg-1.10.1_1 [pfSense] zip-3.0_1 [pfSense] xinetd-2.3.15_2 [pfSense] wrapalixresetbutton-0.0.7 [pfSense] wol-0.7.1_3 [pfSense] whois-5.2.17 [pfSense] webp-0.6.0_4 [pfSense] vstr-1.0.15_1 [pfSense] voucher-0.1_2 [pfSense] vnstat-1.15 [pfSense] unzoo-4.4_2 [pfSense] unbound-1.6.6 [pfSense] uclcmd-0.1_2 [pfSense] tiff-4.0.8 [pfSense] tftp-hpa-5.2 [pfSense] talloc-2.1.9 [pfSense] suricata-4.0.0 [pfSense] sudo-1.8.21p2 [pfSense] strongswan-5.6.0 [pfSense] sshlockout_pf-0.0.2 [pfSense] ssh_tunnel_shell-0.1_1 [pfSense] squidclamav-6.16 [pfSense] squid_radius_auth-1.10 [pfSense] squid-3.5.27 [pfSense] sqlite3-3.20.1_1 [pfSense] smartmontools-6.5_2 [pfSense] scponly-4.8.20110526_2 [pfSense] rrdtool-1.7.0_1 [pfSense] relayd-5.5.20140810_2 [pfSense] readline-7.0.3 [pfSense] rate-0.9_1 [pfSense] radvd-1.9.1 [pfSense] qstats-0.2 [pfSense] python27-2.7.14 [pfSense] python2-2_3 [pfSense] py27-speedtest-cli-1.0.6 [pfSense] py27-setuptools-36.2.2 [pfSense] postgresql95-client-9.5.9 [pfSense] png-1.6.34 [pfSense] pkgconf-1.3.7,1 [pfSense] php56-zlib-5.6.31 [pfSense] php56-xmlwriter-5.6.31 [pfSense] php56-xmlreader-5.6.31 [pfSense] php56-xml-5.6.31 [pfSense] php56-tokenizer-5.6.31 [pfSense] php56-sysvshm-5.6.31 [pfSense] php56-sysvsem-5.6.31 [pfSense] php56-sysvmsg-5.6.31 [pfSense] php56-sqlite3-5.6.31 [pfSense] php56-sockets-5.6.31 [pfSense] php56-simplexml-5.6.31 [pfSense] php56-shmop-5.6.31 [pfSense] php56-session-5.6.31 [pfSense] php56-readline-5.6.31 [pfSense] php56-posix-5.6.31 [pfSense] php56-pfSense-module-0.54 [pfSense] php56-pdo_sqlite-5.6.31 [pfSense] php56-pdo-5.6.31 [pfSense] php56-pcntl-5.6.31 [pfSense] php56-openssl-5.6.31 [pfSense] php56-opcache-5.6.31 [pfSense] php56-mcrypt-5.6.31 [pfSense] php56-mbstring-5.6.31_1 [pfSense] php56-ldap-5.6.31 [pfSense] php56-json-5.6.31 [pfSense] php56-hash-5.6.31 [pfSense] php56-gettext-5.6.31 [pfSense] php56-ftp-5.6.31 [pfSense] php56-filter-5.6.31 [pfSense] php56-dom-5.6.31 [pfSense] php56-curl-5.6.31 [pfSense] php56-ctype-5.6.31 [pfSense] php56-bz2-5.6.31 [pfSense] php56-bcmath-5.6.31 [pfSense] php56-5.6.31 [pfSense] php-xdebug-2.5.0 [pfSense] php-suhosin-0.9.38_3 [pfSense] pftop-0.7_8 [pfSense] pfSense-upgrade-0.33 [pfSense] pfSense-repo-2.4.1.a.20171009.1753 [pfSense] pfSense-rc-2.4.1.a.20171009.1853 [pfSense-core] pfSense-pkg-tftpd-0.1.3_1 [pfSense] pfSense-pkg-suricata-4.0.0_1 [pfSense] pfSense-pkg-sudo-0.3 [pfSense] pfSense-pkg-squid-0.4.39 [pfSense] pfSense-pkg-pfBlockerNG-2.1.1_10 [pfSense] pfSense-pkg-openvpn-client-export-1.4.14 [pfSense] pfSense-pkg-nmap-1.4.4_1 [pfSense] pfSense-pkg-mtr-nox11-0.85.6_1 [pfSense] pfSense-pkg-iftop-0.17_2 [pfSense] pfSense-pkg-freeradius3-0.15.1 [pfSense] pfSense-pkg-arping-1.2.2_1 [pfSense] pfSense-pkg-acme-0.1.19_1 [pfSense] pfSense-pkg-System_Patches-1.1.7 [pfSense] pfSense-pkg-Status_Traffic_Totals-1.2.1_1 [pfSense] pfSense-pkg-Shellcmd-1.0.3 [pfSense] pfSense-pkg-Service_Watchdog-1.8.4 [pfSense] pfSense-pkg-RRD_Summary-1.3.2_2 [pfSense] pfSense-pkg-Lightsquid-3.0.6_4 [pfSense] pfSense-pkg-FTP_Client_Proxy-0.3_3 [pfSense] pfSense-pkg-Cron-0.3.7_2 [pfSense] pfSense-pkg-AutoConfigBackup-1.47 [pfSense] pfSense-kernel-pfSense-2.4.1.a.20171009.1853 [pfSense-core] pfSense-default-config-serial-2.4.1.a.20171009.1853 [pfSense-core] pfSense-base-2.4.1.a.20171009.1853 [pfSense-core] pfSense-Status_Monitoring-1.7.2 [pfSense] pfSense-2.4.1.a.20171009.1753 [pfSense] perl5-5.24.3 [pfSense] pecl-zmq-1.1.3_2 [pfSense] pecl-ssh2-0-0.13 [pfSense] pecl-rrd1-1.1.3 [pfSense] pecl-radius-1.4.0.b1 [pfSense] pecl-intl-3.0.0_10 [pfSense] pear-XML_RPC2-1.1.3 [pfSense] pear-Net_URL2-2.2.1 [pfSense] pear-Net_Socket-1.0.14 [pfSense] pear-Net_SMTP-1.8.0 [pfSense] pear-Net_IPv6-1.3.0.b2_2 [pfSense] pear-Net_Growl-2.7.0 [pfSense] pear-Mail-1.4.1,1 [pfSense] pear-HTTP_Request2-2.3.0,1 [pfSense] pear-Crypt_CHAP-1.5.0 [pfSense] pear-Cache_Lite-1.7.16,1 [pfSense] pear-Auth_RADIUS-1.1.0 [pfSense] pear-1.10.5 [pfSense] pcre-8.40_1 [pfSense] p7zip-16.02 [pfSense] p5-HTML-Tagset-3.20_1 [pfSense] p5-HTML-Parser-3.72 [pfSense] p5-GD-2.66 [pfSense] p5-ExtUtils-PkgConfig-1.16 [pfSense] p5-CGI-4.36 [pfSense] openvpn-client-export-2.4.4 [pfSense] openvpn-2.4.4 [pfSense] openldap-client-2.4.45 [pfSense] oniguruma6-6.4.0 [pfSense] ntp-4.2.8p10_2 [pfSense] nss-3.33_1 [pfSense] nspr-4.17 [pfSense] norm-1.5r6 [pfSense] nmap-7.40_1 [pfSense] nginx-1.12.1_1,2 [pfSense] nettle-3.3 [pfSense] nano-2.8.7 [pfSense] mysql56-client-5.6.37_1 [pfSense] mtr-nox11-0.92 [pfSense] mpd5-5.8_2 [pfSense] mobile-broadband-provider-info-20170310 [pfSense] miniupnpd-1.9.20160113,1 [pfSense] minicron-0.0.2 [pfSense] lzo2-2.10_1 [pfSense] luajit-2.0.5 [pfSense] lua52-5.2.4 [pfSense] links-2.13,1 [pfSense] lighttpd-1.4.45_1 [pfSense] lightsquid-1.8_5 [pfSense] libzmq4-4.2.2 [pfSense] libyaml-0.1.6_2 [pfSense] libxml2-2.9.4 [pfSense] libunistring-0.9.7 [pfSense] libucl-0.8.0 [pfSense] libssh2-1.8.0,3 [pfSense] libpcap-1.8.1 [pfSense] libnghttp2-1.26.0 [pfSense] libnet-1.1.6_5,1 [pfSense] libmcrypt-2.5.8_3 [pfSense] liblz4-1.8.0,1 [pfSense] libltdl-2.4.6 [pfSense] libidn2-2.0.4 [pfSense] libiconv-1.14_11 [pfSense] libhtp-0.5.25 [pfSense] libgd-2.2.4_1,1 [pfSense] libffi-3.2.1_1 [pfSense] libevent-2.1.8 [pfSense] libedit-3.1.20170329_2,1 [pfSense] libdaemon-0.14_1 [pfSense] lha-1.14i_7 [pfSense] ldns-1.7.0_1 [pfSense] krb5-1.15.2 [pfSense] json-c-0.12.1 [pfSense] jpeg-turbo-1.5.2 [pfSense] jbigkit-2.1_1 [pfSense] jansson-2.10 [pfSense] isc-dhcp43-server-4.3.6_1 [pfSense] isc-dhcp43-relay-4.3.6 [pfSense] isc-dhcp43-client-4.3.6 [pfSense] ipmitool-1.8.18_1 [pfSense] indexinfo-0.2.6 [pfSense] igmpproxy-0.1_5,1 [pfSense] iftop-1.0.p4 [pfSense] idnkit-1.0_6 [pfSense] icu-59.1,1 [pfSense] hyperscan-4.5.2 [pfSense] hiredis-0.13.3 [pfSense] grepcidr-2.0 [pfSense] gmp-6.1.2 [pfSense] glib-2.50.2_6,1 [pfSense] giflib-5.1.4 [pfSense] gettext-runtime-0.19.8.1_1 [pfSense] gdbm-1.13_1 [pfSense] freetype2-2.8 [pfSense] freeradius3-3.0.15_1 [pfSense] filterlog-0.1_5 [pfSense] filterdns-1.0_16 [pfSense] expiretable-0.6_1 [pfSense] expat-2.2.1 [pfSense] dpinger-2.0_1 [pfSense] dnsmasq-2.78,1 [pfSense] dmidecode-3.1_1 [pfSense] dhcpleases6-0.1_2 [pfSense] dhcpleases-0.3_1 [pfSense] dhcp6-20080615.2 [pfSense] cyrus-sasl-2.1.26_12 [pfSense] curl-7.56.0 [pfSense] cpustats-0.1_1 [pfSense] cpdup-1.18 [pfSense] clog-1.0.1_1 [pfSense] clamav-0.99.2_5 [pfSense] choparp-20150613 [pfSense] check_reload_status-0.0.8 [pfSense] ca_root_nss-3.32.1 [pfSense] c-icap-modules-0.4.5 [pfSense] c-icap-0.4.4,2 [pfSense] bwi-firmware-kmod-3.130.20 [pfSense] bsnmp-ucd-0.4.2 [pfSense] bsnmp-regex-0.6_1 [pfSense] broccoli-1.97,1 [pfSense] bind-tools-9.11.2 [pfSense] beep-1.0_1 [pfSense] bash-4.4.12_2 [pfSense] barnyard2-1.13_1 [pfSense] ataidle-2.7.2 [pfSense] arping-2.15_1 [pfSense] arc-5.21p [pfSense] aggregate-1.6_1 [pfSense] GeoIP-1.6.11 [pfSense] Number of packages to be reinstalled: 236 The process will require 2 MiB more space. 102 MiB to be downloaded.??? ::) :o
Is there any way to get to consistent system state when upgrading after this grand screw-up, or should I just reinstall altogether?
-
Erm… I have a strong feeling that the pre-kaboom snapshots should be taken, released as 2.4.0 and this insane messing should be moved to 2.4.1. WTH is anyone doing changes like this days before release, after months and months of beta testing.
It's not quite that simple, but in a vague sort of way that's what we did.
The change made was a small change to a data structure size meant to fix an interface issue. This caused older binaries for services and other programs to fail on an updated kernel, which left the system in an inconsistent state and likely without connectivity. The packages were recompiled with the new structures, but their version numbers didn't change, so pkg did not know it had to reinstall all affected packages. New installs made during this time were OK, and anyone who forced all packages to reinstall would have been OK as well, because in those cases the firewall had a fully consistent set of binaries for kernel, world, and packages.
We ultimately decided to take down those problem snapshots and back out that change, so that people who had not yet upgraded to a problem snapshot would be safe from the update issue. After reverting that change and rebuilding all of the package sets, then testing internally, we turned everything back on. We tried a few alternate solutions such as forcing pkg to update everything on upgrades between certain timestamps but we were not satisfied with any of the workarounds.
For those who did not update during the problem time, updating to a current snapshot is safe.
If you updated a firewall to a broken snapshot and fixed it manually by using "pkg upgrade -f", or if you made a fresh install from a problem image, you will need to manually update to the latest set of packages using "pkg upgrade -f" to forcefully reinstall everything. Otherwise, the firewall will end up in an inconsistent broken state again. Alternately, reinstall from a new snapshot.
Unrelated to that problem, we also had an issue with logging late last week. A problem with the clog program generated log files in an invalid format, leading to a variety of error message and a lack of logging data. The problem has been corrected on current snapshots, but if a firewall had generated new log files during the problematic period, such as on a fresh installation or after resetting/clearing logs, then those log files must be reset.
To reset the log files, navigate to Status > System Logs, Settings tab. Click "Reset Log Files", then OK to confirm.
At the moment all of the current snapshots should be OK and trouble-free.
-
@doktonotor The problem is the pkg consider only the version numbers when it looks for updates, if you did 'pkg upgrade' after the repos had been rebuilt you still wouldn't get the rebuilt dhcpd package because the version number had stayed the same. Only 'pkg install -f isc-dhcpd43-server' (or just 'pkg upgrade -f' to reinstall everything) gave you the updated package because that forced a refetch and reinstall. This is not really a fault of pkg because it just does what it's told, the real fault is the version numbering used in FreeBSD ports that doesn't allow the packages to have their version numbers upped if the repository has to be rebuilt because of API breakage as was the case here. Yes you could up them in the port Makefiles manually but that's a real mess and should be avoided.
-
Eh, I'll just reinstall the bunch of affected testing boxes. I don't really trust any pkg magic to make things consistent.
-
Hmm. I get these errors:
Enter an option: 13 pkg: Unable to determine ABI pkg: Cannot parse configuration file! pkg: Unable to determine ABI pkg: Cannot parse configuration file! ERROR: It was not possible to identify which pfSense kernel is installed *** Welcome to pfSense 2.4.0-RC (amd64) on firewall ***pkg upgrade -f Child process pid=27587 terminated abnormally: Segmentation faultIs there anything I can do, except complete reinstall?
-
pkg upgrade -f Child process pid=27587 terminated abnormally: Segmentation faultTry using "pkg-static" instead of pkg, but from the sound of it, a reinstall would be safer, faster, and less problematic.
-
While talking about pkg-static, isn't that something that should be preferably used on upgrades in general? The flood of console hints about using it on upgrading from 2.3.x would suggest it to be the case.
-
While talking about pkg-static, isn't that something that should be preferably used on upgrades in general? The flood of console hints about using it on upgrading from 2.3.x would suggest it to be the case.
It depends on the situation and circumstances, usually it would be used in a case when there could be a library mismatch (or missing entirely) in something pkg wants, or just to be safe.
-
Try using "pkg-static" instead of pkg, but from the sound of it, a reinstall would be safer, faster, and less problematic.
pkg-static didn't help either. Complete reinstall succeeded. Thanks anyway!
