Don't upgrade to x86 CE build 2.4.0.r.20171008.0625
-
In this next sentence I say Something witty in Klingon about ZFS boot environments in pfSense 2.4+ as a mitigating factor for contingencies such as this:
"ZFS nIvbogh be'vam SUPERBORK SeH 'oH alows SoH ghaH nom RECOVER."
That was one really nice thing about Nano.
Will be a snapshots a thing in pfSense? Wold be great a auto snapshot before upgrade option, maybe. Config recovery do the job beautifully.
-
@mais_um:
Will be a snapshots a thing in pfSense? Wold be great a auto snapshot before upgrade option, maybe. Config recovery do the job beautifully.
It's a ZFS thing, and ZFS is now an install option in pfSense 2.4.
AFAIK, you can't recover from ZFS snapshots in pfSense just yet. I'm not sure where that is on the road-map, but if I were King/President/Jesus, I'd make it a high priority.
I've been anticipating ZFS in pfSense for years and am still giddy when I think about a ZPool vDEV made of 2 flash DOMs. https://twitter.com/karlfife/status/878833005426561024
FreeNAS has been doing GUI-integrated ZFS boot environments for a while, and the feature has saved my bacon more than once (especially in remote installs without remote technical hands). As long as the system comes back up after an update/upgrade, you can pick a boot snapshot (via GUI/SSH–no IPMI needed).
Yes, snapshots are auto-generated at time of update, and (of course) they each contain a private copy of the config (e.g. usable, non-migrated).
-
I just applied the newly available 10/9 build. It seems to have fixed my broken install.
That is, my 2.4 remote instance was non-functional, notably had a DHCP server that wouldn't start, but the borked instance retained WAN connectivity. Just now, it called home to the update servers, which pushed the 10/9 build, then rebooted, bringing the network back to the land of the living. Nice work gentlemen.
-
Too late, totally messed me up. Took about 4 hours to figure out what happened and get things fixed and going again. :(
-
Installed the latest update. Seems okay so far.
-
But dhcpd v4 not working:
Oct 10 06:21:15 dhcpd Can't attach interface bridge0 to bpf device /dev/bpf0: Invalid argument -
But dhcpd v4 not working:
Oct 10 06:21:15 dhcpd Can't attach interface bridge0 to bpf device /dev/bpf0: Invalid argumentI'm not seeing that error or any other malfunction of dhcpd.
-
Manual upgrade "pkg upgrade -f isc-dhcp43-server-4.3.6_1" and clean chroot env. /var/dhcpd resolve it.
-
Fresh install of the new version and restore from old config.xml has fixed everything.
-
Did a fresh install and restored my old config and its all working fine now.
-
Manual upgrade "pkg upgrade -f isc-dhcp43-server-4.3.6_1" and clean chroot env. /var/dhcpd resolve it.
That did the trick for me as well.
For developers, I know it's hard to keep up with the changes but this is a case where you could improve greatly. It should not be possible to upgrade your system and be left with an old version of a chroot environment that has the wrong device nodes or other files, every upgrade should clean up expendable content so that the services are started with a clean slate after an upgrade.Edit: I realize now that the real problem was a bad dhcpd binary and the package system didn't include an updated dhcpd package, it actually did but since the version number hadn't changed pkg didn't offer it as an update and the only way to get the fixed dhcpd package was to use pkg install/upgrade -f.
You could investigate a possiblity of including a build number into your package version numbers. It's not afaik supported directly by poudriere which assumes that the version numbers are what's in the ports tree and nothing else. A build number would solve a lot of problems similar to this and enable a forced reinstall of all packages regarless of what the version numbers are in the ports tree.
-
Installed the latest update. Seems okay so far.
Logos', my pfSense logos' are screwed up on both my units, not critical but visually obvious. :)
-
Erm… I have a strong feeling that the pre-kaboom snapshots should be taken, released as 2.4.0 and this insane messing should be moved to 2.4.1. WTH is anyone doing changes like this days before release, after months and months of beta testing.
-
Erm… I have a strong feeling that the pre-kaboom snapshots should be taken, released as 2.4.0 and this insane messing should be moved to 2.4.1. WTH is anyone doing changes like this days before release, after months and months of beta testing.
-
@marjohn56:
Installed the latest update. Seems okay so far.
Logos', my pfSense logos' are screwed up on both my units, not critical but visually obvious. :)
Refresh browser cache, fixed the broken logo for me.
-
@marjohn56:
Installed the latest update. Seems okay so far.
Logos', my pfSense logos' are screwed up on both my units, not critical but visually obvious. :)
Refresh browser cache, fixed the broken logo for me.
Yup.. I'll go sit in the corner with my dunces hat on… :-[
-
Updating to build 2.4.0.r.20171009.1758 worked like a charme.
pfSense is now running about 12 hours without showing any problems so I'd say "Good work and thank you" to the pfSense Team.
-
I'm also running good & stable on 2.4.1.a.20171009.2023 since last night. Thank you guys 8)
-
@kpa:
Edit: I realize now that the real problem was a bad dhcpd binary and the package system didn't include an updated dhcpd package, it actually did but since the version number hadn't changed pkg didn't offer it as an update and the only way to get the fixed dhcpd package was to use pkg install/upgrade -f.
Errr… Uh. So, WTH is the deal here?
$ cat /etc/version.buildtime Mon Oct 09 18:53:43 CDT 2017Enter an option: 13 >>> Updating repositories metadata... Updating Unofficial repository catalogue... Fetching meta.txz: . done Fetching packagesite.txz: . done Processing entries: .. done Unofficial repository update completed. 11 packages processed. Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Your packages are up to date# pkg upgrade -f Updating Unofficial repository catalogue... Fetching meta.txz: 100% 260 B 0.3kB/s 00:01 Fetching packagesite.txz: 100% 5 KiB 4.7kB/s 00:01 Processing entries: 100% Unofficial repository update completed. 11 packages processed. Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Checking for upgrades (236 candidates): 100% Processing candidates (236 candidates): 100% The following 236 package(s) will be affected (of 0 checked): Installed packages to be REINSTALLED: pkg-1.10.1_1 [pfSense] zip-3.0_1 [pfSense] xinetd-2.3.15_2 [pfSense] wrapalixresetbutton-0.0.7 [pfSense] wol-0.7.1_3 [pfSense] whois-5.2.17 [pfSense] webp-0.6.0_4 [pfSense] vstr-1.0.15_1 [pfSense] voucher-0.1_2 [pfSense] vnstat-1.15 [pfSense] unzoo-4.4_2 [pfSense] unbound-1.6.6 [pfSense] uclcmd-0.1_2 [pfSense] tiff-4.0.8 [pfSense] tftp-hpa-5.2 [pfSense] talloc-2.1.9 [pfSense] suricata-4.0.0 [pfSense] sudo-1.8.21p2 [pfSense] strongswan-5.6.0 [pfSense] sshlockout_pf-0.0.2 [pfSense] ssh_tunnel_shell-0.1_1 [pfSense] squidclamav-6.16 [pfSense] squid_radius_auth-1.10 [pfSense] squid-3.5.27 [pfSense] sqlite3-3.20.1_1 [pfSense] smartmontools-6.5_2 [pfSense] scponly-4.8.20110526_2 [pfSense] rrdtool-1.7.0_1 [pfSense] relayd-5.5.20140810_2 [pfSense] readline-7.0.3 [pfSense] rate-0.9_1 [pfSense] radvd-1.9.1 [pfSense] qstats-0.2 [pfSense] python27-2.7.14 [pfSense] python2-2_3 [pfSense] py27-speedtest-cli-1.0.6 [pfSense] py27-setuptools-36.2.2 [pfSense] postgresql95-client-9.5.9 [pfSense] png-1.6.34 [pfSense] pkgconf-1.3.7,1 [pfSense] php56-zlib-5.6.31 [pfSense] php56-xmlwriter-5.6.31 [pfSense] php56-xmlreader-5.6.31 [pfSense] php56-xml-5.6.31 [pfSense] php56-tokenizer-5.6.31 [pfSense] php56-sysvshm-5.6.31 [pfSense] php56-sysvsem-5.6.31 [pfSense] php56-sysvmsg-5.6.31 [pfSense] php56-sqlite3-5.6.31 [pfSense] php56-sockets-5.6.31 [pfSense] php56-simplexml-5.6.31 [pfSense] php56-shmop-5.6.31 [pfSense] php56-session-5.6.31 [pfSense] php56-readline-5.6.31 [pfSense] php56-posix-5.6.31 [pfSense] php56-pfSense-module-0.54 [pfSense] php56-pdo_sqlite-5.6.31 [pfSense] php56-pdo-5.6.31 [pfSense] php56-pcntl-5.6.31 [pfSense] php56-openssl-5.6.31 [pfSense] php56-opcache-5.6.31 [pfSense] php56-mcrypt-5.6.31 [pfSense] php56-mbstring-5.6.31_1 [pfSense] php56-ldap-5.6.31 [pfSense] php56-json-5.6.31 [pfSense] php56-hash-5.6.31 [pfSense] php56-gettext-5.6.31 [pfSense] php56-ftp-5.6.31 [pfSense] php56-filter-5.6.31 [pfSense] php56-dom-5.6.31 [pfSense] php56-curl-5.6.31 [pfSense] php56-ctype-5.6.31 [pfSense] php56-bz2-5.6.31 [pfSense] php56-bcmath-5.6.31 [pfSense] php56-5.6.31 [pfSense] php-xdebug-2.5.0 [pfSense] php-suhosin-0.9.38_3 [pfSense] pftop-0.7_8 [pfSense] pfSense-upgrade-0.33 [pfSense] pfSense-repo-2.4.1.a.20171009.1753 [pfSense] pfSense-rc-2.4.1.a.20171009.1853 [pfSense-core] pfSense-pkg-tftpd-0.1.3_1 [pfSense] pfSense-pkg-suricata-4.0.0_1 [pfSense] pfSense-pkg-sudo-0.3 [pfSense] pfSense-pkg-squid-0.4.39 [pfSense] pfSense-pkg-pfBlockerNG-2.1.1_10 [pfSense] pfSense-pkg-openvpn-client-export-1.4.14 [pfSense] pfSense-pkg-nmap-1.4.4_1 [pfSense] pfSense-pkg-mtr-nox11-0.85.6_1 [pfSense] pfSense-pkg-iftop-0.17_2 [pfSense] pfSense-pkg-freeradius3-0.15.1 [pfSense] pfSense-pkg-arping-1.2.2_1 [pfSense] pfSense-pkg-acme-0.1.19_1 [pfSense] pfSense-pkg-System_Patches-1.1.7 [pfSense] pfSense-pkg-Status_Traffic_Totals-1.2.1_1 [pfSense] pfSense-pkg-Shellcmd-1.0.3 [pfSense] pfSense-pkg-Service_Watchdog-1.8.4 [pfSense] pfSense-pkg-RRD_Summary-1.3.2_2 [pfSense] pfSense-pkg-Lightsquid-3.0.6_4 [pfSense] pfSense-pkg-FTP_Client_Proxy-0.3_3 [pfSense] pfSense-pkg-Cron-0.3.7_2 [pfSense] pfSense-pkg-AutoConfigBackup-1.47 [pfSense] pfSense-kernel-pfSense-2.4.1.a.20171009.1853 [pfSense-core] pfSense-default-config-serial-2.4.1.a.20171009.1853 [pfSense-core] pfSense-base-2.4.1.a.20171009.1853 [pfSense-core] pfSense-Status_Monitoring-1.7.2 [pfSense] pfSense-2.4.1.a.20171009.1753 [pfSense] perl5-5.24.3 [pfSense] pecl-zmq-1.1.3_2 [pfSense] pecl-ssh2-0-0.13 [pfSense] pecl-rrd1-1.1.3 [pfSense] pecl-radius-1.4.0.b1 [pfSense] pecl-intl-3.0.0_10 [pfSense] pear-XML_RPC2-1.1.3 [pfSense] pear-Net_URL2-2.2.1 [pfSense] pear-Net_Socket-1.0.14 [pfSense] pear-Net_SMTP-1.8.0 [pfSense] pear-Net_IPv6-1.3.0.b2_2 [pfSense] pear-Net_Growl-2.7.0 [pfSense] pear-Mail-1.4.1,1 [pfSense] pear-HTTP_Request2-2.3.0,1 [pfSense] pear-Crypt_CHAP-1.5.0 [pfSense] pear-Cache_Lite-1.7.16,1 [pfSense] pear-Auth_RADIUS-1.1.0 [pfSense] pear-1.10.5 [pfSense] pcre-8.40_1 [pfSense] p7zip-16.02 [pfSense] p5-HTML-Tagset-3.20_1 [pfSense] p5-HTML-Parser-3.72 [pfSense] p5-GD-2.66 [pfSense] p5-ExtUtils-PkgConfig-1.16 [pfSense] p5-CGI-4.36 [pfSense] openvpn-client-export-2.4.4 [pfSense] openvpn-2.4.4 [pfSense] openldap-client-2.4.45 [pfSense] oniguruma6-6.4.0 [pfSense] ntp-4.2.8p10_2 [pfSense] nss-3.33_1 [pfSense] nspr-4.17 [pfSense] norm-1.5r6 [pfSense] nmap-7.40_1 [pfSense] nginx-1.12.1_1,2 [pfSense] nettle-3.3 [pfSense] nano-2.8.7 [pfSense] mysql56-client-5.6.37_1 [pfSense] mtr-nox11-0.92 [pfSense] mpd5-5.8_2 [pfSense] mobile-broadband-provider-info-20170310 [pfSense] miniupnpd-1.9.20160113,1 [pfSense] minicron-0.0.2 [pfSense] lzo2-2.10_1 [pfSense] luajit-2.0.5 [pfSense] lua52-5.2.4 [pfSense] links-2.13,1 [pfSense] lighttpd-1.4.45_1 [pfSense] lightsquid-1.8_5 [pfSense] libzmq4-4.2.2 [pfSense] libyaml-0.1.6_2 [pfSense] libxml2-2.9.4 [pfSense] libunistring-0.9.7 [pfSense] libucl-0.8.0 [pfSense] libssh2-1.8.0,3 [pfSense] libpcap-1.8.1 [pfSense] libnghttp2-1.26.0 [pfSense] libnet-1.1.6_5,1 [pfSense] libmcrypt-2.5.8_3 [pfSense] liblz4-1.8.0,1 [pfSense] libltdl-2.4.6 [pfSense] libidn2-2.0.4 [pfSense] libiconv-1.14_11 [pfSense] libhtp-0.5.25 [pfSense] libgd-2.2.4_1,1 [pfSense] libffi-3.2.1_1 [pfSense] libevent-2.1.8 [pfSense] libedit-3.1.20170329_2,1 [pfSense] libdaemon-0.14_1 [pfSense] lha-1.14i_7 [pfSense] ldns-1.7.0_1 [pfSense] krb5-1.15.2 [pfSense] json-c-0.12.1 [pfSense] jpeg-turbo-1.5.2 [pfSense] jbigkit-2.1_1 [pfSense] jansson-2.10 [pfSense] isc-dhcp43-server-4.3.6_1 [pfSense] isc-dhcp43-relay-4.3.6 [pfSense] isc-dhcp43-client-4.3.6 [pfSense] ipmitool-1.8.18_1 [pfSense] indexinfo-0.2.6 [pfSense] igmpproxy-0.1_5,1 [pfSense] iftop-1.0.p4 [pfSense] idnkit-1.0_6 [pfSense] icu-59.1,1 [pfSense] hyperscan-4.5.2 [pfSense] hiredis-0.13.3 [pfSense] grepcidr-2.0 [pfSense] gmp-6.1.2 [pfSense] glib-2.50.2_6,1 [pfSense] giflib-5.1.4 [pfSense] gettext-runtime-0.19.8.1_1 [pfSense] gdbm-1.13_1 [pfSense] freetype2-2.8 [pfSense] freeradius3-3.0.15_1 [pfSense] filterlog-0.1_5 [pfSense] filterdns-1.0_16 [pfSense] expiretable-0.6_1 [pfSense] expat-2.2.1 [pfSense] dpinger-2.0_1 [pfSense] dnsmasq-2.78,1 [pfSense] dmidecode-3.1_1 [pfSense] dhcpleases6-0.1_2 [pfSense] dhcpleases-0.3_1 [pfSense] dhcp6-20080615.2 [pfSense] cyrus-sasl-2.1.26_12 [pfSense] curl-7.56.0 [pfSense] cpustats-0.1_1 [pfSense] cpdup-1.18 [pfSense] clog-1.0.1_1 [pfSense] clamav-0.99.2_5 [pfSense] choparp-20150613 [pfSense] check_reload_status-0.0.8 [pfSense] ca_root_nss-3.32.1 [pfSense] c-icap-modules-0.4.5 [pfSense] c-icap-0.4.4,2 [pfSense] bwi-firmware-kmod-3.130.20 [pfSense] bsnmp-ucd-0.4.2 [pfSense] bsnmp-regex-0.6_1 [pfSense] broccoli-1.97,1 [pfSense] bind-tools-9.11.2 [pfSense] beep-1.0_1 [pfSense] bash-4.4.12_2 [pfSense] barnyard2-1.13_1 [pfSense] ataidle-2.7.2 [pfSense] arping-2.15_1 [pfSense] arc-5.21p [pfSense] aggregate-1.6_1 [pfSense] GeoIP-1.6.11 [pfSense] Number of packages to be reinstalled: 236 The process will require 2 MiB more space. 102 MiB to be downloaded.??? ::) :o
Is there any way to get to consistent system state when upgrading after this grand screw-up, or should I just reinstall altogether?
-
Erm… I have a strong feeling that the pre-kaboom snapshots should be taken, released as 2.4.0 and this insane messing should be moved to 2.4.1. WTH is anyone doing changes like this days before release, after months and months of beta testing.
It's not quite that simple, but in a vague sort of way that's what we did.
The change made was a small change to a data structure size meant to fix an interface issue. This caused older binaries for services and other programs to fail on an updated kernel, which left the system in an inconsistent state and likely without connectivity. The packages were recompiled with the new structures, but their version numbers didn't change, so pkg did not know it had to reinstall all affected packages. New installs made during this time were OK, and anyone who forced all packages to reinstall would have been OK as well, because in those cases the firewall had a fully consistent set of binaries for kernel, world, and packages.
We ultimately decided to take down those problem snapshots and back out that change, so that people who had not yet upgraded to a problem snapshot would be safe from the update issue. After reverting that change and rebuilding all of the package sets, then testing internally, we turned everything back on. We tried a few alternate solutions such as forcing pkg to update everything on upgrades between certain timestamps but we were not satisfied with any of the workarounds.
For those who did not update during the problem time, updating to a current snapshot is safe.
If you updated a firewall to a broken snapshot and fixed it manually by using "pkg upgrade -f", or if you made a fresh install from a problem image, you will need to manually update to the latest set of packages using "pkg upgrade -f" to forcefully reinstall everything. Otherwise, the firewall will end up in an inconsistent broken state again. Alternately, reinstall from a new snapshot.
Unrelated to that problem, we also had an issue with logging late last week. A problem with the clog program generated log files in an invalid format, leading to a variety of error message and a lack of logging data. The problem has been corrected on current snapshots, but if a firewall had generated new log files during the problematic period, such as on a fresh installation or after resetting/clearing logs, then those log files must be reset.
To reset the log files, navigate to Status > System Logs, Settings tab. Click "Reset Log Files", then OK to confirm.
At the moment all of the current snapshots should be OK and trouble-free.
