Upgrade 23.05.1 to 23.09: disk I/O error and fail
-
I upgraded pfSense+ from 23.05.1 to 23.09 and it failed with the following messages:
>>> Updating repositories metadata... Updating pfSense-core repository catalogue... pkg-static: sqlite error while executing SELECT count(name) FROM sqlite_master WHERE type='table' AND name='repodata'; in file pkgdb.c:2367: database is locked pkg-static: Repository pfSense-core contains no repodata table, need to re-create database Fetching meta.conf: . done Fetching packagesite.pkg: . done pkg-static: sqlite error while executing CREATE TABLE packages (id INTEGER PRIMARY KEY,origin TEXT,name TEXT NOT NULL,version TEXT NOT NULL,comment TEXT NOT NULL,desc TEXT NOT NULL,osversion TEXT,arch TEXT NOT NULL,maintainer TEXT NOT NULL,www TEXT,prefix TEXT NOT NULL,pkgsize INTEGER NOT NULL,flatsize INTEGER NOT NULL,licenselogic INTEGER NOT NULL,cksum TEXT NOT NULL,path TEXT NOT NULL,pkg_format_version INTEGER,manifestdigest TEXT NULL,olddigest TEXT NULL,dep_formula TEXT NULL,vital INTEGER NOT NULL DEFAULT 0);CREATE TABLE deps (origin TEXT,name TEXT,version TEXT,package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,UNIQUE(package_id, name));CREATE TABLE categories (id INTEGER PRIMARY KEY, name TEXT NOT NULL UNIQUE );CREATE TABLE pkg_categories (package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,category_id INTEGER REFERENCES categories(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, category_id));CREATE TABLE licenses (id INTEGER PRIMARY KEY,name TEXT NOT NULL UNIQUE);CREATE TABLE pkg_licenses (package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,license_id INTEGER REFERENCES licenses(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, license_id));CREATE TABLE option (option_id INTEGER PRIMARY KEY,option TEXT NOT NULL UNIQUE);CREATE TABLE option_desc (option_desc_id INTEGER PRIMARY KEY,option_desc TEXT NOT NULL UNIQUE);CREATE TABLE pkg_option (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,option_id INTEGER NOT NULL REFERENCES option(option_id) ON DELETE RESTRICT ON UPDATE CASCADE,value TEXT NOT NULL,PRIMARY KEY(package_id, option_id));CREATE TABLE pkg_option_desc (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,option_id INTEGER NOT NULL REFERENCES option(option_id) ON DELETE RESTRICT ON UPDATE CASCADE,option_desc_id INTEGER NOT NULL REFERENCES option_desc(option_desc_id) ON DELETE RESTRICT ON UPDATE CASCADE,PRIMARY KEY(package_id, option_id));CREATE TABLE pkg_option_default (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,option_id INTEGER NOT NULL REFERENCES option(option_id) ON DELETE RESTRICT ON UPDATE CASCADE,default_value TEXT NOT NULL,PRIMARY KEY(package_id, option_id));CREATE TABLE shlibs (id INTEGER PRIMARY KEY,name TEXT NOT NULL UNIQUE );CREATE TABLE pkg_shlibs_required (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,shlib_id INTEGER NOT NULL REFERENCES shlibs(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, shlib_id));CREATE TABLE pkg_shlibs_provided (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,shlib_id INTEGER NOT NULL REFERENCES shlibs(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, shlib_id));CREATE TABLE annotation (annotation_id INTEGER PRIMARY KEY,annotation TEXT NOT NULL UNIQUE);CREATE TABLE pkg_annotation (package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE RESTRICT,tag_id INTEGER NOT NULL REFERENCES annotation(annotation_id) ON DELETE CASCADE ON UPDATE RESTRICT,value_id INTEGER NOT NULL REFERENCES annotation(annotation_id) ON DELETE CASCADE ON UPDATE RESTRICT,UNIQUE (package_id, tag_id));CREATE TABLE pkg_conflicts (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,conflict_id INTEGER NOT NULL,UNIQUE(package_id, conflict_id));CREATE TABLE provides( id INTEGER PRIMARY KEY, provide TEXT NOT NULL);CREATE TABLE pkg_provides (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,provide_id INTEGER NOT NULL REFERENCES provides(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, provide_id));CREATE TABLE requires( id INTEGER PRIMARY KEY, require TEXT NOT NULL);CREATE TABLE pkg_requires (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,require_id INTEGER NOT NULL REFERENCES requires(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, require_id));PRAGMA user_version=2014; in file pkgdb.c:2343: disk I/O error Unable to create repository pfSense-core Unable to update repository pfSense-core Updating pfSense repository catalogue... Fetching meta.conf: . done Fetching packagesite.pkg: ....... done pkg-static: sqlite error while executing CREATE TABLE packages (id INTEGER PRIMARY KEY,origin TEXT,name TEXT NOT NULL,version TEXT NOT NULL,comment TEXT NOT NULL,desc TEXT NOT NULL,osversion TEXT,arch TEXT NOT NULL,maintainer TEXT NOT NULL,www TEXT,prefix TEXT NOT NULL,pkgsize INTEGER NOT NULL,flatsize INTEGER NOT NULL,licenselogic INTEGER NOT NULL,cksum TEXT NOT NULL,path TEXT NOT NULL,pkg_format_version INTEGER,manifestdigest TEXT NULL,olddigest TEXT NULL,dep_formula TEXT NULL,vital INTEGER NOT NULL DEFAULT 0);CREATE TABLE deps (origin TEXT,name TEXT,version TEXT,package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,UNIQUE(package_id, name));CREATE TABLE categories (id INTEGER PRIMARY KEY, name TEXT NOT NULL UNIQUE );CREATE TABLE pkg_categories (package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,category_id INTEGER REFERENCES categories(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, category_id));CREATE TABLE licenses (id INTEGER PRIMARY KEY,name TEXT NOT NULL UNIQUE);CREATE TABLE pkg_licenses (package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,license_id INTEGER REFERENCES licenses(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, license_id));CREATE TABLE option (option_id INTEGER PRIMARY KEY,option TEXT NOT NULL UNIQUE);CREATE TABLE option_desc (option_desc_id INTEGER PRIMARY KEY,option_desc TEXT NOT NULL UNIQUE);CREATE TABLE pkg_option (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,option_id INTEGER NOT NULL REFERENCES option(option_id) ON DELETE RESTRICT ON UPDATE CASCADE,value TEXT NOT NULL,PRIMARY KEY(package_id, option_id));CREATE TABLE pkg_option_desc (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,option_id INTEGER NOT NULL REFERENCES option(option_id) ON DELETE RESTRICT ON UPDATE CASCADE,option_desc_id INTEGER NOT NULL REFERENCES option_desc(option_desc_id) ON DELETE RESTRICT ON UPDATE CASCADE,PRIMARY KEY(package_id, option_id));CREATE TABLE pkg_option_default (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,option_id INTEGER NOT NULL REFERENCES option(option_id) ON DELETE RESTRICT ON UPDATE CASCADE,default_value TEXT NOT NULL,PRIMARY KEY(package_id, option_id));CREATE TABLE shlibs (id INTEGER PRIMARY KEY,name TEXT NOT NULL UNIQUE );CREATE TABLE pkg_shlibs_required (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,shlib_id INTEGER NOT NULL REFERENCES shlibs(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, shlib_id));CREATE TABLE pkg_shlibs_provided (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,shlib_id INTEGER NOT NULL REFERENCES shlibs(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, shlib_id));CREATE TABLE annotation (annotation_id INTEGER PRIMARY KEY,annotation TEXT NOT NULL UNIQUE);CREATE TABLE pkg_annotation (package_id INTEGER REFERENCES packages(id) ON DELETE CASCADE ON UPDATE RESTRICT,tag_id INTEGER NOT NULL REFERENCES annotation(annotation_id) ON DELETE CASCADE ON UPDATE RESTRICT,value_id INTEGER NOT NULL REFERENCES annotation(annotation_id) ON DELETE CASCADE ON UPDATE RESTRICT,UNIQUE (package_id, tag_id));CREATE TABLE pkg_conflicts (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,conflict_id INTEGER NOT NULL,UNIQUE(package_id, conflict_id));CREATE TABLE provides( id INTEGER PRIMARY KEY, provide TEXT NOT NULL);CREATE TABLE pkg_provides (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,provide_id INTEGER NOT NULL REFERENCES provides(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, provide_id));CREATE TABLE requires( id INTEGER PRIMARY KEY, require TEXT NOT NULL);CREATE TABLE pkg_requires (package_id INTEGER NOT NULL REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,require_id INTEGER NOT NULL REFERENCES requires(id) ON DELETE RESTRICT ON UPDATE RESTRICT,UNIQUE(package_id, require_id));PRAGMA user_version=2014; in file pkgdb.c:2343: disk I/O error Unable to create repository pfSense Unable to update repository pfSense Error updating repositories! ERROR: It was not possible to determine pfSense-upgrade remote version ERROR: It was not possible to determine pfSense-upgrade remote version ERROR: It was not possible to determine pfSense-upgrade remote version ERROR: It was not possible to determine pfSense-upgrade remote version >>> Upgrading pfSense-upgrade... >>> Upgrading pfSense-upgrade... failed. __RC=1 __REBOOT_AFTER=10 failed.Trying a second time worked flawlessly:
>>> Updating repositories metadata... Updating pfSense-core repository catalogue... Fetching meta.conf: . done Fetching packagesite.pkg: . done Processing entries: . done pfSense-core repository update completed. 5 packages processed. Updating pfSense repository catalogue... Fetching meta.conf: . done Fetching packagesite.pkg: ..... done Processing entries: Processing entries............. done pfSense repository update completed. 726 packages processed. All repositories are up to date. >>> Setting vital flag on pkg... done. >>> Removing vital flag from php82... done. >>> Downloading upgrade packages... Updating pfSense-core repository catalogue... Fetching meta.conf: pfSense-core repository is up to date. Updating pfSense repository catalogue... Fetching meta.conf: pfSense repository is up to date. All repositories are up to date. Checking for upgrades (85 candidates): .......... done Processing candidates (85 candidates): .......... done The following 95 package(s) will be affected (of 0 checked): New packages to be INSTALLED: boost-libs: 1.82.0_1 [pfSense] dhcpcd: 10.0.2 [pfSense] dhcpleases6: 0.1_3 [pfSense] gmp: 6.3.0 [pfSense] kea: 2.4.0 [pfSense] log4cplus: 2.1.0 [pfSense] pfSense-composer-deps: 0.1 [pfSense] php82-gmp: 8.2.11 [pfSense] py311-libzfs: 1.1.2023020700 [pfSense] py311-setuptools: 63.1.0_1 [pfSense] Installed packages to be UPGRADED: aws-sdk-php82: 3.232.3 -> 3.273.3 [pfSense] bind-tools: 9.18.13 -> 9.18.16 [pfSense] ca_root_nss: 3.89 -> 3.92 [pfSense] curl: 8.0.1 -> 8.4.0 [pfSense] cyrus-sasl: 2.1.28 -> 2.1.28_1 [pfSense] devcpu-data: 20230219 -> 20230617_1 [pfSense] devcpu-data-amd: 20230219 -> 20230724 [pfSense] devcpu-data-intel: 20230214 -> 20230808 [pfSense] filterlog: 0.1_9 -> 0.1_10 [pfSense] gettext-runtime: 0.21.1 -> 0.22_1 [pfSense] glib: 2.76.1,2 -> 2.76.4,2 [pfSense] icu: 72.1,1 -> 73.2,1 [pfSense] igmpproxy: 0.3_1,1 -> 0.4,1 [pfSense] isc-dhcp44-relay: 4.4.3P1 -> 4.4.3P1_3 [pfSense] isc-dhcp44-server: 4.4.3P1 -> 4.4.3P1_3 [pfSense] libgcrypt: 1.9.4_1 -> 1.10.2 [pfSense] libgpg-error: 1.46 -> 1.47 [pfSense] libnghttp2: 1.52.0 -> 1.55.1_1 [pfSense] libpsl: 0.21.2_2 -> 0.21.2_3 [pfSense] libssh2: 1.10.0_1,3 -> 1.11.0_1,3 [pfSense] libuv: 1.44.2 -> 1.46.0 [pfSense] libxml2: 2.10.3_1 -> 2.10.4 [pfSense] links: 2.28,1 -> 2.29_1,1 [pfSense] lua-resty-core: 0.1.24 -> 0.1.27 [pfSense] luajit-openresty: 2.1.20230119 -> 2.1.20230410 [pfSense] miniupnpd: 2.2.1_1,1 -> 2.3.3_1,1 [pfSense] mobile-broadband-provider-info: 20221107 -> 20230416 [pfSense] mpd5: 5.9_14 -> 5.9_16 [pfSense] nginx: 1.22.1_5,3 -> 1.24.0_11,3 [pfSense] ntp: 4.2.8p15_5 -> 4.2.8p17_1 [pfSense] openldap26-client: 2.6.4 -> 2.6.6 [pfSense] openvpn: 2.6.2 -> 2.6.5 [pfSense] pcsc-lite: 1.9.9,2 -> 2.0.0,2 [pfSense] perl5: 5.32.1_3 -> 5.34.1_3 [pfSense] pfSense: 23.05.1 -> 23.09 [pfSense] pfSense-base: 23.05.1 -> 23.09 [pfSense-core] pfSense-boot: 23.05.1 -> 23.09 [pfSense-core] pfSense-default-config-serial: 23.05.1 -> 23.09 [pfSense] pfSense-kernel-pfSense: 23.05.1 -> 23.09 [pfSense-core] pfSense-pkg-ipsec-profile-wizard: 1.1_1 -> 1.2 [pfSense] pfSense-repo: 23.05.1 -> 23.09 [pfSense] pftop: 0.8_2 -> 0.8_4 [pfSense] php82: 8.2.4 -> 8.2.11 [pfSense] php82-bcmath: 8.2.4 -> 8.2.11 [pfSense] php82-bz2: 8.2.4 -> 8.2.11 [pfSense] php82-ctype: 8.2.4 -> 8.2.11 [pfSense] php82-curl: 8.2.4 -> 8.2.11 [pfSense] php82-dom: 8.2.4 -> 8.2.11 [pfSense] php82-filter: 8.2.4 -> 8.2.11 [pfSense] php82-gettext: 8.2.4 -> 8.2.11 [pfSense] php82-intl: 8.2.4_1 -> 8.2.11 [pfSense] php82-ldap: 8.2.4 -> 8.2.11 [pfSense] php82-mbstring: 8.2.4 -> 8.2.11 [pfSense] php82-opcache: 8.2.4 -> 8.2.11 [pfSense] php82-openssl_x509_crl: 1.3_2 -> 1.3_3 [pfSense] php82-pcntl: 8.2.4 -> 8.2.11 [pfSense] php82-pdo: 8.2.4 -> 8.2.11 [pfSense] php82-pdo_sqlite: 8.2.4 -> 8.2.11 [pfSense] php82-pear-Mail: 1.4.1,1 -> 1.5.0,1 [pfSense] php82-posix: 8.2.4 -> 8.2.11 [pfSense] php82-readline: 8.2.4 -> 8.2.11 [pfSense] php82-session: 8.2.4 -> 8.2.11 [pfSense] php82-shmop: 8.2.4 -> 8.2.11 [pfSense] php82-simplexml: 8.2.4 -> 8.2.11 [pfSense] php82-sockets: 8.2.4 -> 8.2.11 [pfSense] php82-sqlite3: 8.2.4 -> 8.2.11 [pfSense] php82-sysvmsg: 8.2.4 -> 8.2.11 [pfSense] php82-sysvsem: 8.2.4 -> 8.2.11 [pfSense] php82-sysvshm: 8.2.4 -> 8.2.11 [pfSense] php82-tokenizer: 8.2.4 -> 8.2.11 [pfSense] php82-xml: 8.2.4 -> 8.2.11 [pfSense] php82-xmlreader: 8.2.4 -> 8.2.11 [pfSense] php82-xmlwriter: 8.2.4 -> 8.2.11 [pfSense] php82-zlib: 8.2.4 -> 8.2.11 [pfSense] py39-setuptools: 63.1.0 -> 63.1.0_1 [pfSense] python311: 3.11.2_2 -> 3.11.4 [pfSense] python39: 3.9.16_2 -> 3.9.17 [pfSense] smartmontools: 7.3 -> 7.4 [pfSense] sqlite3: 3.41.0_1,1 -> 3.42.0,1 [pfSense] strongswan: 5.9.10_2 -> 5.9.11_1 [pfSense] uclcmd: 0.1_3 -> 0.2.20211204 [pfSense] unbound: 1.17.1_3 -> 1.18.0 [pfSense] voucher: 0.1_2 -> 0.1_3 [pfSense] zstd: 1.5.4_1 -> 1.5.5 [pfSense] Installed packages to be REINSTALLED: rrdtool-1.8.0_2 [pfSense] (direct dependency changed: perl5) Number of packages to be installed: 10 Number of packages to be upgraded: 84 Number of packages to be reinstalled: 1 The process will require 275 MiB more space. 311 MiB to be downloaded. [1/95] Fetching unbound-1.18.0.pkg: ......... done [2/95] Fetching php82-session-8.2.11.pkg: . done [3/95] Fetching nginx-1.24.0_11,3.pkg: .. done [4/95] Fetching php82-gmp-8.2.11.pkg: . done [5/95] Fetching pfSense-base-23.09.pkg: .......... done [6/95] Fetching filterlog-0.1_10.pkg: . done [7/95] Fetching devcpu-data-20230617_1.pkg: . done [8/95] Fetching libuv-1.46.0.pkg: ..... done [9/95] Fetching boost-libs-1.82.0_1.pkg: .......... done [10/95] Fetching uclcmd-0.2.20211204.pkg: . done [11/95] Fetching php82-zlib-8.2.11.pkg: . done [12/95] Fetching php82-dom-8.2.11.pkg: .. done [13/95] Fetching php82-simplexml-8.2.11.pkg: . done [14/95] Fetching devcpu-data-amd-20230724.pkg: . done [15/95] Fetching php82-shmop-8.2.11.pkg: . done [16/95] Fetching lua-resty-core-0.1.27.pkg: . done [17/95] Fetching pcsc-lite-2.0.0,2.pkg: .. done [18/95] Fetching py311-setuptools-63.1.0_1.pkg: .... done [19/95] Fetching openvpn-2.6.5.pkg: . done [20/95] Fetching php82-pdo-8.2.11.pkg: . done [21/95] Fetching libnghttp2-1.55.1_1.pkg: . done [22/95] Fetching libxml2-2.10.4.pkg: . done [23/95] Fetching php82-readline-8.2.11.pkg: . done [24/95] Fetching php82-curl-8.2.11.pkg: . done [25/95] Fetching python39-3.9.17.pkg: .......... done [26/95] Fetching icu-73.2,1.pkg: ......... done [27/95] Fetching bind-tools-9.18.16.pkg: ......... done [28/95] Fetching rrdtool-1.8.0_2.pkg: . done [29/95] Fetching aws-sdk-php82-3.273.3.pkg: .. done [30/95] Fetching pfSense-23.09.pkg: ........ done [31/95] Fetching php82-mbstring-8.2.11.pkg: . done [32/95] Fetching isc-dhcp44-server-4.4.3P1_3.pkg: . done [33/95] Fetching php82-tokenizer-8.2.11.pkg: . done [34/95] Fetching ntp-4.2.8p17_1.pkg: . done [35/95] Fetching libpsl-0.21.2_3.pkg: . done [36/95] Fetching voucher-0.1_3.pkg: . done [37/95] Fetching php82-ldap-8.2.11.pkg: . done [38/95] Fetching pfSense-boot-23.09.pkg: ........ done [39/95] Fetching miniupnpd-2.3.3_1,1.pkg: ... done [40/95] Fetching ca_root_nss-3.92.pkg: ....... done [41/95] Fetching python311-3.11.4.pkg: .......... done [42/95] Fetching links-2.29_1,1.pkg: ...... done [43/95] Fetching pfSense-repo-23.09.pkg: . done [44/95] Fetching pfSense-default-config-serial-23.09.pkg: . done [45/95] Fetching php82-sysvmsg-8.2.11.pkg: . done [46/95] Fetching pfSense-pkg-ipsec-profile-wizard-1.2.pkg: . done [47/95] Fetching kea-2.4.0.pkg: ....... done [48/95] Fetching dhcpcd-10.0.2.pkg: . done [49/95] Fetching pfSense-composer-deps-0.1.pkg: . done [50/95] Fetching mobile-broadband-provider-info-20230416.pkg: . done [51/95] Fetching php82-posix-8.2.11.pkg: . done [52/95] Fetching dhcpleases6-0.1_3.pkg: . done [53/95] Fetching libgcrypt-1.10.2.pkg: . done [54/95] Fetching php82-sockets-8.2.11.pkg: ... done [55/95] Fetching php82-8.2.11.pkg: .... done [56/95] Fetching php82-sqlite3-8.2.11.pkg: . done [57/95] Fetching pfSense-kernel-pfSense-23.09.pkg: .......... done [58/95] Fetching zstd-1.5.5.pkg: ....... done [59/95] Fetching php82-xmlreader-8.2.11.pkg: . done [60/95] Fetching log4cplus-2.1.0.pkg: .. done [61/95] Fetching php82-pcntl-8.2.11.pkg: . done [62/95] Fetching php82-xml-8.2.11.pkg: . done [63/95] Fetching libgpg-error-1.47.pkg: . done [64/95] Fetching curl-8.4.0.pkg: .. done [65/95] Fetching gmp-6.3.0.pkg: . done [66/95] Fetching igmpproxy-0.4,1.pkg: . done [67/95] Fetching gettext-runtime-0.22_1.pkg: . done [68/95] Fetching php82-intl-8.2.11.pkg: . done [69/95] Fetching php82-bcmath-8.2.11.pkg: . done [70/95] Fetching php82-bz2-8.2.11.pkg: . done [71/95] Fetching py39-setuptools-63.1.0_1.pkg: . done [72/95] Fetching cyrus-sasl-2.1.28_1.pkg: . done [73/95] Fetching php82-openssl_x509_crl-1.3_3.pkg: . done [74/95] Fetching php82-sysvshm-8.2.11.pkg: . done [75/95] Fetching php82-xmlwriter-8.2.11.pkg: . done [76/95] Fetching mpd5-5.9_16.pkg: . done [77/95] Fetching sqlite3-3.42.0,1.pkg: . done [78/95] Fetching php82-gettext-8.2.11.pkg: . done [79/95] Fetching openldap26-client-2.6.6.pkg: . done [80/95] Fetching glib-2.76.4,2.pkg: ..... done [81/95] Fetching devcpu-data-intel-20230808.pkg: ....... done [82/95] Fetching luajit-openresty-2.1.20230410.pkg: .......... done [83/95] Fetching perl5-5.34.1_3.pkg: .......... done [84/95] Fetching strongswan-5.9.11_1.pkg: .......... done [85/95] Fetching php82-sysvsem-8.2.11.pkg: . done [86/95] Fetching php82-pear-Mail-1.5.0,1.pkg: . done [87/95] Fetching php82-ctype-8.2.11.pkg: . done [88/95] Fetching smartmontools-7.4.pkg: ... done [89/95] Fetching libssh2-1.11.0_1,3.pkg: . done [90/95] Fetching isc-dhcp44-relay-4.4.3P1_3.pkg: .... done [91/95] Fetching php82-opcache-8.2.11.pkg: . done [92/95] Fetching pftop-0.8_4.pkg: . done [93/95] Fetching php82-filter-8.2.11.pkg: . done [94/95] Fetching py311-libzfs-1.1.2023020700.pkg: . done [95/95] Fetching php82-pdo_sqlite-8.2.11.pkg: . done Checking integrity... done (1 conflicting) - pfSense-23.09 [pfSense] conflicts with pfSense-rc-23.05.1 [installed] on /etc/pfSense-rc Checking integrity... done (0 conflicting) Conflicts with the existing packages have been found. One more solver iteration is needed to resolve them. The following 96 package(s) will be affected (of 0 checked): Installed packages to be REMOVED: pfSense-rc: 23.05.1 New packages to be INSTALLED: boost-libs: 1.82.0_1 [pfSense] dhcpcd: 10.0.2 [pfSense] dhcpleases6: 0.1_3 [pfSense] gmp: 6.3.0 [pfSense] kea: 2.4.0 [pfSense] log4cplus: 2.1.0 [pfSense] pfSense-composer-deps: 0.1 [pfSense] php82-gmp: 8.2.11 [pfSense] py311-libzfs: 1.1.2023020700 [pfSense] py311-setuptools: 63.1.0_1 [pfSense] Installed packages to be UPGRADED: aws-sdk-php82: 3.232.3 -> 3.273.3 [pfSense] bind-tools: 9.18.13 -> 9.18.16 [pfSense] ca_root_nss: 3.89 -> 3.92 [pfSense] curl: 8.0.1 -> 8.4.0 [pfSense] cyrus-sasl: 2.1.28 -> 2.1.28_1 [pfSense] devcpu-data: 20230219 -> 20230617_1 [pfSense] devcpu-data-amd: 20230219 -> 20230724 [pfSense] devcpu-data-intel: 20230214 -> 20230808 [pfSense] filterlog: 0.1_9 -> 0.1_10 [pfSense] gettext-runtime: 0.21.1 -> 0.22_1 [pfSense] glib: 2.76.1,2 -> 2.76.4,2 [pfSense] icu: 72.1,1 -> 73.2,1 [pfSense] igmpproxy: 0.3_1,1 -> 0.4,1 [pfSense] isc-dhcp44-relay: 4.4.3P1 -> 4.4.3P1_3 [pfSense] isc-dhcp44-server: 4.4.3P1 -> 4.4.3P1_3 [pfSense] libgcrypt: 1.9.4_1 -> 1.10.2 [pfSense] libgpg-error: 1.46 -> 1.47 [pfSense] libnghttp2: 1.52.0 -> 1.55.1_1 [pfSense] libpsl: 0.21.2_2 -> 0.21.2_3 [pfSense] libssh2: 1.10.0_1,3 -> 1.11.0_1,3 [pfSense] libuv: 1.44.2 -> 1.46.0 [pfSense] libxml2: 2.10.3_1 -> 2.10.4 [pfSense] links: 2.28,1 -> 2.29_1,1 [pfSense] lua-resty-core: 0.1.24 -> 0.1.27 [pfSense] luajit-openresty: 2.1.20230119 -> 2.1.20230410 [pfSense] miniupnpd: 2.2.1_1,1 -> 2.3.3_1,1 [pfSense] mobile-broadband-provider-info: 20221107 -> 20230416 [pfSense] mpd5: 5.9_14 -> 5.9_16 [pfSense] nginx: 1.22.1_5,3 -> 1.24.0_11,3 [pfSense] ntp: 4.2.8p15_5 -> 4.2.8p17_1 [pfSense] openldap26-client: 2.6.4 -> 2.6.6 [pfSense] openvpn: 2.6.2 -> 2.6.5 [pfSense] pcsc-lite: 1.9.9,2 -> 2.0.0,2 [pfSense] perl5: 5.32.1_3 -> 5.34.1_3 [pfSense] pfSense: 23.05.1 -> 23.09 [pfSense] pfSense-base: 23.05.1 -> 23.09 [pfSense-core] pfSense-boot: 23.05.1 -> 23.09 [pfSense-core] pfSense-default-config-serial: 23.05.1 -> 23.09 [pfSense] pfSense-kernel-pfSense: 23.05.1 -> 23.09 [pfSense-core] pfSense-pkg-ipsec-profile-wizard: 1.1_1 -> 1.2 [pfSense] pfSense-repo: 23.05.1 -> 23.09 [pfSense] pftop: 0.8_2 -> 0.8_4 [pfSense] php82: 8.2.4 -> 8.2.11 [pfSense] php82-bcmath: 8.2.4 -> 8.2.11 [pfSense] php82-bz2: 8.2.4 -> 8.2.11 [pfSense] php82-ctype: 8.2.4 -> 8.2.11 [pfSense] php82-curl: 8.2.4 -> 8.2.11 [pfSense] php82-dom: 8.2.4 -> 8.2.11 [pfSense] php82-filter: 8.2.4 -> 8.2.11 [pfSense] php82-gettext: 8.2.4 -> 8.2.11 [pfSense] php82-intl: 8.2.4_1 -> 8.2.11 [pfSense] php82-ldap: 8.2.4 -> 8.2.11 [pfSense] php82-mbstring: 8.2.4 -> 8.2.11 [pfSense] php82-opcache: 8.2.4 -> 8.2.11 [pfSense] php82-openssl_x509_crl: 1.3_2 -> 1.3_3 [pfSense] php82-pcntl: 8.2.4 -> 8.2.11 [pfSense] php82-pdo: 8.2.4 -> 8.2.11 [pfSense] php82-pdo_sqlite: 8.2.4 -> 8.2.11 [pfSense] php82-pear-Mail: 1.4.1,1 -> 1.5.0,1 [pfSense] php82-posix: 8.2.4 -> 8.2.11 [pfSense] php82-readline: 8.2.4 -> 8.2.11 [pfSense] php82-session: 8.2.4 -> 8.2.11 [pfSense] php82-shmop: 8.2.4 -> 8.2.11 [pfSense] php82-simplexml: 8.2.4 -> 8.2.11 [pfSense] php82-sockets: 8.2.4 -> 8.2.11 [pfSense] php82-sqlite3: 8.2.4 -> 8.2.11 [pfSense] php82-sysvmsg: 8.2.4 -> 8.2.11 [pfSense] php82-sysvsem: 8.2.4 -> 8.2.11 [pfSense] php82-sysvshm: 8.2.4 -> 8.2.11 [pfSense] php82-tokenizer: 8.2.4 -> 8.2.11 [pfSense] php82-xml: 8.2.4 -> 8.2.11 [pfSense] php82-xmlreader: 8.2.4 -> 8.2.11 [pfSense] php82-xmlwriter: 8.2.4 -> 8.2.11 [pfSense] php82-zlib: 8.2.4 -> 8.2.11 [pfSense] py39-setuptools: 63.1.0 -> 63.1.0_1 [pfSense] python311: 3.11.2_2 -> 3.11.4 [pfSense] python39: 3.9.16_2 -> 3.9.17 [pfSense] smartmontools: 7.3 -> 7.4 [pfSense] sqlite3: 3.41.0_1,1 -> 3.42.0,1 [pfSense] strongswan: 5.9.10_2 -> 5.9.11_1 [pfSense] uclcmd: 0.1_3 -> 0.2.20211204 [pfSense] unbound: 1.17.1_3 -> 1.18.0 [pfSense] voucher: 0.1_2 -> 0.1_3 [pfSense] zstd: 1.5.4_1 -> 1.5.5 [pfSense] Installed packages to be REINSTALLED: rrdtool-1.8.0_2 [pfSense] (direct dependency changed: perl5) Number of packages to be removed: 1 Number of packages to be installed: 10 Number of packages to be upgraded: 84 Number of packages to be reinstalled: 1 The process will require 275 MiB more space. >>> Downloading pkg... No packages are required to be fetched. Integrity check was successful. >>> Upgrading pfSense-boot...>>> Unmounting /boot/efi... done. Updating pfSense-core repository catalogue... Fetching meta.conf: pfSense-core repository is up to date. Updating pfSense repository catalogue... Fetching meta.conf: pfSense repository is up to date. All repositories are up to date. Checking integrity... done (0 conflicting) The following 1 package(s) will be affected (of 0 checked): Installed packages to be UPGRADED: pfSense-boot: 23.05.1 -> 23.09 [pfSense-core] Number of packages to be upgraded: 1 [1/1] Upgrading pfSense-boot from 23.05.1 to 23.09... [1/1] Extracting pfSense-boot-23.09: .......... done >>> Removing vital flag from pfSense-rc... done. >>> Upgrading pfSense kernel... Checking integrity... done (0 conflicting) The following 1 package(s) will be affected (of 0 checked): Installed packages to be UPGRADED: pfSense-kernel-pfSense: 23.05.1 -> 23.09 [pfSense-core] Number of packages to be upgraded: 1 The operation will free 123 MiB. [1/1] Upgrading pfSense-kernel-pfSense from 23.05.1 to 23.09... [1/1] Extracting pfSense-kernel-pfSense-23.09: .......... done ===> Keeping a copy of current kernel in /boot/kernel.old >>> Removing unnecessary packages... done. System is going to be upgraded. Rebooting in 10 seconds. SuccessNow the question is if I need to pay attention to the first failure? I see a "disk I/O error" there and that worries me a bit. Is there anything I can do to test if everything is fine? Or was this just some temporary hickup due to "database in use" or something like that?
It's now updated and seems to work fine, but I don't want to risk a sudden loss of the device. It's a SG-4860 by the way and I do have a spare one if it completely breaks, but I don't like to have an emergency installation on Christmas or something like that.
-
Hmm, odd error for pkg. Were you running ram disks? Are you running from eMMC?
-
@stephenw10 I used the regular user interface and clicked update. So probably not using a ram disk. It's also the standard hardware with no manual cards inserted or anything.
-
Hmm, if you're running from eMMC on a 4860 I would definitely check the drive write indicators because that will have seen some hours at this point:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc -
@stephenw10 Thanks for these hints. Indeed a good idea to check that. Although the device has quite some hours in use now, I doubt that the write counter is very high, because the only write-operation are probably the logs. I haven't used VPN or anything with it, just simple firewall operation. But who knows?
So I connected to a shell and installed the mmc-utils, which worked fine:
[23.09-RELEASE][admin@xxx]/root: pkg install -y mmc-utils; rehash Updating pfSense-core repository catalogue... Fetching meta.conf: 0% pfSense-core repository is up to date. Updating pfSense repository catalogue... Fetching meta.conf: 0% pfSense repository is up to date. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): New packages to be INSTALLED: mmc-utils: 20230612 [pfSense] Number of packages to be installed: 1 47 KiB to be downloaded. [1/1] Fetching mmc-utils-20230612.pkg: 100% 47 KiB 48.6kB/s 00:01 Checking integrity... done (0 conflicting) [1/1] Installing mmc-utils-20230612... [1/1] Extracting mmc-utils-20230612: 100% [23.09-RELEASE][admin@xxx]/root:But checking life doesn't work; there's no such device:
[23.09-RELEASE][admin@xxx]/root: mmc extcsd read /dev/mmcsd0rpmb open: No such file or directory [23.09-RELEASE][admin@xxx]/root: ls /dev/m* /dev/mdctl /dev/mem /dev/midistat /dev/mlx5ctl /dev/music0 [23.09-RELEASE][admin@xxx]/root: dmesg | grep mmc [23.09-RELEASE][admin@xxx]/root: df -h Filesystem Size Used Avail Capacity Mounted on /dev/gptid/some_guid 26G 2.7G 21G 11% / devfs 1.0K 0B 1.0K 0% /dev tmpfs 4.0M 148K 3.9M 4% /var/run devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev [23.09-RELEASE][admin@xxx]/root: mmc extcsd read /dev/gptid/some_guid open: Operation not permitted [23.09-RELEASE][admin@xxx]/root:I also tried to find the disk, but I was not successful as you can see above.
I just opened the spare device to take a look inside. I can find three sockets "1726" which seem like for SSD drives or something, but they are all three not populated. There's also a SIM card socket (or for memory), which is also unpopulated. Does this device have eMMC disks?
Update: yes, I found a chip Kingston EMMC32G soldered directly to the board. So yes, there is eMMC.
I also noticed a CR2032 battery. Not sure what it's needed for, but that might also wear out over the years. What data will it forget when I replace that battery?
-
Ah, yes I was forgetting the 4860 uses da0 as the eMMC. I'm not sure it can be read. Let me check....
Edit: Yup, no way to test it via that connection type AFAIK.
-
@stephenw10 Thanks for checking. So this essentially means to just wait and see until anything bad happens? If the eMMC reaches its end of life, more things should break and not only a single update only once.
But shouldn't it first reallocate unused blocks and reduce the available disk size? From the commands above I see that only 2.7GB are used from 26GB total, so there's plenty of reallocation blocks free. Not sure how that filesystem works though.
Is there any other test for data consistency I can run?
-
I would fit an mSATA drive in advance of that. Then install ZFS to it.
Though I will also point out that hardware is quite old at this point anyway.
-
@stephenw10 What do you mean with that exactly? Boot the system from that mSATA then and check the built-in disk that way somehow? Or was the idea that I have an emergency drive in case the main one fails entirely?
My plan would be to switch to the complete spare device if it really fails and then maybe replace the soldered-on eMMC. Not sure if that works though. Even if the soldering works fine, I have no idea how I can start the device with an empty eMMC. I assume the boot code is somewhere in some other flash or something, so it should at least turn on and boot somehow.
But we're not there yet. I was asking if I can do some disk consistency test somehow. Googling around, I found fsck, but I assume for fully working mode, you need to unmount it first, which is not possible with the root drive. So I expected some read-only checks for a mounted drive, which seems to work:
[23.09-RELEASE][admin@xxx]/root: fsck /dev/gptid/some_guid ** /dev/gptid/some_guid (NO WRITE) ** SU+J Recovering /dev/gptid/some_guid USE JOURNAL? no Skipping journal, falling through to full fsck ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes INCORRECT BLOCK COUNT I=321136 (8 should be 0) CORRECT? no INCORRECT BLOCK COUNT I=321137 (32 should be 0) CORRECT? no ** Phase 2 - Check Pathnames UPDATE FILESYSTEM TO TRACK DIRECTORY DEPTH? no ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts UNREF FILE I=321112 OWNER=root MODE=100666 SIZE=0 MTIME=Nov 14 22:09 2023 CLEAR? no UNREF FILE I=3451132 OWNER=root MODE=100555 SIZE=888368 MTIME=Jun 28 06:48 2023 CLEAR? no ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? no SUMMARY INFORMATION BAD SALVAGE? no BLK(S) MISSING IN BIT MAPS SALVAGE? no 54605 files, 707906 used, 6143057 free (9625 frags, 766679 blocks, 0.1% fragmentation)Do you think these errors are important (without knowing what the two files are) or is that similar to other devices after some years and some power outages?
Ah, and by the way, how can I uninstall the mmc-utils package again? I couldn't find anything for that. pkg doesn't seem to have an uninstall option.
-
Use
pkg remove.I meant install an mSATA drive into the 4860 and switch to booting from that instead. But if you have a complete spare unit it's not really an issue.
