Undefined symbol "__libc_start1@FBSD_1.7" on pfSense Plus 23.01

lucasbender

If others need to install a package without upgrading/crashing pkg, i had success by lock pkg, install needed package, unlock pkg and then disable the FreeBSD repository to avoid any future issues on pfSense:

[23.01-RELEASE][root@router.domain]/root: pkg lock pkg
pkg-1.18.4_4: lock this package? [y/N]: y
Locking pkg-1.18.4_4
[23.01-RELEASE][root@router.domain]/root: pkg install autossh
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        autossh: 1.4g [FreeBSD]

Number of packages to be installed: 1

24 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching autossh-1.4g.pkg: 100%   24 KiB  24.3kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing autossh-1.4g...
[1/1] Extracting autossh-1.4g: 100%
[23.01-RELEASE][root@router.domain]/root: pkg unlock pkg
pkg-1.18.4_4: unlock this package? [y/N]: y
Unlocking pkg-1.18.4_4

martys

@lucasbender I do appreciate the success you had, and wish it had worked for me… but it did not. Here's what I got:

[23.01-RELEASE][admin@gateway.home]/root: pkg-static install -f pkg-1.18.4_4
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
pkg-static: No packages available to install matching 'pkg-1.18.4_4' have been found in the repositories

[23.01-RELEASE][admin@gateway.home]/root: pkg-static info pkg
pkg-1.19.1_1
Name           : pkg
Version        : 1.19.1_1
Installed on   : Tue May 30 22:31:25 2023 EDT
Origin         : ports-mgmt/pkg
Architecture   : FreeBSD:14:amd64
Prefix         : /usr/local
Categories     : ports-mgmt
Licenses       : BSD2CLAUSE
Maintainer     : pkg@FreeBSD.org
WWW            : https://github.com/freebsd/pkg
Comment        : Package manager
Options        :
	DOCS           : off
Shared Libs provided:
	libpkg.so.4
Annotations    :
	FreeBSD_version: 1400085
	build_timestamp: 2023-05-23T17:56:06+0000
	built_by       : poudriere-git-3.3.99.20220831
	port_checkout_unclean: no
	port_git_hash  : 38f48ce64acc
	ports_top_checkout_unclean: yes
	ports_top_git_hash: 5ceffeaef96e
	repo_type      : binary
	repository     : pfSense
Flat size      : 34.1MiB
Description    :
Package management tool

WWW: https://github.com/freebsd/pkg

Notice how there is no mention about FreeBSD repository (2 lines in your output). And pkg-static seems to think this 'bad' version of pkg is in the PfSense repository.

I will admit that while I know lots of other package managers very well, I know next to nothing about pkg. So I'm probably missing something obvious or reading more into the output I'm getting than I should.

stephenw10

Did you set the repo to 23.01 before doing that?

Since 23.05 was released it will have become the default and pulled in pkg from that. It's not usually an issue, the upgrade process uses pkg-static for everything for just that reason.

martys

@stephenw10 said in Undefined symbol "__libc_start1@FBSD_1.7" on pfSense Plus 23.01:

Did you set the repo to 23.01 before doing that?

Since 23.05 was released it will have become the default and pulled in pkg from that. It's not usually an issue, the upgrade process uses pkg-static for everything for just that reason.

Now how was I to know to do that? I'm just minding my business running 23.01 and without using the System > Update GUI out comes a newer version for me to eventually choose… and something in the background seems to have chosen to spring forward the CLI's repository view to the newer version that I'm not ready for, and upgraded pkg to match.

You're saying to visit the System > Update GUI to 'drop back' was the final key to getting pkg back. This is probably one of the unwritten things about the upgrade notes warning about not upgrading packages before an upgrade.

So after dropping back the setting to 23.01 and reinstalling from pkg-1.18.4_4, all is well on my installation.

I do thank you all for responding as quickly, and completely, as you did. Here's to a healthy community of users!

/Marty

stephenw10

It shouldn't matter that pkg has been updated. It's only an issue if you're using pkg at the CLI in which case you will see that error if you don't use pkg-static.

sgw

I somehow lost my active repos. Toggling back and forth between 23.01 and 23.05 doesn't help.
I wonder if I could scp some files from a working machine into the system. Or somehow fix a symlink or so.

[23.01-RELEASE][root@some.host]/root: pkg-static clean -ay ; pkg-static install -fy pkg pfSense-repo pfSense-upgrade
pkg-static: No active remote repositories configured
pkg-static: No packages available to install matching 'pfSense-repo' have been found in the repositories
pkg-static: No packages available to install matching 'pfSense-upgrade' have been found in the repositories

Unfortunately this box is in a protected environment, I can't let Netgate support in right now.

help appreciated, thanks

sgw

Some more infos:

# pkg --version
ld-elf.so.1: /usr/local/sbin/pkg: Undefined symbol "__libc_start1@FBSD_1.7"

# pkg-static install -f pkg-1.18.4_4
pkg-static: No packages available to install matching 'pkg-1.18.4_4' have been found in the repositories

According to the GUI I am on 23.01 branch ... but some repo-information seems to be lost right now.

stephenw10

Try running: pfSense-repoc-static

That should pull in repo data. Or show an error.

sgw

@stephenw10 "Command not found"

I managed to downgrade "pkg" as far as I see. And I reinstalled the repos, and somehow fixed that symlink:

ln -s /usr/local/etc/pfSense/pkg/repos/pfSense-repo-prev.conf pfSense.conf

I assume I should get things right on 23.01 at first?

Currently:

# pkg update
Updating pfSense-core repository catalogue...
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg01.atx.netgate.com
2427236352:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/ssl/statem/statem_clnt.c:1921:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg01.atx.netgate.com
2427236352:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/ssl/statem/statem_clnt.c:1921:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg01.atx.netgate.com
2427236352:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/ssl/statem/statem_clnt.c:1921:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg01.atx.netgate.com

[..]

thanks ...

stephenw10

If you re-installed the repo pkg and it repopulated the symlink then it looks like it might not have the correct cert for the repo.

Try running pkg -d update to see the full error.

Check what cert it's using:

cat /usr/local/etc/pkg.conf

stephenw10

Oh, yeah if you added the symlink manually it probably also needs the pkg.conf updating manually. In 23.01 when set to 'Prev' it should look like:

ABI=FreeBSD:14:amd64
ALTABI=freebsd:14:x86:64
PKG_ENV {
	SSL_CA_CERT_FILE=/etc/ssl/netgate-ca.pem
	SSL_CLIENT_CERT_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-prev-cert.pem
	SSL_CLIENT_KEY_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-prev-key.pem
}

sgw

@stephenw10

Compared /usr/local/etc/pkg.conf with another 23.01 box.

The whole block "PKG_ENV" was missing, added it.

Now:

pkg -d update
DBG(1)[32839]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[32839]> PkgRepo: verifying update for pfSense-core
DBG(1)[32839]> PkgRepo: need forced update of pfSense-core
DBG(1)[32839]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-core/meta.conf
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_aarch64-core/meta.conf with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-core/packagesite.pkg
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_aarch64-core/packagesite.pkg with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching packagesite.pkg: 100%    2 KiB   1.8kB/s    00:01    
DBG(1)[32839]> PkgRepo: extracting packagesite.yaml of repo pfSense-core
DBG(1)[32947]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[32839]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense-core.sqlite'
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
DBG(1)[32839]> PkgRepo: verifying update for pfSense
DBG(1)[32839]> PkgRepo: need forced update of pfSense
DBG(1)[32839]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/meta.conf
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/meta.conf with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/packagesite.pkg
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/packagesite.pkg with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching packagesite.pkg: 100%  143 KiB 146.3kB/s    00:01    
DBG(1)[32839]> PkgRepo: extracting packagesite.yaml of repo pfSense
DBG(1)[33279]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[32839]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense.sqlite'
Processing entries: 100%
pfSense repository update completed. 503 packages processed.
All repositories are up to date.

Looks better, thanks ;-)

How to proceed now? Switch to 23.05 in the GUI?

sgw

@stephenw10

Currently I have:

ABI=FreeBSD:14:aarch64
ALTABI=freebsd:14:aarch64:64

PKG_ENV {
	SSL_CA_CERT_FILE=/etc/ssl/netgate-ca.pem
	SSL_CLIENT_CERT_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-stable-cert.pem
	SSL_CLIENT_KEY_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-stable-key.pem
}

but as mentioned it seemed to work.

stephenw10

Yes, try switching to 23.05 and see if it allows you to update.

sgw

@stephenw10

Unfortunately not.

Option 13 gives me:

Enter an option: 13

ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
/usr/local/libexec/pfSense-upgrade: /usr/local/sbin/-update-repos.php: not found
cp: /usr/local/etc/pkg/repos/.conf: No such file or directory
/usr/local/libexec/pfSense-upgrade: /usr/local/sbin/-update-repos.php: not found
failed to update the repository settings!!!
ld-elf.so.1: /usr/local/bin/php-cgi: Undefined symbol "__libc_start1@FBSD_1.7"

even after toggling 23.01 -> 23.05 -> 23.01 -> 23.05 ...

Again "No active remote repositories configured", seems that the symlink is gone again.

stephenw10

Hmm, OK.
What do these show:

pfSense-upgrade -d

pkg-static -d update

sgw

@stephenw10

I think I messed it up even more. Currently php is messed up, GUI disfunctional.

# pfSense-upgrade -d

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(51): require_once('/etc/inc/notice...')
#3 /usr/local/sbin/pfSense-update-repos.php(23): require_once('/etc/inc/config...')
#4 {main}
  thrown in /etc/inc/certs.inc on line 48
failed to update the repository settings!!!

pkg-static -d update runs through OK.

stephenw10

Hmm, I assume reinstalling this would be difficult?

pkg-static -d update runs successfully against the 23.05 repo?

sgw

@stephenw10

reinstalling would include sending the appliance from one country to the other .. I am a few hundred kms away from that box. I have a 2nd SG1100 there (already on 23.05, no such issues there!), so it's possible, but it would be great to avoid that.

Yes, pkg-static -d update succeeds while:

# /usr/local/etc/pkg/repos: ls -l
total 1
-rw-r--r--  1 root  wheel  26 Feb 10 21:04 FreeBSD.conf
lrwxr-xr-x  1 root  wheel  57 Jun 22 15:18 pfSense.conf -> /usr/local/etc/pfSense/pkg/repos/pfSense-repo-stable.conf

# cat pfSense.conf 
FreeBSD: { enabled: no }

pfSense-core: {
    url: "pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_05_aarch64-core",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/local/share/pfSense/keys/pkg",
    enabled: yes
}

pfSense: {
    url: "pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_05_aarch64-pfSense_plus_v23_05",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/local/share/pfSense/keys/pkg",
    enabled: yes
}

stephenw10

Ah, it's an 1100?

Is it installed as ZFS? You can probably roll back to a snapshot before you tried to upgrade if so.