Undefined symbol "__libc_start1@FBSD_1.7" on pfSense Plus 23.01

stephenw10

If you re-installed the repo pkg and it repopulated the symlink then it looks like it might not have the correct cert for the repo.

Try running pkg -d update to see the full error.

Check what cert it's using:

cat /usr/local/etc/pkg.conf

stephenw10

Oh, yeah if you added the symlink manually it probably also needs the pkg.conf updating manually. In 23.01 when set to 'Prev' it should look like:

ABI=FreeBSD:14:amd64
ALTABI=freebsd:14:x86:64
PKG_ENV {
	SSL_CA_CERT_FILE=/etc/ssl/netgate-ca.pem
	SSL_CLIENT_CERT_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-prev-cert.pem
	SSL_CLIENT_KEY_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-prev-key.pem
}

sgw

@stephenw10

Compared /usr/local/etc/pkg.conf with another 23.01 box.

The whole block "PKG_ENV" was missing, added it.

Now:

pkg -d update
DBG(1)[32839]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[32839]> PkgRepo: verifying update for pfSense-core
DBG(1)[32839]> PkgRepo: need forced update of pfSense-core
DBG(1)[32839]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-core/meta.conf
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_aarch64-core/meta.conf with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-core/packagesite.pkg
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_aarch64-core/packagesite.pkg with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching packagesite.pkg: 100%    2 KiB   1.8kB/s    00:01    
DBG(1)[32839]> PkgRepo: extracting packagesite.yaml of repo pfSense-core
DBG(1)[32947]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[32839]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense-core.sqlite'
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
DBG(1)[32839]> PkgRepo: verifying update for pfSense
DBG(1)[32839]> PkgRepo: need forced update of pfSense
DBG(1)[32839]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/meta.conf
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/meta.conf with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
DBG(1)[32839]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/packagesite.pkg
DBG(1)[32839]> opening libfetch fetcher
DBG(1)[32839]> Fetch > libfetch: connecting
DBG(1)[32839]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_01_aarch64-pfSense_plus_v23_01/packagesite.pkg with opts "i"
DBG(1)[32839]> Fetch: fetcher chosen: https
Fetching packagesite.pkg: 100%  143 KiB 146.3kB/s    00:01    
DBG(1)[32839]> PkgRepo: extracting packagesite.yaml of repo pfSense
DBG(1)[33279]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[32839]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense.sqlite'
Processing entries: 100%
pfSense repository update completed. 503 packages processed.
All repositories are up to date.

Looks better, thanks ;-)

How to proceed now? Switch to 23.05 in the GUI?

sgw

@stephenw10

Currently I have:

ABI=FreeBSD:14:aarch64
ALTABI=freebsd:14:aarch64:64

PKG_ENV {
	SSL_CA_CERT_FILE=/etc/ssl/netgate-ca.pem
	SSL_CLIENT_CERT_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-stable-cert.pem
	SSL_CLIENT_KEY_FILE=/usr/local/etc/pfSense/pkg/repos/pfSense-repo-stable-key.pem
}

but as mentioned it seemed to work.

stephenw10

Yes, try switching to 23.05 and see if it allows you to update.

sgw

@stephenw10

Unfortunately not.

Option 13 gives me:

Enter an option: 13

ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
ld-elf.so.1: /usr/local/bin/php: Undefined symbol "__libc_start1@FBSD_1.7"
/usr/local/libexec/pfSense-upgrade: /usr/local/sbin/-update-repos.php: not found
cp: /usr/local/etc/pkg/repos/.conf: No such file or directory
/usr/local/libexec/pfSense-upgrade: /usr/local/sbin/-update-repos.php: not found
failed to update the repository settings!!!
ld-elf.so.1: /usr/local/bin/php-cgi: Undefined symbol "__libc_start1@FBSD_1.7"

even after toggling 23.01 -> 23.05 -> 23.01 -> 23.05 ...

Again "No active remote repositories configured", seems that the symlink is gone again.

stephenw10

Hmm, OK.
What do these show:

pfSense-upgrade -d

pkg-static -d update

sgw

@stephenw10

I think I messed it up even more. Currently php is messed up, GUI disfunctional.

# pfSense-upgrade -d

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:48
Stack trace:
#0 /etc/inc/functions.inc(34): require_once()
#1 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(51): require_once('/etc/inc/notice...')
#3 /usr/local/sbin/pfSense-update-repos.php(23): require_once('/etc/inc/config...')
#4 {main}
  thrown in /etc/inc/certs.inc on line 48
failed to update the repository settings!!!

pkg-static -d update runs through OK.

stephenw10

Hmm, I assume reinstalling this would be difficult?

pkg-static -d update runs successfully against the 23.05 repo?

sgw

@stephenw10

reinstalling would include sending the appliance from one country to the other .. I am a few hundred kms away from that box. I have a 2nd SG1100 there (already on 23.05, no such issues there!), so it's possible, but it would be great to avoid that.

Yes, pkg-static -d update succeeds while:

# /usr/local/etc/pkg/repos: ls -l
total 1
-rw-r--r--  1 root  wheel  26 Feb 10 21:04 FreeBSD.conf
lrwxr-xr-x  1 root  wheel  57 Jun 22 15:18 pfSense.conf -> /usr/local/etc/pfSense/pkg/repos/pfSense-repo-stable.conf

# cat pfSense.conf 
FreeBSD: { enabled: no }

pfSense-core: {
    url: "pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_05_aarch64-core",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/local/share/pfSense/keys/pkg",
    enabled: yes
}

pfSense: {
    url: "pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_05_aarch64-pfSense_plus_v23_05",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/local/share/pfSense/keys/pkg",
    enabled: yes
}

stephenw10

Ah, it's an 1100?

Is it installed as ZFS? You can probably roll back to a snapshot before you tried to upgrade if so.

sgw

@stephenw10 yeah, zfs.

tried to reinstall php stuff and gettext etc ... a mess

gui broken, can I roll back from cli?

stephenw10

Yes, you can using bectl. Or from the loader if you have console access.

First check what there is:

bectl list -s

sgw

@stephenw10

 bectl list -s
BE/Dataset/Snapshot    Active Mountpoint Space Created

default
  pfSense/ROOT/default NR     /          897M  2023-02-10 22:00

nothing else

sgw

Could I get some list of php-packages to reinstall maybe? That might fix something ...

Rolling back is not possible as far as I understand. Looks as if everything has been written into this first boot env.

thanks

stephenw10

Yup no BE snaps other than default, nothing to roll back to.

What pkg versions have you ended up with:

pkg-static info -x pfsense

sgw

@stephenw10 said in Undefined symbol "__libc_start1@FBSD_1.7" on pfSense Plus 23.01:

pkg-static info -x pfsense

# pkg-static info -x pfsense
pfSense-base-23.01
pfSense-boot-23.01
pfSense-default-config-serial-23.01
pfSense-kernel-pfSense-23.01
pfSense-pkg-aws-wizard-0.10
pfSense-pkg-ipsec-profile-wizard-1.1_1
pfSense-pkg-squid-0.4.46
pfSense-rc-23.01
pfSense-repo-23.01
pfSense-u-boot-1100-20220428
pfSense-u-boot-2100-20210930_1
pfSense-u-boot-env-20230123
pfSense-upgrade-1.0_62

thanks for your patience!

stephenw10

Ok those are old versions of pfSense-upgrade.

Try running: pkg-static upgrade pfSense-upgrade

Then try to run that again once it's updated.

sgw

@stephenw10

done:

pkg-static info -x pfsense
pfSense-base-23.01
pfSense-boot-23.01
pfSense-default-config-serial-23.01
pfSense-kernel-pfSense-23.01
pfSense-pkg-aws-wizard-0.10
pfSense-pkg-ipsec-profile-wizard-1.1_1
pfSense-pkg-squid-0.4.46
pfSense-rc-23.01
pfSense-repo-23.01
pfSense-u-boot-1100-20220428
pfSense-u-boot-2100-20210930_1
pfSense-u-boot-env-20230123
pfSense-upgrade-1.0_62
php81-pfSense-module-0.91

sgw

@sgw said in Undefined symbol "__libc_start1@FBSD_1.7" on pfSense Plus 23.01:

php81-pfSense-module-0.91

... comes from my attempts to "fix things".

I suggested the customer to send the appliance, will get his reply tmrw.

I think we can't trust it anymore, right now this:

*** Welcome to Netgate pfSense Plus 23.01-RELEASE (arm64) on xxx ***

Fatal error: Uncaught Error: Call to undefined function ctype_digit() in /etc/inc/util.inc:257
Stack trace:
#0 /etc/inc/interfaces.inc(162): is_numericint('4090')
#1 /etc/inc/interfaces.inc(247): vlan_valid_tag('4090')
#2 /etc/inc/interfaces.inc(5932): interface_is_vlan('mvneta0.4090')
#3 /etc/inc/interfaces.inc(6375): get_real_interface('mvneta0.4090')
#4 /etc/inc/interfaces.inc(6618): find_interface_ipv6('mvneta0.4090', false)
#5 /etc/rc.banner(147): get_interface_ipv6('wan')
#6 {main}
  thrown in /etc/inc/util.inc on line 257
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 257, Message: Uncaught Error: Call to undefined function ctype_digit() in /etc/inc/util.inc:257
Stack trace:
#0 /etc/inc/interfaces.inc(162): is_numericint('4090')
#1 /etc/inc/interfaces.inc(247): vlan_valid_tag('4090')
#2 /etc/inc/interfaces.inc(5932): interface_is_vlan('mvneta0.4090')
#3 /etc/inc/interfaces.inc(6375): get_real_interface('mvneta0.4090')
#4 /etc/inc/interfaces.inc(6618): find_interface_ipv6('mvneta0.4090', false)
#5 /etc/rc.banner(147): get_interface_ipv6('wan')
#6 {main}
  thrown
 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + Netgate pfSense Plus tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell