Pkg - need to re-create database -> Segmentation fault
-
Do you have enough disk space on the flash drive?
-
@kpa:
Try removing the pfSense* and repo-* files from /var/db/pkg, make sure you don't remove the local.sqlite file. Then run 'pkg update -f' again.
With that, I get:
# pkg-static install -f pkg Updating pfSense-core repository catalogue... pkg-static: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory meta.txz : 100% 944 B 0.9kB/s 00:01 Child process pid=28234 terminated abnormally: Segmentation fault -
Do you have enough disk space on the flash drive?
df -h Filesystem Size Used Avail Capacity Mounted on /dev/ufs/pfsense0 1.8G 725M 973M 43% / devfs 1.0K 1.0K 0B 100% /dev /dev/ufs/cf 49M 6.9M 39M 15% /cf /dev/md0 38M 452K 35M 1% /tmp /dev/md1 58M 24M 29M 45% /var devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev devfs 1.0K 1.0K 0B 100% /cf/named/dev -
Well:
$ lldb -c pkg-static.core (lldb) target create --core "pkg-static.core" Core file '/usr/home/dan/pkg-static.core' (x86_64) was loaded. (lldb) bt thread #1: tid = 100238, 0x000000000045014f, name = 'pkg-static', stop reason = signal SIGSEGV frame #0: 0x000000000045014f frame #1: 0x000000000044f1e0 frame #2: 0x000000000044f3ec frame #3: 0x000000000044f00c frame #4: 0x000000000044d791 frame #5: 0x00000000005adcd0 frame #6: 0x000000000041108d frame #7: 0x0000000000409abb frame #8: 0x000000000040ad50 frame #9: 0x000000000040032f (lldb) -
It's unusual for that process to segfault the way it did in your case. It makes me wonder if there might be some other corruption in the filesystem, though usually I'd expect that to panic not just segfault. If you can handle a few minutes of downtime, consider booting into single user mode and running "fsck -y /" a few times until fsck no longer finds anything, might take 3-4 passes, but you probably already know how quirky fsck can be with ufs. :-)
The last time we saw "Repository pfSense-core load error" come up, it was because it had trouble verifying the signature on the repository, but that doesn't quite look like what is happening here. It's possible, but unlikely.
Is there something between this firewall and the Internet in general that might be interfering with its DNS responses and/or HTTP/HTTPS connections?
-
@dvl:
Well:
$ lldb -c pkg-static.core (lldb) target create --core "pkg-static.core" Core file '/usr/home/dan/pkg-static.core' (x86_64) was loaded. (lldb) bt thread #1: tid = 100238, 0x000000000045014f, name = 'pkg-static', stop reason = signal SIGSEGV frame #0: 0x000000000045014f frame #1: 0x000000000044f1e0 frame #2: 0x000000000044f3ec frame #3: 0x000000000044f00c frame #4: 0x000000000044d791 frame #5: 0x00000000005adcd0 frame #6: 0x000000000041108d frame #7: 0x0000000000409abb frame #8: 0x000000000040ad50 frame #9: 0x000000000040032f (lldb)Hi Dan,
I suspect I know what is causing that. Can you please run 'truss pkg update -f' and send me the result?
-
Hi Dan,
I suspect I know what is causing that. Can you please run 'truss pkg update -f' and send me the result?
Send directly via email @FreeBSD.org
-
It's unusual for that process to segfault the way it did in your case. It makes me wonder if there might be some other corruption in the filesystem, though usually I'd expect that to panic not just segfault. If you can handle a few minutes of downtime, consider booting into single user mode and running "fsck -y /" a few times until fsck no longer finds anything, might take 3-4 passes, but you probably already know how quirky fsck can be with ufs. :-)
I can't do downtime during working hours, but FWIW:
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: fsck -n ** /dev/ufs/pfsense0 (NO WRITE) ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 20135 files, 1484442 used, 2295586 free (8426 frags, 285895 blocks, 0.2% fragmentation) ** /dev/ufs/cf (NO WRITE) ** Last Mounted on /cf ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 78 files, 14110 used, 86945 free (113 frags, 10854 blocks, 0.1% fragmentation) -
Is there something between this firewall and the Internet in general that might be interfering with its DNS responses and/or HTTP/HTTPS connections?
The pfSense box is connected to a cable modem, and from there to the world. The cable modem is in bridge mode.
The DNS servers are localhost and two DNS servers in-house.
-
@dvl:
Hi Dan,
I suspect I know what is causing that. Can you please run 'truss pkg update -f' and send me the result?
Send directly via email @FreeBSD.org
It's exactly what I suspected. There is a bug in pkg that causes this crash. You can see details here:
https://github.com/freebsd/pkg/pull/1586
We already fixed it in pfSense before submit the patch to upstream, and based on truss output we can see:
openat(AT_FDCWD,"/usr/local/share/pfSense/keys/pkg/revoked/.empty",O_RDONLY,00) = 8 (0x8) fstat(8,{ mode=-rw-r–r-- ,inode=140382,size=0,blksize=4096 }) = 0 (0x0) close(8) = 0 (0x0) SIGNAL 11 (SIGSEGV) process killed, signal = 11 (core dumped)The presence of the empty file /usr/local/share/pfSense/keys/pkg/revoked/.empty is the cause. I suspect there is a /usr/local/share/pfSense/keys/pkg/trusted/.empty file too. Just remove them and it will work as expected. As soon as you update pkg to latest version bug will be fixed.
-
One file is there. I have removed it.
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/revoked/.empty -rw-r--r-- 1 root wheel 0 Jul 19 2016 /usr/local/share/pfSense/keys/pkg/revoked/.empty [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/trusted/.empty ls: /usr/local/share/pfSense/keys/pkg/trusted/.empty: No such file or directory [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root:After that, I retried the command:
pkg-static install -f pkg Updating pfSense-core repository catalogue... pkg-static: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory meta.txz : 100% 944 B 0.9kB/s 00:01 pkg-static: No trusted public keys found repository pfSense-core has no meta file, using default settings packagesite.txz : 100% 2 KiB 2.3kB/s 00:01 pkg-static: No trusted public keys found Unable to update repository pfSense-core Updating pfSense repository catalogue... pkg-static: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory meta.txz : 100% 944 B 0.9kB/s 00:01 pkg-static: No trusted public keys found repository pfSense has no meta file, using default settings packagesite.txz : 100% 119 KiB 121.8kB/s 00:01 pkg-static: No trusted public keys found Unable to update repository pfSense Error updating repositories!What do we have there?
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /var/db/pkg/ total 4486 -rw-r--r-- 1 root wheel 246 Mar 29 04:38 local.meta -rw-r--r-- 1 root wheel 4578304 Mar 29 23:03 local.sqlite -rw-r--r-- 1 root wheel 246 Aug 4 16:07 pfSense-core.meta -rw-r--r-- 1 root wheel 246 Aug 4 16:07 pfSense.meta [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root:I don't want to try anything else in case there are forensics to be used.
-
Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?
You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.
-
Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?
You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/trusted total 1 -rw-r--r-- 1 root wheel 95 Jul 19 2016 beta.pfsense.org.20151223 -rw-r--r-- 1 root wheel 95 Mar 30 11:34 pkg.pfsense.org.20160406 -
@dvl:
Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?
You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/trusted total 1 -rw-r--r-- 1 root wheel 95 Jul 19 2016 beta.pfsense.org.20151223 -rw-r--r-- 1 root wheel 95 Mar 30 11:34 pkg.pfsense.org.20160406Do you see any error messages when you run 'pkg update -f'? No need to use pkg-static now since the empty file that was causing the crash was removed.
If you see the errors, run 'pkg -dd update -f' and share the output
-
@dvl:
Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?
You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/trusted total 1 -rw-r--r-- 1 root wheel 95 Jul 19 2016 beta.pfsense.org.20151223 -rw-r--r-- 1 root wheel 95 Mar 30 11:34 pkg.pfsense.org.20160406Do you see any error messages when you run 'pkg update -f'? No need to use pkg-static now since the empty file that was causing the crash was removed.
If you see the errors, run 'pkg -dd update -f' and share the output
Here we go
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: pkg -dd update -f DBG(1)[50590]> pkg initialized Updating pfSense-core repository catalogue... DBG(1)[50590]> PkgRepo: verifying update for pfSense-core pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory DBG(1)[50590]> PkgRepo: need forced update of pfSense-core DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite' DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/meta.txz with opts "i" meta.txz : 100% 944 B 0.9kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense-core DBG(1)[50716]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found repository pfSense-core has no meta file, using default settings DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/packagesite.txz with opts "i" packagesite.txz : 100% 2 KiB 2.3kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense-core DBG(1)[50987]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found Unable to update repository pfSense-core Updating pfSense repository catalogue... DBG(1)[50590]> PkgRepo: verifying update for pfSense pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory DBG(1)[50590]> PkgRepo: need forced update of pfSense DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite' DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/meta.txz with opts "i" meta.txz : 100% 944 B 0.9kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense DBG(1)[51205]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found repository pfSense has no meta file, using default settings DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/packagesite.txz with opts "i" packagesite.txz : 100% 119 KiB 121.8kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense DBG(1)[51295]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found Unable to update repository pfSense Error updating repositories! [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: -
@dvl:
@dvl:
Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?
You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/trusted total 1 -rw-r--r-- 1 root wheel 95 Jul 19 2016 beta.pfsense.org.20151223 -rw-r--r-- 1 root wheel 95 Mar 30 11:34 pkg.pfsense.org.20160406Do you see any error messages when you run 'pkg update -f'? No need to use pkg-static now since the empty file that was causing the crash was removed.
If you see the errors, run 'pkg -dd update -f' and share the output
Here we go
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: pkg -dd update -f DBG(1)[50590]> pkg initialized Updating pfSense-core repository catalogue... DBG(1)[50590]> PkgRepo: verifying update for pfSense-core pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory DBG(1)[50590]> PkgRepo: need forced update of pfSense-core DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite' DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/meta.txz with opts "i" meta.txz : 100% 944 B 0.9kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense-core DBG(1)[50716]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found repository pfSense-core has no meta file, using default settings DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/packagesite.txz with opts "i" packagesite.txz : 100% 2 KiB 2.3kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense-core DBG(1)[50987]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found Unable to update repository pfSense-core Updating pfSense repository catalogue... DBG(1)[50590]> PkgRepo: verifying update for pfSense pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory DBG(1)[50590]> PkgRepo: need forced update of pfSense DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite' DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/meta.txz with opts "i" meta.txz : 100% 944 B 0.9kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense DBG(1)[51205]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found repository pfSense has no meta file, using default settings DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/packagesite.txz with opts "i" packagesite.txz : 100% 119 KiB 121.8kB/s 00:01 DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense DBG(1)[51295]> PkgRepo: extracting signature of repo in a sandbox pkg: No trusted public keys found Unable to update repository pfSense Error updating repositories! [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root:Just to make sure you are using the proper keys. What is the content of /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 ?
-
Just to make sure you are using the proper keys. What is the content of /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 ?
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: cat /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 function: sha256 fingerprint: 30be9cc2e7b2b3ba1ff3b2be1795f3f0719ab9a65322695703dc7b8f004035a8 -
@dvl:
Just to make sure you are using the proper keys. What is the content of /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 ?
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: cat /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 function: sha256 fingerprint: 30be9cc2e7b2b3ba1ff3b2be1795f3f0719ab9a65322695703dc7b8f004035a8Interesting, you have CE keys but your system is configured to use factory repositories. I just don't know how this key ended up there. Did you make any custom change on this system? Is it expected to be a factory system or CE?
-
@dvl:
Just to make sure you are using the proper keys. What is the content of /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 ?
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: cat /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 function: sha256 fingerprint: 30be9cc2e7b2b3ba1ff3b2be1795f3f0719ab9a65322695703dc7b8f004035a8Interesting, you have CE keys but your system is configured to use factory repositories. I just don't know how this key ended up there. Did you make any custom change on this system? Is it expected to be a factory system or CE?
CE is community edition, I'm guessing.
pfSense was installed when I got this box (Netgate APU2).
Does that answer?
-
@dvl:
@dvl:
Just to make sure you are using the proper keys. What is the content of /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 ?
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: cat /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 function: sha256 fingerprint: 30be9cc2e7b2b3ba1ff3b2be1795f3f0719ab9a65322695703dc7b8f004035a8Interesting, you have CE keys but your system is configured to use factory repositories. I just don't know how this key ended up there. Did you make any custom change on this system? Is it expected to be a factory system or CE?
CE is community edition, I'm guessing.
pfSense was installed when I got this box (Netgate APU2).
Does that answer?
Yes, CE is community edition.
If you are using Netgate hardware you are supposed to be running factory. Now the question is how the CE key ended up on your system. Did you install any package manually? changed anything while testing?
I'll send you factory public key.
