Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Pkg - need to re-create database -> Segmentation fault

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    29 Posts 6 Posters 15.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      dvl
      last edited by

      I'm on:

      10.3-RELEASE-p9 FreeBSD 10.3-RELEASE-p9 #1 5fc1b19(RELENG_2_3_2): Tue Sep 27 12:25:49 CDT 2016     root@factory23-amd64-builder:/builder/factory-232/tmp/obj/builder/factory-232/tmp/FreeBSD-src/sys/pfSense  amd64

      I'm trying option 13 from the ssh menu:

      Enter an option: 13

      >>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
pkg: Repository pfSense-core has a wrong packagesite, need to re-create database
meta.txz                                 : . done
Child process pid=92513 terminated abnormally: Segmentation fault
*** Welcome to pfSense 2.3.2-RELEASE-p1 (amd64 nanobsd) on bast ***

      I have tried 'pkg update -f' from the command line, it gives exactly the same message.

      From pkg -vv:

      
Repositories:
  pfSense-core: { 
    url             : "pkg+https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/local/share/pfSense/keys/pkg"
  }
  pfSense: { 
    url             : "pkg+https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/local/share/pfSense/keys/pkg"
  }

      Jump here for the cause/solution: https://forum.pfsense.org/index.php?topic=128154.msg738541#msg738541

      1 Reply Last reply Reply Quote 0
      • D Offline
        dvl
        last edited by

        I have /root/pkg.core if someone wants that.

        1 Reply Last reply Reply Quote 0
        • K Offline
          kpa
          last edited by

          Try removing the pfSense* and repo-* files from /var/db/pkg, make sure you don't remove the local.sqlite file. Then run 'pkg update -f' again.

          1 Reply Last reply Reply Quote 0
          • H Offline
            heper
            last edited by

            Do you have enough disk space on the flash drive?

            1 Reply Last reply Reply Quote 0
            • D Offline
              dvl
              last edited by

              @kpa:

              Try removing the pfSense* and repo-* files from /var/db/pkg, make sure you don't remove the local.sqlite file. Then run 'pkg update -f' again.

              With that, I get:

              # pkg-static install -f pkg
Updating pfSense-core repository catalogue...
pkg-static: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
meta.txz                                 : 100%  944 B     0.9kB/s    00:01    
Child process pid=28234 terminated abnormally: Segmentation fault
              1 Reply Last reply Reply Quote 0
              • D Offline
                dvl
                last edited by

                @heper:

                Do you have enough disk space on the flash drive?

                df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/pfsense0    1.8G    725M    973M    43%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/ufs/cf           49M    6.9M     39M    15%    /cf
/dev/md0              38M    452K     35M     1%    /tmp
/dev/md1              58M     24M     29M    45%    /var
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev
devfs                1.0K    1.0K      0B   100%    /cf/named/dev
                1 Reply Last reply Reply Quote 0
                • D Offline
                  dvl
                  last edited by

                  Well:

                  $ lldb -c pkg-static.core
(lldb) target create --core "pkg-static.core"
Core file '/usr/home/dan/pkg-static.core' (x86_64) was loaded.
(lldb) bt

thread #1: tid = 100238, 0x000000000045014f, name = 'pkg-static', stop reason = signal SIGSEGV
frame #0: 0x000000000045014f
frame #1: 0x000000000044f1e0
frame #2: 0x000000000044f3ec
frame #3: 0x000000000044f00c
frame #4: 0x000000000044d791
frame #5: 0x00000000005adcd0
frame #6: 0x000000000041108d
frame #7: 0x0000000000409abb
frame #8: 0x000000000040ad50
frame #9: 0x000000000040032f
(lldb)
                  1 Reply Last reply Reply Quote 0
                  • jimpJ Offline
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    It's unusual for that process to segfault the way it did in your case. It makes me wonder if there might be some other corruption in the filesystem, though usually I'd expect that to panic not just segfault. If you can handle a few minutes of downtime, consider booting into single user mode and running "fsck -y /" a few times until fsck no longer finds anything, might take 3-4 passes, but you probably already know how quirky fsck can be with ufs. :-)

                    The last time we saw "Repository pfSense-core load error" come up, it was because it had trouble verifying the signature on the repository, but that doesn't quite look like what is happening here. It's possible, but unlikely.

                    Is there something between this firewall and the Internet in general that might be interfering with its DNS responses and/or HTTP/HTTPS connections?

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • rbgargaR Offline
                      rbgarga Developer Netgate Administrator
                      last edited by

                      @dvl:

                      Well:

                      $ lldb -c pkg-static.core
(lldb) target create --core "pkg-static.core"
Core file '/usr/home/dan/pkg-static.core' (x86_64) was loaded.
(lldb) bt

thread #1: tid = 100238, 0x000000000045014f, name = 'pkg-static', stop reason = signal SIGSEGV
frame #0: 0x000000000045014f
frame #1: 0x000000000044f1e0
frame #2: 0x000000000044f3ec
frame #3: 0x000000000044f00c
frame #4: 0x000000000044d791
frame #5: 0x00000000005adcd0
frame #6: 0x000000000041108d
frame #7: 0x0000000000409abb
frame #8: 0x000000000040ad50
frame #9: 0x000000000040032f
(lldb)

                      Hi Dan,

                      I suspect I know what is causing that. Can you please run 'truss pkg update -f' and send me the result?

                      Renato Botelho

                      1 Reply Last reply Reply Quote 0
                      • D Offline
                        dvl
                        last edited by

                        @Renato:

                        Hi Dan,

                        I suspect I know what is causing that. Can you please run 'truss pkg update -f' and send me the result?

                        Send directly via email @FreeBSD.org

                        1 Reply Last reply Reply Quote 0
                        • D Offline
                          dvl
                          last edited by

                          @jimp:

                          It's unusual for that process to segfault the way it did in your case. It makes me wonder if there might be some other corruption in the filesystem, though usually I'd expect that to panic not just segfault. If you can handle a few minutes of downtime, consider booting into single user mode and running "fsck -y /" a few times until fsck no longer finds anything, might take 3-4 passes, but you probably already know how quirky fsck can be with ufs. :-)

                          I can't do downtime during working hours, but FWIW:

                          [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: fsck -n
** /dev/ufs/pfsense0 (NO WRITE)
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
20135 files, 1484442 used, 2295586 free (8426 frags, 285895 blocks, 0.2% fragmentation)
** /dev/ufs/cf (NO WRITE)
** Last Mounted on /cf
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
78 files, 14110 used, 86945 free (113 frags, 10854 blocks, 0.1% fragmentation)
                          1 Reply Last reply Reply Quote 0
                          • D Offline
                            dvl
                            last edited by

                            @jimp:

                            Is there something between this firewall and the Internet in general that might be interfering with its DNS responses and/or HTTP/HTTPS connections?

                            The pfSense box is connected to a cable modem, and from there to the world.  The cable modem is in bridge mode.

                            The DNS servers are localhost and two DNS servers in-house.

                            1 Reply Last reply Reply Quote 0
                            • rbgargaR Offline
                              rbgarga Developer Netgate Administrator
                              last edited by

                              @dvl:

                              @Renato:

                              Hi Dan,

                              I suspect I know what is causing that. Can you please run 'truss pkg update -f' and send me the result?

                              Send directly via email @FreeBSD.org

                              It's exactly what I suspected. There is a bug in pkg that causes this crash. You can see details here:

                              https://github.com/freebsd/pkg/pull/1586

                              We already fixed it in pfSense before submit the patch to upstream, and based on truss output we can see:

                              
openat(AT_FDCWD,"/usr/local/share/pfSense/keys/pkg/revoked/.empty",O_RDONLY,00) = 8 (0x8)
fstat(8,{ mode=-rw-r–r-- ,inode=140382,size=0,blksize=4096 }) = 0 (0x0)
close(8)					 = 0 (0x0)
SIGNAL 11 (SIGSEGV)
process killed, signal = 11 (core dumped)

                              The presence of the empty file /usr/local/share/pfSense/keys/pkg/revoked/.empty is the cause. I suspect there is a /usr/local/share/pfSense/keys/pkg/trusted/.empty file too. Just remove them and it will work as expected. As soon as you update pkg to latest version bug will be fixed.

                              Renato Botelho

                              1 Reply Last reply Reply Quote 0
                              • D Offline
                                dvl
                                last edited by

                                One file is there. I have removed it.

                                [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/revoked/.empty
-rw-r--r--  1 root  wheel  0 Jul 19  2016 /usr/local/share/pfSense/keys/pkg/revoked/.empty

[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /usr/local/share/pfSense/keys/pkg/trusted/.empty
ls: /usr/local/share/pfSense/keys/pkg/trusted/.empty: No such file or directory
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root:

                                After that, I retried the command:

                                 pkg-static install -f pkg
Updating pfSense-core repository catalogue...
pkg-static: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
meta.txz                                 : 100%  944 B     0.9kB/s    00:01    
pkg-static: No trusted public keys found
repository pfSense-core has no meta file, using default settings
packagesite.txz                          : 100%    2 KiB   2.3kB/s    00:01    
pkg-static: No trusted public keys found
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg-static: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
meta.txz                                 : 100%  944 B     0.9kB/s    00:01    
pkg-static: No trusted public keys found
repository pfSense has no meta file, using default settings
packagesite.txz                          : 100%  119 KiB 121.8kB/s    00:01    
pkg-static: No trusted public keys found
Unable to update repository pfSense
Error updating repositories!

                                What do we have there?

                                [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l /var/db/pkg/
total 4486
-rw-r--r--  1 root  wheel      246 Mar 29 04:38 local.meta
-rw-r--r--  1 root  wheel  4578304 Mar 29 23:03 local.sqlite
-rw-r--r--  1 root  wheel      246 Aug  4 16:07 pfSense-core.meta
-rw-r--r--  1 root  wheel      246 Aug  4 16:07 pfSense.meta
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root:

                                I don't want to try anything else in case there are forensics to be used.

                                1 Reply Last reply Reply Quote 0
                                • jimpJ Offline
                                  jimp Rebel Alliance Developer Netgate
                                  last edited by

                                  Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?

                                  You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.

                                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                  Need help fast? Netgate Global Support!

                                  Do not Chat/PM for help!

                                  1 Reply Last reply Reply Quote 0
                                  • D Offline
                                    dvl
                                    last edited by

                                    @jimp:

                                    Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?

                                    You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.

                                    [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l  /usr/local/share/pfSense/keys/pkg/trusted
total 1
-rw-r--r--  1 root  wheel  95 Jul 19  2016 beta.pfsense.org.20151223
-rw-r--r--  1 root  wheel  95 Mar 30 11:34 pkg.pfsense.org.20160406
                                    1 Reply Last reply Reply Quote 0
                                    • rbgargaR Offline
                                      rbgarga Developer Netgate Administrator
                                      last edited by

                                      @dvl:

                                      @jimp:

                                      Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?

                                      You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.

                                      [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l  /usr/local/share/pfSense/keys/pkg/trusted
total 1
-rw-r--r--  1 root  wheel  95 Jul 19  2016 beta.pfsense.org.20151223
-rw-r--r--  1 root  wheel  95 Mar 30 11:34 pkg.pfsense.org.20160406

                                      Do you see any error messages when you run 'pkg update -f'? No need to use pkg-static now since the empty file that was causing the crash was removed.

                                      If you see the errors, run 'pkg -dd update -f' and share the output

                                      Renato Botelho

                                      1 Reply Last reply Reply Quote 0
                                      • D Offline
                                        dvl
                                        last edited by

                                        @Renato:

                                        @dvl:

                                        @jimp:

                                        Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?

                                        You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.

                                        [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l  /usr/local/share/pfSense/keys/pkg/trusted
total 1
-rw-r--r--  1 root  wheel  95 Jul 19  2016 beta.pfsense.org.20151223
-rw-r--r--  1 root  wheel  95 Mar 30 11:34 pkg.pfsense.org.20160406

                                        Do you see any error messages when you run 'pkg update -f'? No need to use pkg-static now since the empty file that was causing the crash was removed.

                                        If you see the errors, run 'pkg -dd update -f' and share the output

                                        Here we go

                                        [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: pkg -dd update -f
DBG(1)[50590]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[50590]> PkgRepo: verifying update for pfSense-core
pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
DBG(1)[50590]> PkgRepo: need forced update of pfSense-core
DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/meta.txz with opts "i"
meta.txz                                 : 100%  944 B     0.9kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense-core
DBG(1)[50716]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
repository pfSense-core has no meta file, using default settings
DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/packagesite.txz with opts "i"
packagesite.txz                          : 100%    2 KiB   2.3kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense-core
DBG(1)[50987]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
DBG(1)[50590]> PkgRepo: verifying update for pfSense
pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
DBG(1)[50590]> PkgRepo: need forced update of pfSense
DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/meta.txz with opts "i"
meta.txz                                 : 100%  944 B     0.9kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense
DBG(1)[51205]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
repository pfSense has no meta file, using default settings
DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/packagesite.txz with opts "i"
packagesite.txz                          : 100%  119 KiB 121.8kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense
DBG(1)[51295]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
Unable to update repository pfSense
Error updating repositories!
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root:
                                        1 Reply Last reply Reply Quote 0
                                        • rbgargaR Offline
                                          rbgarga Developer Netgate Administrator
                                          last edited by

                                          @dvl:

                                          @Renato:

                                          @dvl:

                                          @jimp:

                                          Do you have key files in /usr/local/share/pfSense/keys/pkg/trusted/?

                                          You should have, for example, pkg.pfsense.org.20160406 and beta.pfsense.org.20151223 in there.

                                          [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: ls -l  /usr/local/share/pfSense/keys/pkg/trusted
total 1
-rw-r--r--  1 root  wheel  95 Jul 19  2016 beta.pfsense.org.20151223
-rw-r--r--  1 root  wheel  95 Mar 30 11:34 pkg.pfsense.org.20160406

                                          Do you see any error messages when you run 'pkg update -f'? No need to use pkg-static now since the empty file that was causing the crash was removed.

                                          If you see the errors, run 'pkg -dd update -f' and share the output

                                          Here we go

                                          [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: pkg -dd update -f
DBG(1)[50590]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[50590]> PkgRepo: verifying update for pfSense-core
pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
DBG(1)[50590]> PkgRepo: need forced update of pfSense-core
DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/meta.txz with opts "i"
meta.txz                                 : 100%  944 B     0.9kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense-core
DBG(1)[50716]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
repository pfSense-core has no meta file, using default settings
DBG(1)[50590]> Fetch: fetching from: https://www.firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-core/packagesite.txz with opts "i"
packagesite.txz                          : 100%    2 KiB   2.3kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense-core
DBG(1)[50987]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
DBG(1)[50590]> PkgRepo: verifying update for pfSense
pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
DBG(1)[50590]> PkgRepo: need forced update of pfSense
DBG(1)[50590]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/meta.txz with opts "i"
meta.txz                                 : 100%  944 B     0.9kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting meta of repo pfSense
DBG(1)[51205]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
repository pfSense has no meta file, using default settings
DBG(1)[50590]> Fetch: fetching from: https://firmware.netgate.com/pkg/pfSense_factory-v2_3_3_amd64-pfSense_factory-v2_3_3/packagesite.txz with opts "i"
packagesite.txz                          : 100%  119 KiB 121.8kB/s    00:01    
DBG(1)[50590]> PkgRepo: extracting packagesite.yaml of repo pfSense
DBG(1)[51295]> PkgRepo: extracting signature of repo in a sandbox
pkg: No trusted public keys found
Unable to update repository pfSense
Error updating repositories!
[2.3.2-RELEASE][admin@bast.int.unixathome.org]/root:

                                          Just to make sure you are using the proper keys. What is the content of /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 ?

                                          Renato Botelho

                                          1 Reply Last reply Reply Quote 0
                                          • D Offline
                                            dvl
                                            last edited by

                                            @Renato:

                                            Just to make sure you are using the proper keys. What is the content of /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406 ?

                                            [2.3.2-RELEASE][admin@bast.int.unixathome.org]/root: cat /usr/local/share/pfSense/keys/pkg/trusted/pkg.pfsense.org.20160406
function: sha256
fingerprint: 30be9cc2e7b2b3ba1ff3b2be1795f3f0719ab9a65322695703dc7b8f004035a8
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.