Upgraded to 2.7.0. Now pkg and pkg-static segfault, even after power cycle.

Piquan

After upgrading to 2.7.0, pkg, pkg-static, and sudo all give Segmentation fault (core dumped), with no other output. Most other programss seem to be working ok now. A ktrace of pkg-static indicates no system calls before the segfault:

26883 ktrace   RET   ktrace 0
26883 ktrace   CALL  execve(0xa0edc37d59a,0xa0edc37d260,0xa0edc37d270)
26883 ktrace   NAMI  "/usr/local/sbin/pkg-static"
26883 pkg-static RET   execve JUSTRETURN
26883 pkg-static PSIG  SIGSEGV SIG_DFL code=SEGV_MAPERR
26883 pkg-static NAMI  "/root/pkg-static.core"

Similarly, pkg and sudo don't have any system calls other than loading their shared objects and processing libmap.d.

When I first upgraded, it did not complete its reboot, but went fine after a power cycle. I don’t see anything of concern in /conf/upgrade_log.latest.txt.

I had not removed or upgraded the packages yet (my mistake), so had to move squid.inc away (it had some constructs that current PHP doesn’t accept). Initially, lighttpd also segfaulted, but started working again. Currently, snort is segfaulting, but I don’t mind about that for now.

I saw in Troubleshooting Upgrades that some crypto hardware can cause pkg to segfault; the example message they give includes some OpenSSL logs:

## Example from the webpage
1085486128:error:14099044:SSL routines:ssl3_send_client_verify:internal error:
Child process pid=30149 terminated abnormally: Segmentation fault

Here, I’m not seeing that; just Segmentation fault (core dumped). That said, here’s a summary of my CPU:

Origin="AuthenticAMD"  Id=0x500f20  Family=0x14  Model=0x2  Stepping=0
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x802209<SSE3,MON,SSSE3,CX16,POPCNT>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x35ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,IBS,SKINIT,WDT>
SVM: NP,NRIP,NAsids=8
TSC: P-state invariant, performance statistics

or, if you prefer, this from the summary page:

AMD G-T40E Processor
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: No
QAT Crypto: No

I did note that the cryptodev KLD was loaded. I disabled cryptographic hardware support in System > Advanced and rebooted, and while cryptodev was no longer loaded, it had no other apparent effect.

Does anybody have any suggestions about how I might troubleshoot this? It’s going to be interesting to try to fix a machine without pkg!

stephenw10

The crypto device issue would not affect that device.

If you look in /root what cores are actually being dumped?

Did it actually complete the upgrade?

If you have the option I would just reinstall 2.7 clean and restore your config to be sure.

Steve

Piquan

@stephenw10 Sorry I didn't get back to you; I must have missed the reply.

I saw cores for lighttpd, pkg-static, sudo, and I think squidGuard and snort.

It looked like everything went through, according to /conf/upgrade_log.latest.txt, but the last reboot didn't finish. I had to manually reset it. After that, it said it was fine.

I had to put this on hold for a bit (I didn't have a USB-serial adapter handy), but eventually got it working without a reinstall:

I happened across pkg-static.pkgsave, which did work. (It has the same md5sum as the pkg-static on the installation CD.)
I used the command prompt at /diag_command.php, which I discovered runs commands as root, to create an ersatz su (a copy of /bin/sh that's mode 4755)
I used that to run pkg-static.pkgsave bootstrap -f to reinstall pkg.
The pkg command, because of a quirk of implementation, can’t update the packages database when run under my ersatz su (it uses faccessat to check the database, which checks with the real uid, not the euid). But, now that pkg was working, I could use the GUI to reinstall sudo.
Using sudo, I was able to run pkg-static install -fy pkg pfSense-repo pfSense-upgrade and pkg-static upgrade -f.
I later found all of my add-on packages were locked; the pkg command and the GUI would just note that they were updating the database. They wouldn’t upgrade the packages, or print a message. I ended up using pkg unlock on them.

I know this is all pretty rough-and-ready work, and I’ll need to do a clean install to make sure that everything really is cleaned up. But this at least got me going until I can afford the downtime to do a clean install.

The key here had been pkg-static.pkgsave; I only happened across it by accident while I was tab-completing. It may be valuable to put a note about this on the Troubleshooting Upgrades page.