after upgrade to 24.11: squid doesn´t start
-
@stephenw10 said in after upgrade to 24.11: squid doesn´t start:
pkg info -x squid
this is mine and what I suspect everyone elses with version applied via the package manager:
pfSense-pkg-squid-0.5 squid-6.10 squid_radius_auth-1.10 squidclamav-7.3_2is there a way to pull from the git commit that was mentioned or is it a waiting game for the next release?
-
You can apply both using the System Patches pkg, just create patches from the commits.
However that won't help with the undefined symbol errors some users are seeing.
-
D dauhee referenced this topic on
-
Thread got sort of hijacked with another problem... but I'm here to tell that the exact same thing also happened to me.
/pkg_edit.php: The command '/usr/local/sbin/squid -z -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was 'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'After the first boot after upgrading to 24.11, Squid doesn't work anymore, no matter what I do. I've tried:
- Changed a lot of settings;
- Reset squid settings to default;
- Uninstalled squid completely...
- ...reinstall via gui
- ...or command line:
pkg install -f pfSense-pkg-squid-0.5 squid-6.10 squid_radius_auth-1.10 squidclamav-7.3_2
Nothing. Nada :(
Any help?
-
Also in x86?
-
@stephenw10 yes, amd64
-
I have to be completely honest with you, I have not updated for a while because of issues with this package, it is my favorite package to tinker with.
-
@marcosm @stephenw10 said in after upgrade to 24.11: squid doesn´t start:
Also in x86?
Hi! Have to chime in: I was called into a "prod down incident" of one of our bigger customers. Same problem as above: Customer updated their clusters and on all 2nd machines where they failed over to test, Squid didn't came up. Same ELF linker problem as described above, exactly same output:
@intellq said in after upgrade to 24.11: squid doesn´t start:
'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'Same failure description and path: System is a x86-64 platform, was running 23.09_1 / 24.03 with squid before and had no problems. We also tried steps above like checking the package version and in addition to that I tried removing, purging, clearing the pkg cache and re-downloading all packages related to squid to check if that ld-elf error was related to a broken/defective package somewhere, but got the same error after reinstalling/redownloading the packages anyway.
- Is there any way we can get this escalated or checked?
- Any way we could help or contribute this problem?
- what's the official status now about the Squid package in general? As the last official statement is still the blog from last year citing the package will be removed in 24.03 but here we are in 24.11 and still have the package (which IS a GOOD thing, we have many customers that WANT/NEED it). But after today they want a statement about the future of the package and if it really stays in the repo or if the breakage is the first sign of "it's gonna be removed"? Would be nice to have something official as to what's the way forward.
Temporary solution was to roll back all systems to 24.03 via snapshots (thank god for them :)) but that's not really a final solution
Cheers :)
Edit: added information
Linking error and linker dependencies: [24.11-RELEASE][admin@pfSense-test.nh00.local]/root: /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE" [24.11-RELEASE][admin@pfSense-test.nh00.local]/root: ldd /usr/local/sbin/squid /usr/local/sbin/squid: librt.so.1 => /usr/lib/librt.so.1 (0x1a48bf1b9000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x1a48be888000) libregex.so.1 => /usr/lib/libregex.so.1 (0x1a48bf64e000) libcrypto.so.30 => /lib/libcrypto.so.30 (0x1a48c0686000) libssl.so.30 => /usr/lib/libssl.so.30 (0x1a48c2702000) libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x1a48c150e000) libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x1a48bf822000) libm.so.5 => /lib/libm.so.5 (0x1a48c241e000) libdl.so.1 => /usr/lib/libdl.so.1 (0x1a48c3534000) libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x1a48c3560000) libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x1a48c3c0a000) libc++.so.1 => /usr/lib/libc++.so.1 (0x1a48c4d02000) libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x1a48c490e000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x1a48c5c07000) libthr.so.3 => /lib/libthr.so.3 (0x1a48c6272000) libc.so.7 => /lib/libc.so.7 (0x1a48c67c4000) libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x1a48c704b000) libintl.so.8 => /usr/local/lib/libintl.so.8 (0x1a48c8394000) libsys.so.7 => /lib/libsys.so.7 (0x1a48c73d6000) [vdso] (0x1a48bdeb7000)Edit-2:
Errors occurs on systems that were updated from pfSense CE 2.7.2 (Xeon-D 2xxx Boxes not from Netgate). We have an 8200 box with exactly the same package revisions and versions where a quick default config of squid resulted in the service starting just fine. So perhaps there's either a dependency from an older version and/or CE update path mingling in(?) or the package installed somehow differs from the one installed by the 8200 box we tested with?
-
UPDATE
We did some further testing on an updated system from CE on a whitelabel hardware box as well as the above mentioned 8200 test system. And the following was made to check for ELF / syncer problems:
On the official 8200 hardware freshly installed we had the following ldd output:
[24.11-RELEASE][admin@test-sense-hof-1.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid /usr/local/sbin/squid: librt.so.1 => /usr/lib/librt.so.1 (0xd97be0a1000) libcrypt.so.5 => /lib/libcrypt.so.5 (0xd97bd35c000) libregex.so.1 => /usr/lib/libregex.so.1 (0xd97bff60000) libcrypto.so.30 => /lib/libcrypto.so.30 (0xd97be522000) libssl.so.30 => /usr/lib/libssl.so.30 (0xd97bebef000) libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0xd97bf382000) libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0xd97c012a000) libm.so.5 => /lib/libm.so.5 (0xd97c1ace000) libdl.so.1 => /usr/lib/libdl.so.1 (0xd97c1073000) libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0xd97c2524000) libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0xd97c3502000) >>> libc++.so.1 => /lib/libc++.so.1 (0xd97c5049000) libcxxrt.so.1 => /lib/libcxxrt.so.1 (0xd97c358c000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xd97c3809000) libthr.so.3 => /lib/libthr.so.3 (0xd97c3c24000) libc.so.7 => /lib/libc.so.7 (0xd97c665b000) libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0xd97c3ed4000) libintl.so.8 => /usr/local/lib/libintl.so.8 (0xd97c49a7000) libsys.so.7 => /lib/libsys.so.7 (0xd97c5764000) [vdso] (0xd97bcbb5000)and we compared with an updated box that throws the ELF error:
[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid /usr/local/sbin/squid: librt.so.1 => /usr/lib/librt.so.1 (0x3a62613d6000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x3a62607fd000) libregex.so.1 => /usr/lib/libregex.so.1 (0x3a6261b24000) libcrypto.so.30 => /lib/libcrypto.so.30 (0x3a6262f73000) libssl.so.30 => /usr/lib/libssl.so.30 (0x3a6262410000) libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x3a6263872000) libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x3a6263d19000) libm.so.5 => /lib/libm.so.5 (0x3a626429a000) libdl.so.1 => /usr/lib/libdl.so.1 (0x3a6264c5a000) libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x3a6265ec5000) libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x3a6264d34000) >>> libc++.so.1 => /usr/lib/libc++.so.1 (0x3a626500f000) libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x3a626643f000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x3a626734a000) libthr.so.3 => /lib/libthr.so.3 (0x3a6267c17000) libc.so.7 => /lib/libc.so.7 (0x3a6268a0c000) libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x3a626a7c7000) libintl.so.8 => /usr/local/lib/libintl.so.8 (0x3a62693d9000) libsys.so.7 => /lib/libsys.so.7 (0x3a6269600000) [vdso] (0x3a625fbb5000)What immediatly jumped into my face was the different selection of libc++ linked in, as the fresh system linked it from /lib/ and the updated system came with an /usr/lib/ link. Checking the new system revealed no installed libc++.so.1 in /usr/lib whereas the upgraded system shows as following:
[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /lib/libc++.so.1 -r--r--r-- 1 root wheel 993752 Nov 22 06:44 /lib/libc++.so.1 [24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /usr/lib/libc++.so.1 -r--r--r-- 1 root wheel 819952 Jan 31 2022 /usr/lib/libc++.so.1So it seems to me, that there was another C++ library from - I think 23.09 or even earlier - that was installed into /usr/lib instead of the new /lib handling and after upgrading to 24.03 and 24.11 wasn't cleared out and had precedence over the correct library in /lib and as it probably was built for FreeBSD 14 (my guess) the ELF problem happens as the library won't match the rest of the userland and system.
The current workaround, that worked on the Test system was to just move the /usr/lib library to /root (so we have it at hand if it was necessary after all) and try to start squid again. And it worked to start.
So workaround:
- login via SSH/console
- as root do
mv /usr/lib/libc++.so.1 /root- afterwards check if you can start squid again.
Cheers
-
Aha, nice catch!
Hmm, that's.... unexpected.
Those systems had Squid installed in CE before upgrading to Plus?
-
-
@stephenw10 Hi,
yes, the systems had Squid installed all along, so coming from 2.7.2 onwards.
-
-
Nice. It would be good to confirm everyone hitting this is using an install upgraded from CE.
-
@dauhee said in after upgrade to 24.11: squid doesn´t start:
Could we do a cp instead just in case?
No that won't fix the problem of the linker pulling the wrong library. If you just copy it, the original is still in /usr/lib and gets pulled via ldd.
That's why I said "move" (mv) it instead of just deleting. Moving IS the backup way instead of plain and simply deleting the old library that seems obsolete. So you're not breaking anything permanently by moving, you can always move it back afterwards if that happens to trigger any other problems, but we have seen none so far on a system with quite a bunch of different packages installed.Cheers
-
@stephenw10 said in after upgrade to 24.11: squid doesn´t start:
Nice. It would be good to confirm everyone hitting this is using an install upgraded from CE.
with out customer that was the case, I'll hit our lab VMs and spin them up to check if some of them has a libc++ in /usr/lib instead of /lib only, but testing will be messy there as our old snapshots aren't recognized as working "plus" anymore (because the VM HW changed after cloning them). But I'll see what we can do.
-
@19pegr69 said in after upgrade to 24.11: squid doesn´t start:
Hallo,
after upgrading to 24.11 squid can not be started
I always get this:
/pkg_edit.php: The command '/usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was 'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'Would be curious, if your system was also updated from CE or just an older Plus version. Hope our workaround works for you, too.
-
This post is deleted! -
Thanks with the updates to fix the major security concerns upstream, I assume that this package will not be removed as the concerns for removing it have been addressed. Please see attached..
This was the email from the Squid development team...
"The Squid Project apologizes for being late in responding to the publication of 55 vulnerabilities disclosed by Joshua Rogers of Opera Software at https://megamansec.github.io/Squid-Security-Audit/ We thank Joshua for discovering these bugs and sharing their details with us. The surprise publication caught us off guard, but Squid developers had worked on addressing some of the disclosed vulnerabilities since before that publication. This message summarizes Squid's status on October 9th, 2024. As of Squid v6.8, the vast majority of high-impact vulnerabilities have been addressed. The following disclosed vulnerabilities are still present: ### Vulnerability “strlen(NULL) Crash Using Digest Authentication” This vulnerability is still present in Squid v6.11. A fix is expected in Squid v6.12, due any day now. Digest authentication is disabled by default; the current workaround is to avoid Digest authentication. To verify whether your Squid configuration is vulnerable, check whether it contains "auth_param” directive. Configurations with auth_param directives mentioning "digest" scheme may be vulnerable. ### pipeline_prefetch (HTTP pipelining of client-to-Squid requests) All reported pipelining-related vulnerabilities may still be present in Squid v6. Pipelining code will probably be removed in master branch and become unavailable in Squid v7. Pipelining is disabled by default. If you do not need pipelining (or do not know for sure that you need it), do not enable that performance optimization. To verify whether your Squid configuration is vulnerable, check whether it contains a pipeline_prefetch directive. Configurations containing a pipeline_prefetch directive set to a positive value may be vulnerable. ### ESI (Edge Side Includes) Most reported ESI-related vulnerabilities are still present in Squid v6. ESI code has been removed in the master branch and will not be available in Squid v7. ESI is disabled in the default build starting with Squid v6.10. In earlier versions, ESI code is enabled by default, but the risk is moderate because exploiting this family of vulnerabilities requires Squid to be configured as a reverse proxy for a malicious origin server. If you do not need ESI (or do not know whether you need it), disable it with `--disable-esi` (default for Squid v6.10 and later). To verify whether your Squid build is vulnerable, run `squid -v`. Squid v6.9 and earlier versions may be vulnerable unless the output contains `--disable-esi`. Squid v6.10 and later versions may be vulnerable if the output contains `--enable-esi`. ### Squid v5 Some fixes were backported to Squid v5, but we lack the resources necessary to support that old version. Folks running Squid v5 and earlier versions should either upgrade to the latest v6 release or rely on their integrator/distributor for support. -- Francesco Chemolli Squid Software Foundation"So it's now fixed upstream and the new version in the package reflects the updated version. Again there is still some software convergence going on with the updates.
Thanks for all you do.
-
Indeed the plan was to remove it from the package repo; it's still there since there haven't been significant challenges to keeping it and it gives more time for those using it to plan ahead. At this time there is no maintainer for the pfSense package and there are no plans to maintain it.
-
@JeGr said in after upgrade to 24.11: squid doesn´t start:
mv /usr/lib/libc++.so.1 /root
I tried and got this error when I try and restart squid. am on 24.11-RELEASE (amd64)
