after upgrade to 24.11: squid doesn´t start

JeGr

@marcosm @stephenw10 said in after upgrade to 24.11: squid doesn´t start:

Also in x86?

Hi! Have to chime in: I was called into a "prod down incident" of one of our bigger customers. Same problem as above: Customer updated their clusters and on all 2nd machines where they failed over to test, Squid didn't came up. Same ELF linker problem as described above, exactly same output:

@intellq said in after upgrade to 24.11: squid doesn´t start:

'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'

Same failure description and path: System is a x86-64 platform, was running 23.09_1 / 24.03 with squid before and had no problems. We also tried steps above like checking the package version and in addition to that I tried removing, purging, clearing the pkg cache and re-downloading all packages related to squid to check if that ld-elf error was related to a broken/defective package somewhere, but got the same error after reinstalling/redownloading the packages anyway.

Is there any way we can get this escalated or checked?
Any way we could help or contribute this problem?
what's the official status now about the Squid package in general? As the last official statement is still the blog from last year citing the package will be removed in 24.03 but here we are in 24.11 and still have the package (which IS a GOOD thing, we have many customers that WANT/NEED it). But after today they want a statement about the future of the package and if it really stays in the repo or if the breakage is the first sign of "it's gonna be removed"? Would be nice to have something official as to what's the way forward.

Temporary solution was to roll back all systems to 24.03 via snapshots (thank god for them :)) but that's not really a final solution

Cheers :)

Edit: added information

Linking error and linker dependencies:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/root: /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"

[24.11-RELEASE][admin@pfSense-test.nh00.local]/root: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0x1a48bf1b9000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x1a48be888000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0x1a48bf64e000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x1a48c0686000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0x1a48c2702000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x1a48c150e000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x1a48bf822000)
        libm.so.5 => /lib/libm.so.5 (0x1a48c241e000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x1a48c3534000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x1a48c3560000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x1a48c3c0a000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x1a48c4d02000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x1a48c490e000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x1a48c5c07000)
        libthr.so.3 => /lib/libthr.so.3 (0x1a48c6272000)
        libc.so.7 => /lib/libc.so.7 (0x1a48c67c4000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x1a48c704b000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x1a48c8394000)
        libsys.so.7 => /lib/libsys.so.7 (0x1a48c73d6000)
        [vdso] (0x1a48bdeb7000)

Edit-2:

Errors occurs on systems that were updated from pfSense CE 2.7.2 (Xeon-D 2xxx Boxes not from Netgate). We have an 8200 box with exactly the same package revisions and versions where a quick default config of squid resulted in the service starting just fine. So perhaps there's either a dependency from an older version and/or CE update path mingling in(?) or the package installed somehow differs from the one installed by the 8200 box we tested with?

JeGr

@marcosm @stephenw10

UPDATE

We did some further testing on an updated system from CE on a whitelabel hardware box as well as the above mentioned 8200 test system. And the following was made to check for ELF / syncer problems:

On the official 8200 hardware freshly installed we had the following ldd output:

[24.11-RELEASE][admin@test-sense-hof-1.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0xd97be0a1000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0xd97bd35c000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0xd97bff60000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0xd97be522000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0xd97bebef000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0xd97bf382000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0xd97c012a000)
        libm.so.5 => /lib/libm.so.5 (0xd97c1ace000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0xd97c1073000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0xd97c2524000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0xd97c3502000)
 >>>    libc++.so.1 => /lib/libc++.so.1 (0xd97c5049000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0xd97c358c000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xd97c3809000)
        libthr.so.3 => /lib/libthr.so.3 (0xd97c3c24000)
        libc.so.7 => /lib/libc.so.7 (0xd97c665b000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0xd97c3ed4000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0xd97c49a7000)
        libsys.so.7 => /lib/libsys.so.7 (0xd97c5764000)
        [vdso] (0xd97bcbb5000)

and we compared with an updated box that throws the ELF error:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0x3a62613d6000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x3a62607fd000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0x3a6261b24000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x3a6262f73000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0x3a6262410000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x3a6263872000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x3a6263d19000)
        libm.so.5 => /lib/libm.so.5 (0x3a626429a000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x3a6264c5a000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x3a6265ec5000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x3a6264d34000)
 >>>    libc++.so.1 => /usr/lib/libc++.so.1 (0x3a626500f000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x3a626643f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x3a626734a000)
        libthr.so.3 => /lib/libthr.so.3 (0x3a6267c17000)
        libc.so.7 => /lib/libc.so.7 (0x3a6268a0c000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x3a626a7c7000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x3a62693d9000)
        libsys.so.7 => /lib/libsys.so.7 (0x3a6269600000)
        [vdso] (0x3a625fbb5000)

What immediatly jumped into my face was the different selection of libc++ linked in, as the fresh system linked it from /lib/ and the updated system came with an /usr/lib/ link. Checking the new system revealed no installed libc++.so.1 in /usr/lib whereas the upgraded system shows as following:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /lib/libc++.so.1
-r--r--r--  1 root wheel 993752 Nov 22 06:44 /lib/libc++.so.1

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /usr/lib/libc++.so.1
-r--r--r--  1 root wheel 819952 Jan 31  2022 /usr/lib/libc++.so.1

So it seems to me, that there was another C++ library from - I think 23.09 or even earlier - that was installed into /usr/lib instead of the new /lib handling and after upgrading to 24.03 and 24.11 wasn't cleared out and had precedence over the correct library in /lib and as it probably was built for FreeBSD 14 (my guess) the ELF problem happens as the library won't match the rest of the userland and system.

The current workaround, that worked on the Test system was to just move the /usr/lib library to /root (so we have it at hand if it was necessary after all) and try to start squid again. And it worked to start.

So workaround:

login via SSH/console
as root do

mv /usr/lib/libc++.so.1 /root

afterwards check if you can start squid again.

Cheers

stephenw10

Aha, nice catch!

Hmm, that's.... unexpected.

Those systems had Squid installed in CE before upgrading to Plus?

dauhee

@JeGr

Could we do a cp instead just in case?

Thanks

JeGr

@stephenw10 Hi,

yes, the systems had Squid installed all along, so coming from 2.7.2 onwards.

intellq

@JeGr you are a life saver!

It's working again now :)

ps: my system also was updated from a 2.7

stephenw10

Nice. It would be good to confirm everyone hitting this is using an install upgraded from CE.

JeGr

@dauhee said in after upgrade to 24.11: squid doesn´t start:

Could we do a cp instead just in case?

No that won't fix the problem of the linker pulling the wrong library. If you just copy it, the original is still in /usr/lib and gets pulled via ldd.
That's why I said "move" (mv) it instead of just deleting. Moving IS the backup way instead of plain and simply deleting the old library that seems obsolete. So you're not breaking anything permanently by moving, you can always move it back afterwards if that happens to trigger any other problems, but we have seen none so far on a system with quite a bunch of different packages installed.

Cheers

JeGr

@stephenw10 said in after upgrade to 24.11: squid doesn´t start:

Nice. It would be good to confirm everyone hitting this is using an install upgraded from CE.

with out customer that was the case, I'll hit our lab VMs and spin them up to check if some of them has a libc++ in /usr/lib instead of /lib only, but testing will be messy there as our old snapshots aren't recognized as working "plus" anymore (because the VM HW changed after cloning them). But I'll see what we can do.

JeGr

@19pegr69 said in after upgrade to 24.11: squid doesn´t start:

Hallo,
after upgrading to 24.11 squid can not be started
I always get this:
/pkg_edit.php: The command '/usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was 'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'

Would be curious, if your system was also updated from CE or just an older Plus version. Hope our workaround works for you, too.

wiliam025

This post is deleted!

JonathanLee

Thanks with the updates to fix the major security concerns upstream, I assume that this package will not be removed as the concerns for removing it have been addressed. Please see attached..

This was the email from the Squid development team...

"The Squid Project apologizes for being late in responding to the
publication of 55 vulnerabilities disclosed by Joshua Rogers of Opera Software
at https://megamansec.github.io/Squid-Security-Audit/

We thank Joshua for discovering these bugs and sharing their details with us.
The surprise publication caught us off guard, but Squid
developers had worked on addressing some of the disclosed vulnerabilities
since before that publication. This message summarizes Squid's status on
October 9th, 2024.

As of Squid v6.8, the vast majority of high-impact vulnerabilities have been
addressed. The following disclosed vulnerabilities are still present:


### Vulnerability “strlen(NULL) Crash Using Digest Authentication”

This vulnerability is still present in Squid v6.11. A fix is expected in Squid
v6.12, due any day now.
Digest authentication is disabled by default; the current workaround is
to avoid Digest authentication.

To verify whether your Squid configuration is vulnerable, check whether it
contains "auth_param” directive. Configurations with auth_param directives
mentioning "digest" scheme may be vulnerable.


### pipeline_prefetch (HTTP pipelining of client-to-Squid requests)

All reported pipelining-related vulnerabilities may still be present in Squid
v6. Pipelining code will probably be removed in master branch and become
unavailable in Squid v7. Pipelining is disabled by default.

If you do not need pipelining (or do not know for sure that you need it), do
not enable that performance optimization.

To verify whether your Squid configuration is vulnerable, check whether it
contains a pipeline_prefetch directive. Configurations containing a
pipeline_prefetch directive set to a positive value may be vulnerable.


### ESI (Edge Side Includes)

Most reported ESI-related vulnerabilities are still present in Squid v6. ESI
code has been removed in the master branch and will not be available
in Squid v7.
ESI is disabled in the default build starting with Squid v6.10. In earlier
versions, ESI code is enabled by default, but the risk is moderate because
exploiting this family of vulnerabilities requires Squid to be
configured as a reverse proxy for a malicious origin server.

If you do not need ESI (or do not know whether you need it), disable it with
`--disable-esi` (default for Squid v6.10 and later).

To verify whether your Squid build is vulnerable, run `squid -v`. Squid v6.9
and earlier versions may be vulnerable unless the output contains
`--disable-esi`. Squid v6.10 and later versions may be vulnerable if the
output contains `--enable-esi`.


### Squid v5

Some fixes were backported to Squid v5, but we lack the resources necessary to
support that old version. Folks running Squid v5 and earlier versions should
either upgrade to the latest v6 release or rely on their
integrator/distributor for support.

--
   Francesco Chemolli
   Squid Software Foundation"

So it's now fixed upstream and the new version in the package reflects the updated version. Again there is still some software convergence going on with the updates.

Thanks for all you do.

marcosm

Indeed the plan was to remove it from the package repo; it's still there since there haven't been significant challenges to keeping it and it gives more time for those using it to plan ahead. At this time there is no maintainer for the pfSense package and there are no plans to maintain it.

dauhee

@JeGr said in after upgrade to 24.11: squid doesn´t start:

mv /usr/lib/libc++.so.1 /root

I tried and got this error when I try and restart squid. am on 24.11-RELEASE (amd64)

JeGr

@dauhee That looks like missing files or stuff from de-/reinstall not finishing completely. If unsure, you can easily remove the package and install again or reinstall. But that it starts AT ALL is due to the bad library being moved so it now can start again. Everything after is likely a config or install thing but seeing it started up again for you is very nice!

Should be possible to restore or roll back to your config now. Also keep in mind the other patches above that mentioned problems with the config file format (noTLSv1 etc.) that perhaps need to be patched, too.

But that should at least fix the point with it not starting due to the ld-elf.so error message.

Cheers :)

JonathanLee

@dauhee you need to recreate the linker file to the error log folder it is listed incorrectly again.

https://redmine.pfsense.org/issues/14406
I am thinking that 14406 is the same issue for the error log folder mix up again. It jumps from en to en-us sometimes

ln -s /usr/local/etc/squid/errors/templates /usr/local/etc/squid/errors/en-us

ln -s /usr/local/etc/squid/errors/templates /usr/local/etc/squid/errors/en