after upgrade to 24.11: squid doesn´t start

stephenw10

Which error is that?

JonathanLee

@stephenw10 2024/12/04 20:00:01| WARNING: Failed to decode EC parameters '/etc/dh-parameters.2048'
OpenSSL-saved error #1: 0x1e08010c

stephenw10

Ah I see. Hmm....

19pegr69

It's nice that everyone seems to be able to solve this problem, but other than the comment that I've probably installed the wrong package, I haven't been able to get any help. I just expect the script to get the right package itself when updating. Can someone please explain to a FreeBSD layman how I can get the apparently correct package onto the system when installing via the GUI?

stephenw10

Check the pkgs you actually have installed: pkg info -x squid

dauhee

@stephenw10 said in after upgrade to 24.11: squid doesn´t start:

pkg info -x squid

this is mine and what I suspect everyone elses with version applied via the package manager:

pfSense-pkg-squid-0.5
squid-6.10
squid_radius_auth-1.10
squidclamav-7.3_2

is there a way to pull from the git commit that was mentioned or is it a waiting game for the next release?

stephenw10

You can apply both using the System Patches pkg, just create patches from the commits.

However that won't help with the undefined symbol errors some users are seeing.

intellq

Thread got sort of hijacked with another problem... but I'm here to tell that the exact same thing also happened to me.

/pkg_edit.php: The command '/usr/local/sbin/squid -z -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was 'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'

After the first boot after upgrading to 24.11, Squid doesn't work anymore, no matter what I do. I've tried:

Changed a lot of settings;
Reset squid settings to default;
Uninstalled squid completely...
...reinstall via gui
...or command line:

pkg install -f pfSense-pkg-squid-0.5 squid-6.10 squid_radius_auth-1.10 squidclamav-7.3_2

Nothing. Nada :(

Any help?

stephenw10

Also in x86?

intellq

@stephenw10 yes, amd64

JonathanLee

I have to be completely honest with you, I have not updated for a while because of issues with this package, it is my favorite package to tinker with.

JeGr

@marcosm @stephenw10 said in after upgrade to 24.11: squid doesn´t start:

Also in x86?

Hi! Have to chime in: I was called into a "prod down incident" of one of our bigger customers. Same problem as above: Customer updated their clusters and on all 2nd machines where they failed over to test, Squid didn't came up. Same ELF linker problem as described above, exactly same output:

@intellq said in after upgrade to 24.11: squid doesn´t start:

'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'

Same failure description and path: System is a x86-64 platform, was running 23.09_1 / 24.03 with squid before and had no problems. We also tried steps above like checking the package version and in addition to that I tried removing, purging, clearing the pkg cache and re-downloading all packages related to squid to check if that ld-elf error was related to a broken/defective package somewhere, but got the same error after reinstalling/redownloading the packages anyway.

Is there any way we can get this escalated or checked?
Any way we could help or contribute this problem?
what's the official status now about the Squid package in general? As the last official statement is still the blog from last year citing the package will be removed in 24.03 but here we are in 24.11 and still have the package (which IS a GOOD thing, we have many customers that WANT/NEED it). But after today they want a statement about the future of the package and if it really stays in the repo or if the breakage is the first sign of "it's gonna be removed"? Would be nice to have something official as to what's the way forward.

Temporary solution was to roll back all systems to 24.03 via snapshots (thank god for them :)) but that's not really a final solution

Cheers :)

Edit: added information

Linking error and linker dependencies:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/root: /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"

[24.11-RELEASE][admin@pfSense-test.nh00.local]/root: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0x1a48bf1b9000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x1a48be888000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0x1a48bf64e000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x1a48c0686000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0x1a48c2702000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x1a48c150e000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x1a48bf822000)
        libm.so.5 => /lib/libm.so.5 (0x1a48c241e000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x1a48c3534000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x1a48c3560000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x1a48c3c0a000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x1a48c4d02000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x1a48c490e000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x1a48c5c07000)
        libthr.so.3 => /lib/libthr.so.3 (0x1a48c6272000)
        libc.so.7 => /lib/libc.so.7 (0x1a48c67c4000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x1a48c704b000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x1a48c8394000)
        libsys.so.7 => /lib/libsys.so.7 (0x1a48c73d6000)
        [vdso] (0x1a48bdeb7000)

Edit-2:

Errors occurs on systems that were updated from pfSense CE 2.7.2 (Xeon-D 2xxx Boxes not from Netgate). We have an 8200 box with exactly the same package revisions and versions where a quick default config of squid resulted in the service starting just fine. So perhaps there's either a dependency from an older version and/or CE update path mingling in(?) or the package installed somehow differs from the one installed by the 8200 box we tested with?

JeGr

@marcosm @stephenw10

UPDATE

We did some further testing on an updated system from CE on a whitelabel hardware box as well as the above mentioned 8200 test system. And the following was made to check for ELF / syncer problems:

On the official 8200 hardware freshly installed we had the following ldd output:

[24.11-RELEASE][admin@test-sense-hof-1.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0xd97be0a1000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0xd97bd35c000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0xd97bff60000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0xd97be522000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0xd97bebef000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0xd97bf382000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0xd97c012a000)
        libm.so.5 => /lib/libm.so.5 (0xd97c1ace000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0xd97c1073000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0xd97c2524000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0xd97c3502000)
 >>>    libc++.so.1 => /lib/libc++.so.1 (0xd97c5049000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0xd97c358c000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xd97c3809000)
        libthr.so.3 => /lib/libthr.so.3 (0xd97c3c24000)
        libc.so.7 => /lib/libc.so.7 (0xd97c665b000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0xd97c3ed4000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0xd97c49a7000)
        libsys.so.7 => /lib/libsys.so.7 (0xd97c5764000)
        [vdso] (0xd97bcbb5000)

and we compared with an updated box that throws the ELF error:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0x3a62613d6000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x3a62607fd000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0x3a6261b24000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x3a6262f73000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0x3a6262410000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x3a6263872000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x3a6263d19000)
        libm.so.5 => /lib/libm.so.5 (0x3a626429a000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x3a6264c5a000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x3a6265ec5000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x3a6264d34000)
 >>>    libc++.so.1 => /usr/lib/libc++.so.1 (0x3a626500f000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x3a626643f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x3a626734a000)
        libthr.so.3 => /lib/libthr.so.3 (0x3a6267c17000)
        libc.so.7 => /lib/libc.so.7 (0x3a6268a0c000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x3a626a7c7000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x3a62693d9000)
        libsys.so.7 => /lib/libsys.so.7 (0x3a6269600000)
        [vdso] (0x3a625fbb5000)

What immediatly jumped into my face was the different selection of libc++ linked in, as the fresh system linked it from /lib/ and the updated system came with an /usr/lib/ link. Checking the new system revealed no installed libc++.so.1 in /usr/lib whereas the upgraded system shows as following:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /lib/libc++.so.1
-r--r--r--  1 root wheel 993752 Nov 22 06:44 /lib/libc++.so.1

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /usr/lib/libc++.so.1
-r--r--r--  1 root wheel 819952 Jan 31  2022 /usr/lib/libc++.so.1

So it seems to me, that there was another C++ library from - I think 23.09 or even earlier - that was installed into /usr/lib instead of the new /lib handling and after upgrading to 24.03 and 24.11 wasn't cleared out and had precedence over the correct library in /lib and as it probably was built for FreeBSD 14 (my guess) the ELF problem happens as the library won't match the rest of the userland and system.

The current workaround, that worked on the Test system was to just move the /usr/lib library to /root (so we have it at hand if it was necessary after all) and try to start squid again. And it worked to start.

So workaround:

login via SSH/console
as root do

mv /usr/lib/libc++.so.1 /root

afterwards check if you can start squid again.

Cheers

stephenw10

Aha, nice catch!

Hmm, that's.... unexpected.

Those systems had Squid installed in CE before upgrading to Plus?

dauhee

@JeGr

Could we do a cp instead just in case?

Thanks

JeGr

@stephenw10 Hi,

yes, the systems had Squid installed all along, so coming from 2.7.2 onwards.

intellq

@JeGr you are a life saver!

It's working again now :)

ps: my system also was updated from a 2.7

stephenw10

Nice. It would be good to confirm everyone hitting this is using an install upgraded from CE.

JeGr

@dauhee said in after upgrade to 24.11: squid doesn´t start:

Could we do a cp instead just in case?

No that won't fix the problem of the linker pulling the wrong library. If you just copy it, the original is still in /usr/lib and gets pulled via ldd.
That's why I said "move" (mv) it instead of just deleting. Moving IS the backup way instead of plain and simply deleting the old library that seems obsolete. So you're not breaking anything permanently by moving, you can always move it back afterwards if that happens to trigger any other problems, but we have seen none so far on a system with quite a bunch of different packages installed.

Cheers

JeGr

@stephenw10 said in after upgrade to 24.11: squid doesn´t start:

Nice. It would be good to confirm everyone hitting this is using an install upgraded from CE.

with out customer that was the case, I'll hit our lab VMs and spin them up to check if some of them has a libc++ in /usr/lib instead of /lib only, but testing will be messy there as our old snapshots aren't recognized as working "plus" anymore (because the VM HW changed after cloning them). But I'll see what we can do.