after upgrade to 24.11: squid doesn´t start

dauhee

I have this also. I've nothing to add to the resolution efforts unfortunately

stephenw10

@JonathanLee said in after upgrade to 24.11: squid doesn´t start:

https://github.com/pfsense/FreeBSD-ports/commit/476a7d0e3dca704b236839970f1d215912184f73

This doesn't seem enough, at least not for all archs, since it also fails on TLSv1_1:

[24.11-RELEASE][admin@2100-3.stevew.lan]/root: /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
CPU Usage: 0.064 seconds = 0.040 user + 0.024 sys
Maximum Resident Size: 54384 KB
Page faults with physical i/o: 0
2024/12/04 20:00:01| Processing Configuration File: /usr/local/etc/squid/squid.conf (depth 0)
2024/12/04 20:00:01| WARNING: Failed to decode EC parameters '/etc/dh-parameters.2048'
    OpenSSL-saved error #1: 0x1e08010c
2024/12/04 20:00:01| FATAL: Unknown http_port option 'NO_TLSv1_1'.
2024/12/04 20:00:01| Not currently OK to rewrite swap log.
2024/12/04 20:00:01| storeDirWriteCleanLogs: Operation aborted.
2024/12/04 20:00:01| FATAL: Bungled /usr/local/etc/squid/squid.conf line 4: http_port 192.168.221.1:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem tls-cafile=/usr/local/share/certs/ca-root-nss.crt capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3, NO_TLSv1_1
2024/12/04 20:00:01| Squid Cache (Version 6.10): Terminated abnormally.

JonathanLee

@stephenw10 I think he fixed this also it lists as a red removed line today for TLSv1_1 also. I wonder why the new Squid 7.0 that resolved all the concerns no longer lets you use this directive. Weird maybe it is just all disabled now for it and they do not need that directive. The V 7 has resolved almost all the concerns that was in that white paper

marcosm

The issue should be fixed after applying both the previous commit and this one:
https://github.com/pfsense/FreeBSD-ports/commit/009dc5f68e0cf1d1a767d1a9119bcbaface44823

JonathanLee

@marcosm thanks I love this package, I am also having that OpenSSL error for some reason.

stephenw10

Which error is that?

JonathanLee

@stephenw10 2024/12/04 20:00:01| WARNING: Failed to decode EC parameters '/etc/dh-parameters.2048'
OpenSSL-saved error #1: 0x1e08010c

stephenw10

Ah I see. Hmm....

19pegr69

It's nice that everyone seems to be able to solve this problem, but other than the comment that I've probably installed the wrong package, I haven't been able to get any help. I just expect the script to get the right package itself when updating. Can someone please explain to a FreeBSD layman how I can get the apparently correct package onto the system when installing via the GUI?

stephenw10

Check the pkgs you actually have installed: pkg info -x squid

dauhee

@stephenw10 said in after upgrade to 24.11: squid doesn´t start:

pkg info -x squid

this is mine and what I suspect everyone elses with version applied via the package manager:

pfSense-pkg-squid-0.5
squid-6.10
squid_radius_auth-1.10
squidclamav-7.3_2

is there a way to pull from the git commit that was mentioned or is it a waiting game for the next release?

stephenw10

You can apply both using the System Patches pkg, just create patches from the commits.

However that won't help with the undefined symbol errors some users are seeing.

intellq

Thread got sort of hijacked with another problem... but I'm here to tell that the exact same thing also happened to me.

/pkg_edit.php: The command '/usr/local/sbin/squid -z -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was 'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'

After the first boot after upgrading to 24.11, Squid doesn't work anymore, no matter what I do. I've tried:

Changed a lot of settings;
Reset squid settings to default;
Uninstalled squid completely...
...reinstall via gui
...or command line:

pkg install -f pfSense-pkg-squid-0.5 squid-6.10 squid_radius_auth-1.10 squidclamav-7.3_2

Nothing. Nada :(

Any help?

stephenw10

Also in x86?

intellq

@stephenw10 yes, amd64

JonathanLee

I have to be completely honest with you, I have not updated for a while because of issues with this package, it is my favorite package to tinker with.

JeGr

@marcosm @stephenw10 said in after upgrade to 24.11: squid doesn´t start:

Also in x86?

Hi! Have to chime in: I was called into a "prod down incident" of one of our bigger customers. Same problem as above: Customer updated their clusters and on all 2nd machines where they failed over to test, Squid didn't came up. Same ELF linker problem as described above, exactly same output:

@intellq said in after upgrade to 24.11: squid doesn´t start:

'ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"'

Same failure description and path: System is a x86-64 platform, was running 23.09_1 / 24.03 with squid before and had no problems. We also tried steps above like checking the package version and in addition to that I tried removing, purging, clearing the pkg cache and re-downloading all packages related to squid to check if that ld-elf error was related to a broken/defective package somewhere, but got the same error after reinstalling/redownloading the packages anyway.

Is there any way we can get this escalated or checked?
Any way we could help or contribute this problem?
what's the official status now about the Squid package in general? As the last official statement is still the blog from last year citing the package will be removed in 24.03 but here we are in 24.11 and still have the package (which IS a GOOD thing, we have many customers that WANT/NEED it). But after today they want a statement about the future of the package and if it really stays in the repo or if the breakage is the first sign of "it's gonna be removed"? Would be nice to have something official as to what's the way forward.

Temporary solution was to roll back all systems to 24.03 via snapshots (thank god for them :)) but that's not really a final solution

Cheers :)

Edit: added information

Linking error and linker dependencies:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/root: /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
ld-elf.so.1: /usr/local/sbin/squid: Undefined symbol "_ZTTNSt3__118basic_stringstreamIcNS_11char_traitsIcEENS_9allocatorIcEEEE"

[24.11-RELEASE][admin@pfSense-test.nh00.local]/root: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0x1a48bf1b9000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x1a48be888000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0x1a48bf64e000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x1a48c0686000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0x1a48c2702000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x1a48c150e000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x1a48bf822000)
        libm.so.5 => /lib/libm.so.5 (0x1a48c241e000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x1a48c3534000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x1a48c3560000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x1a48c3c0a000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x1a48c4d02000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x1a48c490e000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x1a48c5c07000)
        libthr.so.3 => /lib/libthr.so.3 (0x1a48c6272000)
        libc.so.7 => /lib/libc.so.7 (0x1a48c67c4000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x1a48c704b000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x1a48c8394000)
        libsys.so.7 => /lib/libsys.so.7 (0x1a48c73d6000)
        [vdso] (0x1a48bdeb7000)

Edit-2:

Errors occurs on systems that were updated from pfSense CE 2.7.2 (Xeon-D 2xxx Boxes not from Netgate). We have an 8200 box with exactly the same package revisions and versions where a quick default config of squid resulted in the service starting just fine. So perhaps there's either a dependency from an older version and/or CE update path mingling in(?) or the package installed somehow differs from the one installed by the 8200 box we tested with?

JeGr

@marcosm @stephenw10

UPDATE

We did some further testing on an updated system from CE on a whitelabel hardware box as well as the above mentioned 8200 test system. And the following was made to check for ELF / syncer problems:

On the official 8200 hardware freshly installed we had the following ldd output:

[24.11-RELEASE][admin@test-sense-hof-1.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0xd97be0a1000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0xd97bd35c000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0xd97bff60000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0xd97be522000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0xd97bebef000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0xd97bf382000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0xd97c012a000)
        libm.so.5 => /lib/libm.so.5 (0xd97c1ace000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0xd97c1073000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0xd97c2524000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0xd97c3502000)
 >>>    libc++.so.1 => /lib/libc++.so.1 (0xd97c5049000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0xd97c358c000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xd97c3809000)
        libthr.so.3 => /lib/libthr.so.3 (0xd97c3c24000)
        libc.so.7 => /lib/libc.so.7 (0xd97c665b000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0xd97c3ed4000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0xd97c49a7000)
        libsys.so.7 => /lib/libsys.so.7 (0xd97c5764000)
        [vdso] (0xd97bcbb5000)

and we compared with an updated box that throws the ELF error:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ldd /usr/local/sbin/squid
/usr/local/sbin/squid:
        librt.so.1 => /usr/lib/librt.so.1 (0x3a62613d6000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x3a62607fd000)
        libregex.so.1 => /usr/lib/libregex.so.1 (0x3a6261b24000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x3a6262f73000)
        libssl.so.30 => /usr/lib/libssl.so.30 (0x3a6262410000)
        libk5crypto.so.3.1 => /usr/local/lib/libk5crypto.so.3.1 (0x3a6263872000)
        libcom_err.so.3.0 => /usr/local/lib/libcom_err.so.3.0 (0x3a6263d19000)
        libm.so.5 => /lib/libm.so.5 (0x3a626429a000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x3a6264c5a000)
        libkrb5.so.3.3 => /usr/local/lib/libkrb5.so.3.3 (0x3a6265ec5000)
        libgssapi_krb5.so.2.2 => /usr/local/lib/libgssapi_krb5.so.2.2 (0x3a6264d34000)
 >>>    libc++.so.1 => /usr/lib/libc++.so.1 (0x3a626500f000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x3a626643f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x3a626734a000)
        libthr.so.3 => /lib/libthr.so.3 (0x3a6267c17000)
        libc.so.7 => /lib/libc.so.7 (0x3a6268a0c000)
        libkrb5support.so.0.1 => /usr/local/lib/libkrb5support.so.0.1 (0x3a626a7c7000)
        libintl.so.8 => /usr/local/lib/libintl.so.8 (0x3a62693d9000)
        libsys.so.7 => /lib/libsys.so.7 (0x3a6269600000)
        [vdso] (0x3a625fbb5000)

What immediatly jumped into my face was the different selection of libc++ linked in, as the fresh system linked it from /lib/ and the updated system came with an /usr/lib/ link. Checking the new system revealed no installed libc++.so.1 in /usr/lib whereas the upgraded system shows as following:

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /lib/libc++.so.1
-r--r--r--  1 root wheel 993752 Nov 22 06:44 /lib/libc++.so.1

[24.11-RELEASE][admin@pfSense-test.nh00.local]/var/cache/pkg: ls -la /usr/lib/libc++.so.1
-r--r--r--  1 root wheel 819952 Jan 31  2022 /usr/lib/libc++.so.1

So it seems to me, that there was another C++ library from - I think 23.09 or even earlier - that was installed into /usr/lib instead of the new /lib handling and after upgrading to 24.03 and 24.11 wasn't cleared out and had precedence over the correct library in /lib and as it probably was built for FreeBSD 14 (my guess) the ELF problem happens as the library won't match the rest of the userland and system.

The current workaround, that worked on the Test system was to just move the /usr/lib library to /root (so we have it at hand if it was necessary after all) and try to start squid again. And it worked to start.

So workaround:

login via SSH/console
as root do

mv /usr/lib/libc++.so.1 /root

afterwards check if you can start squid again.

Cheers

stephenw10

Aha, nice catch!

Hmm, that's.... unexpected.

Those systems had Squid installed in CE before upgrading to Plus?

dauhee

@JeGr

Could we do a cp instead just in case?

Thanks