Squid 2.7.9 install hangs at Perl?

GX1

Hello.

I'm setting up a new pfSense box, and when I try to install the stable Squid package, it seems to hang while extracting Perl… The box is still working (I am routed through it right now), and the rest of the web interface remains responsive, it is just the package installation that hangs.

Here is the text displayed in the package manager:


Installing squid and its dependencies.

Beginning package installation for squid...
Downloading package configuration file... done.
Saving updated package information... done.
Downloading squid and its dependencies... 
Checking for package installation... 
 Downloading http://files.pfsense.org/packages/8/All/squid-2.7.9_3.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/cyrus-sasl-2.1.26_2.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/openldap-client-2.4.33_1.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/perl-5.14.2_2.tbz ...  (extracting)

… and the system log:

Last 50 system log entries
Apr 26 10:13:13	syslogd: kernel boot file is /boot/kernel/kernel
Apr 26 10:13:51	check_reload_status: Syncing firewall
Apr 26 10:13:52	php: /pkg_mgr_install.php: Beginning package installation for squid.
Apr 26 10:15:12	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libgssapi.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 10:15:12	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libgssapi.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 10:15:12	sshlockout[57468]: sshlockout/webConfigurator v3.0 starting up
Apr 26 10:15:13	sasldblistusers2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libgssapi.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 10:15:13	sasldblistusers2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libgssapi.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 10:15:13	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libgssapi.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 10:15:13	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libgssapi.so.10" not found, required by "libgssapiv2.so.3"

For reference, the current time as of these captures is 11:40… so the install has been running for 1h 25m. It only took a few minutes to get to the point where it was extracting Perl, and at that point there was a lot of activity from 'bsdtar'. However, all instances of 'bsdtar' have now finished...

The system is running from an 8GB compact flash card, installed using the 2GB pfSense-2.0.3-RELEASE-1g-i386-nanobsd_vga-20130412-1022.img.gz image. The Status Dashboard shows the CPU usage at 0%, Memory usage at 16%, and Disk usage at 52%.

Any thoughts?
(Thanks in advance)

GX1

A bit more information:

Even though it doesn't actually complete the install (nothing new in the "services" menu), I found that I can still successfully remove the package.

Backing up libraries... 
Removing package...
Starting package deletion for squid-2.7.9_3...done.
Starting package deletion for cyrus-sasl-2.1...done.
Starting package deletion for openldap-client-2.4...done.
Starting package deletion for perl-5.14...done.
Starting package deletion for squid_radius_auth-1.10...done.
Starting package deletion for libwww-5.4.0_4...done.
Removing squid components...
Tabs items... done.
Menu items... done.
Services... done.
Loading package instructions...
Include file squid.inc could not be found for inclusion.
Deinstall commands... 
Not executing custom deinstall hook because an include is missing.
Removing package instructions...done.
Auxiliary files... done.
Package XML... done.
Configuration... done.
Cleaning up... 
Package deleted.

marcelloc

@GX1:

Apr 26 10:15:13 saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libgssapi.so.10" not found, required by "libgssapiv2.so.3"

GX1, if you think this missing libs a breaking install, you can copy these libs from any 8.1 freebsd.

I have it on my personal repo, if you what to download from there, the link is
i386
http://e-sac.siteseguro.ws/pfsense/8/All/ldd/

amd64
http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/

save it on /usr/lib or /usr/local/lib

How long did you waited for perl extract?

GX1

Thanks for the reply, marcelloc. :)

I waited quite a while for perl to extract… I didn't see any way to abort or otherwise "stop" an install that was in-progress, but I left it open all the way up until right before my last post. So it ran for what, almost 5 1/2 hours? At the time when I did the uninstall, there had been no change in the installation output, and no change to the system log (which I reloaded to refresh). :-\

I downloaded libgssapi.so.10 from your website (thanks!) and placed it into /usr/lib. That was more complicated than I would have thought, as my other machines can't read the freebsd file system on the CF card, and pfsense apparently doesn't have wget installed as a baseline tool. So I had to SCP it in from my Linux box. (Is there a better way?)

After that, I retried the Squid install, and now have this in my system log:

Apr 26 22:00:31	syslogd: kernel boot file is /boot/kernel/kernel
Apr 26 22:01:13	check_reload_status: Syncing firewall
Apr 26 22:01:13	php: /pkg_mgr_install.php: Beginning package installation for squid.
Apr 26 22:02:29	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libheimntlm.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 22:02:29	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libheimntlm.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 22:02:29	sshlockout[38858]: sshlockout/webConfigurator v3.0 starting up
Apr 26 22:02:29	sasldblistusers2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libheimntlm.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 22:02:29	sasldblistusers2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libheimntlm.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 22:02:30	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Shared object "libheimntlm.so.10" not found, required by "libgssapiv2.so.3"
Apr 26 22:02:30	saslpasswd2: unable to dlopen /usr/local/lib/sasl2/libgssapiv2.so.3: Share

It's a different file now, so I might have to go through a few rounds to get all of those shared objects copied over. In the mean time, 'bsdtar' is extracting perl again, so I suppose I will wait until that disappears again, do another uninstall, and then update the shared objects and try again….

I'm not sure if they are related to my Squid issue, but is there something wrong with my pfsense install that it doesn't already have these shared objects?

Thanks again for the help! 8)

marcelloc

Use fetch instead of wget and download all files from that link to get no lib depending issues.

GX1

@marcelloc:

Use fetch instead of wget and download all files from that link to get no lib depending issues.

Sounds good, thanks, I'll do that. :)

Two things I forgot to mention in my last post:

I had previously noticed the presence/absence of the 'bsdtar' command in top, but I see that 'pkg_add' is running too. 'pkg_add' is probably the one I should be watching.

Also, if anyone else has to do something similar in the future, I found that the root file system on the CF card was mounted read-only. So I had to run:

/etc/rc.conf_mount_rw
```before copying the file(s), and then run:

/etc/rc.conf_mount_ro

to set it back when I was done.

GX1

Ok, nope, that didn't fix it. :-\

I copied all of the shared object files from marcelloc's site, and did another installation attempt.

Here is the installation log:

Installing squid and its dependencies.
Beginning package installation for squid...
Downloading package configuration file... done.
Saving updated package information... done.
Downloading squid and its dependencies... 
Checking for package installation... 
 Downloading http://files.pfsense.org/packages/8/All/squid-2.7.9_3.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/cyrus-sasl-2.1.26_2.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/openldap-client-2.4.33_1.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/perl-5.14.2_2.tbz ...  (extracting)

… and here is the sytem log:

Apr 26 23:04:32	syslogd: kernel boot file is /boot/kernel/kernel
Apr 26 23:06:07	check_reload_status: Syncing firewall
Apr 26 23:06:08	php: /pkg_mgr_install.php: Beginning package installation for squid.
Apr 26 23:31:29	apinger: ALARM: WAN(192.168.1.1) *** down ***
Apr 26 23:31:29	apinger: alarm canceled: WAN(192.168.1.1) *** down ***
Apr 26 23:31:39	check_reload_status: Reloading filter

All of the previous errors in the system log are gone this time, but the install still seems to have hung. I've been watching it, and it was working on the install/extraction for about 50 minutes, and then 'pkg_add' dropped out of the process list. :-\

Here is what the system indicates is running now:

last pid: 20348;  load averages:  0.24,  0.30,  0.35                                          up 0+02:29:15  00:03:14
46 processes:  5 running, 41 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle
Mem: 35M Active, 118M Inact, 51M Wired, 1012K Cache, 59M Buf, 277M Free
Swap:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
19591 root        1  45    0 46748K 21492K accept   2:16  0.00% php
  261 root        1  64   20  3348K  1156K kqread   1:55  0.00% check_reload_status
 2002 root        1  44    0  3652K  1996K RUN      0:23  0.00% top
48332 root        1  76   20  3596K  1412K piperd   0:21  0.00% sh
 2144 root        1  44    0  7944K  3592K select   0:17  0.00% sshd
47484 root        1  64   20  3256K  1320K select   0:16  0.00% apinger
21832 root        1  45    0 46748K 18356K accept   0:11  0.00% php
29956 nobody      1  44    0  5504K  2644K select   0:08  0.00% dnsmasq
18977 root        1  44    0  6780K  3496K RUN      0:07  0.00% lighttpd
15285 root        1  64   20  6020K  6044K select   0:05  0.00% ntpd
29390 dhcpd       1  44    0  8376K  5180K select   0:04  0.00% dhcpd
25460 root        1  44    0  4636K  2580K pause    0:01  0.00% tcsh
19017 root        1  76    0 44700K 12316K wait     0:01  0.00% php
19436 root        1  70    0 44700K 12316K wait     0:01  0.00% php
10816 root        1  44    0  5860K  2140K bpf      0:01  0.00% tcpdump
59218 root        1  44    0  4908K  2572K select   0:00  0.00% syslogd
53353 root        1  71    0  3628K  1548K wait     0:00  0.00% login
54600 root        1  76    0  3628K  1552K wait     0:00  0.00% login
40203 root        1  44    0  3348K  1360K nanslp   0:00  0.00% cron
  162 root        1  64   20  4536K  1868K piperd   0:00  0.00% rrdtool
10949 root        1  44    0  3256K   876K piperd   0:00  0.00% logger
53657 root        2  44    0  3472K  1164K nanslp   0:00  0.00% sshlockout_pf
38858 root        2  44    0  3472K  1132K nanslp   0:00  0.00% sshlockout_pf
 7315 root        1  49    0  3596K  1392K wait     0:00  0.00% sh
11682 root        1  44    0  3376K  1292K select   0:00  0.00% inetd
42730 root        1  58    0  3256K  1032K nanslp   0:00  0.00% minicron
53764 root        1  76    0  3596K  1392K wait     0:00  0.00% sh
55495 root        1  76    0  3596K  1392K wait     0:00  0.00% sh
  274 root        1  44    0  1888K   532K select   0:00  0.00% devd
 8032 root        1  44    0  3596K  1396K wait     0:00  0.00% sh
54931 root        1  69    0  3596K  1392K ttyin    0:00  0.00% sh
59304 root        1  76    0  3596K  1392K ttyin    0:00  0.00% sh
22377 root        1  67   20  3256K  1312K select   0:00  0.00% dhclient
 6461 root        1  44    0  5224K  2968K select   0:00  0.00% sshd
43184 root        1  76    0  3256K   988K wait     0:00  0.00% minicron
19972 root        1 116   20  3596K  1412K RUN      0:00  0.00% sh
20348 root        1 116   20   528K   392K RUN      0:00  0.00% awk
43134 root        1  44    0  3256K  1032K nanslp   0:00  0.00% minicron
20316 root        1 116   20   412K   288K RUN      0:00  0.00% cat
42780 root        1  76    0  3256K   988K wait     0:00  0.00% minicron
42189 root        1  76    0  3256K   988K wait     0:00  0.00% minicron
 6648 _dhcp       1  64   20  3256K  1412K select   0:00  0.00% dhclient
  263 root        1  76   20  3348K  1096K kqread   0:00  0.00% check_reload_status
43227 root        1  76    0  3256K  1032K nanslp   0:00  0.00% minicron

Any other thoughts on what to look for? Are there any other log files, or other ways to find more detail on the package installation as it progresses?

Thanks.

GX1

@GX1:

I'm not sure if they are related to my Squid issue, but is there something wrong with my pfsense install that it doesn't already have these shared objects?

In reply to myself, I wonder if it is this issue?

http://redmine.pfsense.org/issues/875
pfSense Bug #875: "Uninstalling packages can remove system libraries"

Description

If you install a package (like squid) which replaces a base system library (like LDAP), it will install its own copy on install and then remove it on uninstall, leaving the system without said library.

Depending on the package it can be a major issue, especially if a package installs and subsequently removes XML libraries.

We may need to register the base system packages and/or somehow prevent a package from removing them. (Perhaps add dummy "pfSense" package entry that would list all of the packages we need as a dependency?)

GX1

I tried installing Squid3, and it had the same behavior… all was going fine until about an hour in, and then 'pkg_add' disappeared, with no updates to the log.

Install log:

Installing squid3 and its dependencies.
Beginning package installation for squid3...
Downloading package configuration file... done.
Saving updated package information... done.
Downloading squid3 and its dependencies... 
Checking for package installation... 
 Downloading http://files.pfsense.org/packages/8/All/squid-3.1.20.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/cyrus-sasl-2.1.23_3.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/openldap-client-2.4.31_1.tbz ...  (extracting)

   Downloading http://files.pfsense.org/packages/8/All/perl-threaded-5.12.4_4.tbz ...  (extracting)

marcelloc

I don't have a nanobsd install to test it. On full install it's working fine, both packages.

try to install this perl version from oficial repo before trying a package reinstall.

Check if you are not going out of disk space on /var.

GX1

@marcelloc:

try to install this perl version from oficila repo before trying a package reinstall.

How do I do that?

I'll check the disk usage on /var…

GX1

So I tried again, using pfSense 2.0.2 this time, and got the same result with Squid 2.7.9.

The usage on /var never changed… It stayed at 14MB used / 40MB available the entire time of the install attempt.

GX1

Ok, so it turns out that /var was out of space!

I tried running:

/etc/rc.conf_mount_rw
pkg_add http://files.pfsense.org/packages/8/All/perl-5.14.2_2.tbz

… and it gives me:

Fetching http://files.pfsense.org/packages/8/All/perl-5.14.2_2.tbz...
/var: write failed, filesystem is full
lib/perl5/5.14.2/unicore/lib/NChar/N.pl: Write failed
lib/perl5/5.14.2/unicore/lib/NChar/Y.pl: Seek failed
lib/perl5/5.14.2/unicore/lib/NFCQC/Y.pl: Seek failed
lib/perl5/5.14.2/unicore/lib/NFDQC/N.pl: Seek failed
lib/perl5/5.14.2/unicore/lib/NFDQC/Y.pl: Seek failed
lib/perl5/5.14.2/unicore/lib/NFKCQC/M.pl: Seek failed
lib/perl5/5.14.2/unicore/lib/NFKCQC/N.pl: Seek failed
lib/perl5/5.14.2/unicore/lib/NFKCQC/Y.pl: Seek failed
... 
lib/perl5/5.14.2/warnings.pm: Seek failed
lib/perl5/5.14.2/warnings/register.pm: Seek failed
tar: (Empty error message)
tar: Error exit delayed from previous errors.
 Done.
pkg_add: package 'perl-5.14.2_2' or its older version already installed

var doesn't "seem" to be full, but pkg_add must be needing to write something larger than the available capacity… ???

df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ufs/pfsense0    443M    232M    175M    57%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/md0              38M    274K     35M     1%    /tmp
/dev/md1              58M     14M     40M    26%    /var
/dev/ufs/cf           49M    166K     45M     0%    /cf
devfs                1.0K    1.0K      0B   100%    /var/dhcpd/dev

I then found this other thread, where someone was having the same problem (var being full) when installing Python:
http://forum.pfsense.org/index.php/topic,44953.msg234429.html

So I tried their solution:

[2.0.3-RELEASE][root@bonfire.localdomain]/var/log(17): setenv PKG_TMPDIR /root/
[2.0.3-RELEASE][root@bonfire.localdomain]/var/log(18): pkg_add http://files.pfsense.org/packages/8/All/perl-5.14.2_2.tbz
Fetching http://files.pfsense.org/packages/8/All/perl-5.14.2_2.tbz... Done.
pkg_add: package 'perl-5.14.2_2' or its older version already installed

Perl is already installed, eh? ???
Let's try Squid…

[2.0.3-RELEASE][root@bonfire.localdomain]/var/log(25): pkg_add http://files.pfsense.org/packages/8/All/squid-2.7.9_3.tbz
Fetching http://files.pfsense.org/packages/8/All/squid-2.7.9_3.tbz... Done.
pkg_add: package 'squid-2.7.9_3' or its older version already installed

Hmm! Both already installed!

So then I went back to the web interface, and selected "reinstall GUI components" (or something to that effect) for Squid…

Removing squid components...
Tabs items... done.
Menu items... done.
Services... done.
Loading package instructions...
Include file squid.inc could not be found for inclusion.
Deinstall commands... 
Not executing custom deinstall hook because an include is missing.
Removing package instructions...done.
Auxiliary files... done.
Package XML... done.
Configuration... done.
Beginning package installation for squid...
Downloading package configuration file... done.
Saving updated package information... done.
Downloading squid and its dependencies... 
Checking for package installation... 
 Downloading http://files.pfsense.org/packages/8/All/libwww-5.4.0_4.tbz ...  (extracting)
Loading package configuration... done.
Configuring package components...
Additional files... done.
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Executing custom_php_resync_config_command()...done.
Custom commands...
Executing custom_php_install_command()...done.
Executing custom_php_resync_config_command()...done.
Menu items... done.
Integrated Tab items... done.
Services... done.
Writing configuration... done.

Package reinstalled.

… and the next time I refreshed the web interface, it now shows "Proxy Server" under services!!! Hooray! :D

The system log seems to confirm this too:

Apr 27 22:45:24	php: /pkg_mgr_install.php: Beginning package installation for squid.
Apr 27 22:47:43	check_reload_status: Syncing firewall
Apr 27 22:47:43	php: /pkg_mgr_install.php: Stopping any running proxy monitors
Apr 27 22:47:45	php: /pkg_mgr_install.php: Starting Squid
Apr 27 22:47:45	php: /pkg_mgr_install.php: Starting a proxy monitor script
Apr 27 22:47:45	squid[54449]: Bungled (null) line 182: http_reply_access allow all
Apr 27 22:47:45	check_reload_status: Reloading filter
Apr 27 22:47:47	php: /pkg_mgr_install.php: Starting Squid
Apr 27 22:47:47	squid[57532]: Squid Parent: child process 57856 started
Apr 27 22:47:48	php: /pkg_mgr_install.php: Reloading Squid for configuration sync

I'm still not sure where this leaves things in terms of the problem. I even tried 2.0.3 in a VirtualBox VM and got the same problem - I just couldn't install Squid on the nanobsd version of pfSense. It seems to be related to the size of /var… is this something that is "fixable", or should the potential problem (or the resulting failure) be detectable by the package installer? Should I file a bug report for it?

Thanks again to everyone for the help! 8)