Squid3 PBI and 2.1
-
I noticed the Jim committed some changes related to getting PBIs for Squid3. So I updated to the latest build on my test system:
2.1-BETA0 (i386)
built on Thu Jun 14 23:20:09 EDT 2012
FreeBSD 8.3-RELEASE-p3Hoping that I could try Squid3 on 2.1
I also noticed that packages that don't have valid PBIs available have been made to not display in the available packages list for 2.1 - which is a good thing until they do actually have PBIs and can install properly.
Squid3 does not appear in the available packages list now.
Was there a problem with getting the PBIs for Squid3?
Also, I know that Squidguard currently has a dependency on Squid (i.e. Squid 2), so installing SquidGuard will want Squid (2) to install. Will there need to be a "SquidGuard3" that depends on Squid3, so that users can choose to install Squid+SquidGuard or Squid3+SquidGuard3.
I am very happy to test this out once it is available for installation.
(Answers to this post could also be added to the Squid3 post in Packages once there is something up and running to advertise for use) -
Here is some post on the mailing list according .PBIs
http://lists.pfsense.org/pipermail/dev/2012-May/000176.htmlsquidguard:
install first squidguard and then install squid3. This will use squid3 + squidguard.
But of course you need .pbis if you want to install from GUI or you do this withpkg_add -r blabla.tbz
from console.
-
Phil,
The problem was, at one time, we told the squid3 maintainer that he couldn't use build_port_path because it conflicted with squid2, with PBIs that's not an issue.
The package maintainer just needs to add the proper build_port_path tag and build_options tag, and also those files for squid3 are not hosted on our servers, they must be moved to pfsense.org servers.
See my post yesterday about package cleanup:
http://forum.pfsense.org/index.php/topic,50473.0.html -
Now we have a PBI for squid3 - great stuff.
Upgraded to:
2.1-BETA0 (i386)
built on Mon Jun 18 17:27:59 EDT 2012
FreeBSD 8.3-RELEASE-p3Installed squid3.
First the minor thing I noticed - the Packages list says that squid3 is 3.1.19 but now the latest PBI is 3.1.20 - it downloads 3.1.20 PBI, but of course the version listed on the "Installed Packages" list is 3.1.19 - I think this is a common issue with packages that have various versions for pfSense platforms/releases.Next the more major problem, squid does not start - system log of installation:
Jun 19 09:56:06 php: /pkg_mgr_install.php: Beginning package installation for squid3 . Jun 19 04:22:23 check_reload_status: Syncing firewall Jun 19 10:07:29 php: /pkg_mgr_install.php: Stopping any running proxy monitors Jun 19 10:07:30 php: /pkg_mgr_install.php: Starting Squid Jun 19 10:07:30 php: /pkg_mgr_install.php: Starting a proxy monitor script Jun 19 10:07:30 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 04:22:30 check_reload_status: Reloading filter Jun 19 10:07:35 Squid_Alarm[12687]: Squid has exited. Reconfiguring filter. Jun 19 10:07:35 Squid_Alarm[13177]: Attempting restart... Jun 19 10:07:35 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 10:07:38 Squid_Alarm[15026]: Reconfiguring filter... Jun 19 04:22:41 check_reload_status: Syncing firewall Jun 19 10:07:41 php: /pkg_mgr_install.php: Creating squid log dir /var/squid/logs/ Jun 19 04:22:41 check_reload_status: Reloading filter Jun 19 10:07:41 php: /pkg_mgr_install.php: Starting Squid Jun 19 10:07:41 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 10:07:45 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:07:46 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:07:47 php: : SQUID is installed but not started. Not installing "filter" rules. Jun 19 10:07:49 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:07:50 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:07:50 php: : SQUID is installed but not started. Not installing "filter" rules. Jun 19 04:22:51 check_reload_status: Reloading filter Jun 19 04:22:52 check_reload_status: Syncing firewall Jun 19 10:07:59 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:08:00 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:08:00 php: : SQUID is installed but not started. Not installing "filter" rules. Jun 19 10:13:48 php: /pkg_edit.php: [squid] xmlrpc sync is starting. Jun 19 10:13:48 php: /pkg_edit.php: Starting Squid Jun 19 10:13:49 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 04:28:59 check_reload_status: Reloading filter Jun 19 04:29:00 check_reload_status: Syncing firewall Jun 19 10:14:06 php: /pkg_edit.php: [squid] xmlrpc sync is starting. Jun 19 10:14:06 php: /pkg_edit.php: Starting Squid Jun 19 10:14:06 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 10:14:10 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:14:10 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:14:11 php: : SQUID is installed but not started. Not installing "filter" rules. Jun 19 04:29:16 check_reload_status: Reloading filter Jun 19 10:14:22 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:14:22 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:14:23 php: : SQUID is installed but not started. Not installing "filter" rules.
Then after trying the restart button in the Status Services display:
Jun 19 10:17:57 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 10:18:02 php: /status_services.php: The command '/usr/local/etc/rc.d/squid.sh stop' returned exit code '1', the output was 'FATAL: getpwnam failed to find userid for effective user 'squid' Squid Cache (Version 3.1.20): Terminated abnormally. CPU Usage: 0.041 seconds = 0.025 user + 0.017 sys Maximum Resident Size: 3996 KB Page faults with physical i/o: 0' Jun 19 10:18:04 squid: getpwnam failed to find userid for effective user 'squid'
Then trying a reboot:
Jun 19 10:25:37 Squid_Alarm[13536]: Squid has exited. Reconfiguring filter. Jun 19 10:25:37 Squid_Alarm[14070]: Attempting restart... Jun 19 10:25:37 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 10:25:39 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:25:40 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:25:40 Squid_Alarm[18390]: Reconfiguring filter... Jun 19 10:25:41 check_reload_status: Reloading filter Jun 19 10:25:42 php: : SQUID is installed but not started. Not installing "filter" rules. Jun 19 10:25:45 php: : Starting Squid Jun 19 10:25:45 squid: getpwnam failed to find userid for effective user 'squid' Jun 19 10:25:48 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:25:49 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:25:49 php: : SQUID is installed but not started. Not installing "filter" rules. Jun 19 10:25:55 check_reload_status: Reloading filter Jun 19 10:26:03 php: : SQUID is installed but not started. Not installing "nat" rules. Jun 19 10:26:04 php: : SQUID is installed but not started. Not installing "pfearly" rules. Jun 19 10:26:05 php: : SQUID is installed but not started. Not installing "filter" rules.
So that's the initial report. This feels like dejavu to me - I'm sure that I have seen this squid effective userid issue before, and fixed it. So I'll now set my brain working…
-
It was not using the conf file written by the pfSense squid package code at /usr/local/etc/squid/squid.conf
This conf file correctly has "cache_effective_user proxy"
Edited /usr/local/pkg/squid.inc
To the end of every command that runs, reconfigures, stops, starts squid ("/usr/local/sbin/squid …") added " -f /usr/local/etc/squid/squid.conf"
Now it happily finds the correct squid.conf and no longer complains about cache_effective_user squid.Should it be necessary to add the "-f" repeatedly in many places?
I guess the conf file path should be in a global var and then use that var everywhere. -
Now it complains about the following options in /usr/local/etc/squid/squid.conf :
unrecognised 'sslcrtd_children'
unrecognised 'delay_pools'
unrecognised 'delay_class'
unrecognised 'delay_parameters'
unrecognised 'delay_initial_bucket_level'
unrecognised 'delay_access'I seem to remember that this was a problem with previous builds of squid - these options need to be built into squid so it knows about them and implements them.
To get it running for now, I edited /usr/local/pkg/squid.inc and put a comment "# " in front of all the places that these options were written to squid.conf
Now squid starts.
-
Now squid is running. But it complains in the log file that transparent proxying is not supported.
2012/06/19 11:39:17| Starting Squid Cache version 3.1.20 for i386-portbld-freebsd8.1... 2012/06/19 11:39:17| Process ID 48017 2012/06/19 11:39:17| With 3405 file descriptors available 2012/06/19 11:39:17| Initializing IP Cache... 2012/06/19 11:39:17| DNS Socket created at [::], FD 12 2012/06/19 11:39:17| DNS Socket created at 0.0.0.0, FD 14 2012/06/19 11:39:17| Adding domain localdomain from /etc/resolv.conf 2012/06/19 11:39:17| Adding nameserver 127.0.0.1 from /etc/resolv.conf 2012/06/19 11:39:17| Adding nameserver 10.49.80.250 from /etc/resolv.conf 2012/06/19 11:39:18| Unlinkd pipe opened on FD 19 2012/06/19 11:39:18| Store logging disabled 2012/06/19 11:39:18| Swap maxSize 0 + 8192 KB, estimated 630 objects 2012/06/19 11:39:18| Target number of buckets: 31 2012/06/19 11:39:18| Using 8192 Store buckets 2012/06/19 11:39:18| Max Mem size: 8192 KB 2012/06/19 11:39:18| Max Swap size: 0 KB 2012/06/19 11:39:18| Using Least Load store dir selection 2012/06/19 11:39:18| Current Directory is /usr/local/www 2012/06/19 11:39:18| Loaded Icons. 2012/06/19 11:39:18| Accepting HTTP connections at 192.168.1.1:3128, FD 22. 2012/06/19 11:39:18| Accepting intercepted HTTP connections at 127.0.0.1:3128, FD 23. 2012/06/19 11:39:18| Accepting ICP messages at [::]:7, FD 24. 2012/06/19 11:39:18| HTCP Disabled. 2012/06/19 11:39:18| Ready to serve requests. 2012/06/19 11:39:19| storeLateRelease: released 0 objects 2012/06/19 11:39:26| Reconfiguring Squid Cache (version 3.1.20)... 2012/06/19 11:39:26| FD 22 Closing HTTP connection 2012/06/19 11:39:26| FD 23 Closing HTTP connection 2012/06/19 11:39:26| FD 24 Closing ICP connection 2012/06/19 11:39:26| Processing Configuration File: /usr/local/etc/squid/squid.conf (depth 0) 2012/06/19 11:39:26| Starting Authentication on port 127.0.0.1:3128 2012/06/19 11:39:26| Disabling Authentication on port 127.0.0.1:3128 (interception enabled) 2012/06/19 11:39:26| Disabling IPv6 on port 127.0.0.1:3128 (interception enabled) 2012/06/19 11:39:26| Store logging disabled 2012/06/19 11:39:26| DNS Socket created at [::], FD 14 2012/06/19 11:39:26| DNS Socket created at 0.0.0.0, FD 15 2012/06/19 11:39:26| Adding domain localdomain from /etc/resolv.conf 2012/06/19 11:39:26| Adding nameserver 127.0.0.1 from /etc/resolv.conf 2012/06/19 11:39:26| Adding nameserver 10.49.80.250 from /etc/resolv.conf 2012/06/19 11:39:26| Accepting HTTP connections at 192.168.1.1:3128, FD 17. 2012/06/19 11:39:26| Accepting intercepted HTTP connections at 127.0.0.1:3128, FD 18. 2012/06/19 11:39:26| Accepting ICP messages at [::]:7, FD 22. 2012/06/19 11:39:26| HTCP Disabled. 2012/06/19 11:39:26| Loaded Icons. 2012/06/19 11:39:26| Ready to serve requests. 2012/06/19 11:40:59| WARNING: transparent proxying not supported 2012/06/19 11:41:56| WARNING: transparent proxying not supported 2012/06/19 11:41:58| WARNING: transparent proxying not supported 2012/06/19 11:41:58| WARNING: transparent proxying not supported 2012/06/19 11:41:59| WARNING: transparent proxying not supported 2012/06/19 11:42:00| WARNING: transparent proxying not supported 2012/06/19 11:42:00| WARNING: transparent proxying not supported 2012/06/19 11:42:00| WARNING: transparent proxying not supported 2012/06/19 11:45:41| WARNING: transparent proxying not supported
Maybe this is another build option that needs to be including when building squid and putting together the PBI?
-
Summary up to now:
- squid.inc - needs changes so that it uses the customised squid.conf at /usr/local/etc/squid/squid.conf
- squid needs to be built with support for the various conf file options (sslcrtd_children, delay_*)
- squid needs to be built with support for transparent proxy (some googling indicates that "–enable-linux-netfilter" is a good thing to add)
With the above 3 items, I expect that at least basic squid3 installations should work.
-
After trying to remember what was done for summary item (1), I think that the previous squid got built with the correct conf file location for pfSense built into the image. So there was no need to add the "-f" parameter to every reference to starting/reconfiguring/stopping squid in squid.inc (I couldn't see where any "-f" had been added in the squid(2) version of squid.inc).
Maybe all 3 items above can be corrected in the build of the PBI used by squid3 - conf file location, support for extra options and support for transparent proxy.
-
most of the was already reported: http://forum.pfsense.org/index.php/topic,48347.msg269453.html#msg269453
I noticed a new issue with installing… Did a complete remove this morning, reboot, then install. Install is stuck at Creating squid cache pools... One moment please... and php process has been at 100% for 5 minutes... Checked the file system, no files being built... Thinking this may because of the -f option that is needed now or squid will use its default settings.
-
The user should be fixed by (a) a new snapshot and (b) a new PBI, as the PBI tools were fixed just yesterday to supporting properly adding needed users.
The config file was never correct in /usr/local/etc - it should be /var/etc, so long as it's fixed, and it must be fixed in squid.inc and such. A global won't work, it'll need to be a constant, for whatever reason globals don't work properly in package .inc files at bootup, there are a few other threads about it.
The options there, delay pools and such, should be fixed also by a new PBI. Not sure why they were not proper in the current one but they were set, might just need a new build.
As for the version, yes that does mismatch but since the .tbz version is still older it wasn't bumped.
-
Thanks Jim! Question, is the new snapshot for binaries? I installed last Sat but gitsync often.
For the conf path, I've noticed a lot of packages that use /usr/local/etc… Is the plan to have all of them moved to /var/etc?
-
All of our configs should have always been in /var/etc, but historically packages haven't really cared quite so much. Many were left in /usr/local/etc simply because it was the default.
As long as changes are being made to manually specify the config path, may as well put them where they're supposed to go.
-
I deleted all my packages first (to avoid any possibility that old binaries were left around) then upgraded to:
2.1-BETA0 (i386)
built on Tue Jun 19 14:25:19 EDT 2012
FreeBSD 8.3-RELEASE-p3Then installed squid3. This latest version of the PBI was on http://files.pfsense.org/packages/8/All/ :
squid-3.1.20-i386.pbi 2012-Jun-19 15:41:12 15.8M application/octet-streamSo it should have loaded this PBI that Jim put there yesterday.
Edited squid.inc to make all the start/stop/reconfigure commands point to the correct squid.conf (I'll submit a pull request for this in Github soon).
[2.1-BETA0][root@test20120614.localdomain]/usr/local/etc/rc.d(28): /usr/local/sbin/squid -D -f /usr/local/etc/squid/squid.conf 2012/06/20 10:05:45| WARNING: -D command-line option is obsolete. 2012/06/20 10:05:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:17 unrecognized: 'sslcrtd_children' 2012/06/20 10:05:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:61 unrecognized: 'delay_pools' 2012/06/20 10:05:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:62 unrecognized: 'delay_class' 2012/06/20 10:05:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:63 unrecognized: 'delay_parameters' 2012/06/20 10:05:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:64 unrecognized: 'delay_initial_bucket_level' 2012/06/20 10:05:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:65 unrecognized: 'delay_access'
I noticed that the "squid -D" command-line option is now obsolete - this is mentioned in a few posts on the WWW such as at
http://squid-web-proxy-cache.1019090.n4.nabble.com/questions-with-squid-3-1-td1557011.html2. # sbin/squid -D
2010/02/16 15:02:41| WARNING: -D command-line option is obsolete.-D is obsolete, why and what's the corresponding one to this option in
squid-3.1?-D existed only to solve one problem which is now fully fixed.
But I have trouble finding this change mentioned anywhere squid 3.1 doco!
I'll remove "-D" in my squid.inc pull request.Edited squid.inc temporarily to comment out all the unrecognized options above. Then squid will start.
/var/squid/logs/cache.log still reports:
2012/06/20 10:12:07| Ready to serve requests. 2012/06/20 10:17:34| WARNING: transparent proxying not supported
Issues that I still have:
- The various squid config options above are unrecognized.
- It gives the warning about transparent proxying not supported.
I think both these issues need to be fixed inside the PBI file?
-
That is odd as I am specifying everything in the build that needs to be there for the options to work, and yet they seem to not be getting pulled in.
Others have said that squid 2.x and squidguard are working, and they both specify options the same way, so I'm not really sure why it would be failing like that. I'll have to run some tests and see for myself what it's doing.
-
OK so I discovered that the pbi.conf variable names changed somewhere between when our scripts were written and the current code for building PBIs that we had to pull in to fix the user issue, so I made a few changes to the build script but that still didn't seem to help yet, I just tried it on a vm (feel free to try it yourself though)
To make sure the new binary gets pulled in, you should probably uninstall/reinstall to make sure it gets the new binary. I think it only removes the binary if there is a version difference in the binary itself, not just if the pfSense package version gets bumped, but I'd have to double check that.
I've got another idea cooking now, will know in a while if it's good.
-
Thank for all your work on getting the PBIs to work… At first I wasn't a fan of them but now I see they are good thing for pfsense...
-
OK - looks good now - have at it!
-
you da man!!!
Squid Cache: Version 3.1.20 configure options: '--with-default-user=squid' '--bindir=/usr/pbi/squid-i386/sbin' '--sbindir=/usr/pbi/squid-i386/sbin' '--datadir=/usr/pbi/squid-i386/etc/squid' '--libexecdir=/usr/pbi/squid-i386/libexec/squid' '--localstatedir=/var/squid' '--sysconfdir=/usr/pbi/squid-i386/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--disable-translation' '--enable-auth=basic digest negotiate ntlm' '--enable-basic-auth-helpers=DB NCSA PAM MSNT SMB squid_radius_auth LDAP SASL YP' '--enable-digest-auth-helpers=password ldap' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group ldap_group' '--enable-ntlm-auth-helpers=smb_lm' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-storeio=ufs diskd aufs' '--enable-disk-io=AIO Blocking DiskDaemon DiskThreads' '--enable-delay-pools' '--enable-ssl' '--with-openssl=/usr' '--enable-ssl-crtd' '--enable-icmp' '--enable-forw-via-db' '--enable-cache-digests' '--disable-wccp' '--enable-wccpv2' '--enable-referer-log' '--enable-useragent-log' '--enable-arp-acl' '--enable-ipfw-transparent' '--enable-pf-transparent' '--enable-follow-x-forwarded-for' '--disable-ecap' '--disable-loadable-modules' '--disable-kqueue' '--with-large-files' '--prefix=/usr/pbi/squid-i386' '--mandir=/usr/pbi/squid-i386/man' '--infodir=/usr/pbi/squid-i386/info/' '--build=i386-portbld-freebsd8.1' 'build_alias=i386-portbld-freebsd8.1' 'CC=cc' 'CFLAGS=-O2 -pipe -I/usr/pbi/squid-i386/include -I/usr/pbi/squid-i386/include -I/usr/include -DLDAP_DEPRECATED -fno-strict-aliasing' 'LDFLAGS= -L/usr/pbi/squid-i386/lib -L/usr/pbi/squid-i386/lib -rpath=/usr/lib:/usr/pbi/squid-i386/lib -L/usr/lib' 'CPPFLAGS=-I/usr/pbi/squid-i386/include' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -I/usr/pbi/squid-i386/include -I/usr/pbi/squid-i386/include -I/usr/include -DLDAP_DEPRECATED -fno-strict-aliasing' 'CPP=cpp' --with-squid=/usr/wrkdirprefix/usr/ports/www/squid31/work/squid-3.1.20 --enable-ltdl-convenience
-
Great :-)
I tested it myself before posting this time so I was sure it was good.
That bug affected all PBI builds, so now I get to go back and rebuild every PBI, which will take more than a day to finish.