Squid3 - New GUI with sync, normal and reverse proxy

Cino · Jun 18, 2012, 6:13 PM

http://lists.pfsense.org/pipermail/dev/2012-June/000178.html

thanks. Guess I should had read the whole thing… I missed the bottom part

EDIT: Squid 3 has been built it looks, http://files.pfsense.com/packages/8/All/squid-3.1.19-i386.pbi

EDIT2: Since the package showed up, I installed it... Looks like it needs some options added to it when the pbi is being built:


2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:17 unrecognized: 'sslcrtd_children'
2012/06/18 13:19:24| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/18 13:19:24| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/18 13:19:24| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/18 13:19:24| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/18 13:19:24| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/18 13:19:24| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:73 unrecognized: 'delay_pools'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:74 unrecognized: 'delay_class'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:75 unrecognized: 'delay_parameters'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:76 unrecognized: 'delay_initial_bucket_level'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:77 unrecognized: 'delay_access'


 2012/06/18 13:24:54| cache_cf.cc(381) parseOneConfigFile: squid-reverse.conf:11 unrecognized: 'netdb_filename'
2012/06/18 13:24:54| cache_cf.cc(381) parseOneConfigFile: squid-reverse.conf:16 unrecognized: 'sslcrtd_children'

It wont start, I manually was able to start squid by taking the unrecognized commands out.. hand edit the squid.inc file so they aren't added

EDIT3: Still testing but looks like option -f will be needed to keep the config files in the same location:

-f file Use given config-file instead of
/usr/pbi/squid-i386/etc/squid/squid.conf

jimp · Jun 18, 2012, 6:38 PM

What build_options were used when making the custom package? I can add whatever is needed to get it building. I tried adding ECAP and that just blew up the build.

If it isn't known, just get /var/db/ports/squid/options from the box that built the current .tbz and post it and I can translate it into the syntax we need.

And yes all packages with config files should be using whatever parameter is there like -f to manually specify where you want the config (should really be /var/etc/something, not /usr/local/etc/something)

Cino · Jun 18, 2012, 6:45 PM

thanks Jim, I'll let Marcelloc charm in on the dev stuff ;-)

marcelloc · Jun 18, 2012, 7:00 PM

Hi jimp,

these are the options on /var/db/ports/squid31/options

# This file is auto-generated by 'make config'.
# No user-servicable parts inside!
# Options for squid-3.1.19
_OPTIONS_READ=squid-3.1.19
WITH_SQUID_KERB_AUTH=true
WITH_SQUID_LDAP_AUTH=true
WITH_SQUID_NIS_AUTH=true
WITH_SQUID_SASL_AUTH=true
WITH_SQUID_IPV6=true
WITH_SQUID_DELAY_POOLS=true
WITH_SQUID_SNMP=true
WITH_SQUID_SSL=true
WITH_SQUID_SSL_CRTD=true
WITH_SQUID_PINGER=true
WITHOUT_SQUID_DNS_HELPER=true
WITH_SQUID_HTCP=true
WITH_SQUID_VIA_DB=true
WITH_SQUID_CACHE_DIGESTS=true
WITHOUT_SQUID_WCCP=true
WITH_SQUID_WCCPV2=true
WITHOUT_SQUID_STRICT_HTTP=true
WITH_SQUID_IDENT=true
WITH_SQUID_REFERER_LOG=true
WITH_SQUID_USERAGENT_LOG=true
WITH_SQUID_ARP_ACL=true
WITH_SQUID_IPFW=true
WITH_SQUID_PF=true
WITHOUT_SQUID_IPFILTER=true
WITH_SQUID_FOLLOW_XFF=true
WITHOUT_SQUID_ECAP=true
WITHOUT_SQUID_ICAP=true
WITHOUT_SQUID_ESI=true
WITH_SQUID_AUFS=true
WITHOUT_SQUID_COSS=true
WITHOUT_SQUID_KQUEUE=true
WITH_SQUID_LARGEFILE=true
WITHOUT_SQUID_STACKTRACES=true
WITHOUT_SQUID_DEBUG=true

jimp · Jun 18, 2012, 7:08 PM

sure your ports tree is up-to-date? (portsnap fetch extract, then go to that port and do 'make config' again) They changed the format of that file recently.

We need the format you posted this time, but I just wanted to make sure you had all of the possible config variables set.

EDIT: Looks like they were all set. I updated the pkg xml, as soon as the builders are done with their current jobs I'll try new builds.

marcelloc · Jun 18, 2012, 9:41 PM

i'll update my ports and check

compile options that are not checked:

SQUID_DNS_HELPER
SQUID_WCCP
SQUID_STRICT_HTTP
SQUID_IPFILTER
SQUID_ECAP
SQUID_YCAP
SQUID_ESI
SQUID_COSS
SQUID_KQUEUE
SQUID_STACKTRACES
SQUID_DEBUG

updated options filemore /var/db/ports/squid31/options


# This file is auto-generated by 'make config'.
# Options for squid-3.1.20
_OPTIONS_READ=squid-3.1.20
_FILE_COMPLETE_OPTIONS_LIST=SQUID_KERB_AUTH SQUID_LDAP_AUTH SQUID_NIS_AUTH SQUID_SASL_AUTH SQUID_IPV6 SQUID_DELAY_POOLS SQUID_SNMP SQUID_SSL SQUID_SSL_CRTD SQUID_PINGER SQUID_DNS_HELPER SQUID_HTCP SQUID_VIA_DB SQUID_CACHE_DIGESTS SQUID_WCCP SQUID_WCCPV2 SQUID_STRICT_HTTP SQUID_IDENT SQUID_REFERER_LOG SQUID_USERAGENT_LOG SQUID_ARP_ACL SQUID_IPFW SQUID_PF SQUID_IPFILTER SQUID_FOLLOW_XFF SQUID_ECAP SQUID_ICAP SQUID_ESI SQUID_AUFS SQUID_COSS SQUID_KQUEUE SQUID_LARGEFILE SQUID_STACKTRACES SQUID_DEBUG
OPTIONS_FILE_SET+=SQUID_KERB_AUTH
OPTIONS_FILE_SET+=SQUID_LDAP_AUTH
OPTIONS_FILE_SET+=SQUID_NIS_AUTH
OPTIONS_FILE_SET+=SQUID_SASL_AUTH
OPTIONS_FILE_SET+=SQUID_IPV6
OPTIONS_FILE_SET+=SQUID_DELAY_POOLS
OPTIONS_FILE_SET+=SQUID_SNMP
OPTIONS_FILE_SET+=SQUID_SSL
OPTIONS_FILE_SET+=SQUID_SSL_CRTD
OPTIONS_FILE_SET+=SQUID_PINGER
OPTIONS_FILE_UNSET+=SQUID_DNS_HELPER
OPTIONS_FILE_SET+=SQUID_HTCP
OPTIONS_FILE_SET+=SQUID_VIA_DB
OPTIONS_FILE_SET+=SQUID_CACHE_DIGESTS
OPTIONS_FILE_UNSET+=SQUID_WCCP
OPTIONS_FILE_SET+=SQUID_WCCPV2
OPTIONS_FILE_UNSET+=SQUID_STRICT_HTTP
OPTIONS_FILE_SET+=SQUID_IDENT
OPTIONS_FILE_SET+=SQUID_REFERER_LOG
OPTIONS_FILE_SET+=SQUID_USERAGENT_LOG
OPTIONS_FILE_SET+=SQUID_ARP_ACL
OPTIONS_FILE_SET+=SQUID_IPFW
OPTIONS_FILE_SET+=SQUID_PF
OPTIONS_FILE_UNSET+=SQUID_IPFILTER
OPTIONS_FILE_SET+=SQUID_FOLLOW_XFF
OPTIONS_FILE_UNSET+=SQUID_ECAP
OPTIONS_FILE_UNSET+=SQUID_ICAP
OPTIONS_FILE_UNSET+=SQUID_ESI
OPTIONS_FILE_SET+=SQUID_AUFS
OPTIONS_FILE_UNSET+=SQUID_COSS
OPTIONS_FILE_UNSET+=SQUID_KQUEUE
OPTIONS_FILE_SET+=SQUID_LARGEFILE
OPTIONS_FILE_UNSET+=SQUID_STACKTRACES
OPTIONS_FILE_UNSET+=SQUID_DEBUG

jimp · Jun 19, 2012, 1:10 AM

ok I think that lines up with what I have on there now (close enough :-)

has anyone tried the PBI in the last couple hours? The new one should be up now, at least for i386. I thought I uploaded another amd64 also that should be fixed.

Cino · Jun 19, 2012, 5:51 PM

@jimp:

ok I think that lines up with what I have on there now (close enough :-)

has anyone tried the PBI in the last couple hours? The new one should be up now, at least for i386. I thought I uploaded another amd64 also that should be fixed.

I just installed it and i'm getting the same errors:


: /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:17 unrecognized: 'sslcrtd_children'
2012/06/19 13:49:45| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/19 13:49:45| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/19 13:49:45| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/19 13:49:45| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/19 13:49:45| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/19 13:49:45| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:73 unrecognized: 'delay_pools'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:74 unrecognized: 'delay_class'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:75 unrecognized: 'delay_parameters'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:76 unrecognized: 'delay_initial_bucket_level'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:77 unrecognized: 'delay_access'

Still able to get squid3 to run with a few hand edits of squid.inc

jimp · Jun 19, 2012, 6:05 PM

Sure you reinstalled it all the way? Is it really 3.1.20?

The options for WITH_SQUID_SSL_CRTD and WITH_SQUID_DELAY_POOLS are present and set on the build config.

podilarius · Jun 19, 2012, 6:22 PM

I just tried to reload squid3 and I cannot get to any web sites. I am not getting any errors on startup any longer but I am am getting:

The following error was encountered while trying to retrieve the URL: /

Invalid URL

Some aspect of the requested URL is incorrect.

Some possible problems are:

Missing or incorrect access protocol (should be http:// or similar)

Missing hostname

Illegal double-escape in the URL-Path

Illegal character in hostname; underscores are not allowed.

Your cache administrator is webmaster.

I get this on google and yahoo with squid3 installed. I have not had this problem in the past. I see a new binary, so I will try that with a gitsync to see if any new changes will fix that.

Cino · Jun 19, 2012, 6:52 PM

its 3.1.20… I don't see it as a configured option ???

For a band-aid, option '--sysconfdir=/usr/pbi/squid-i386/etc/squid' should be '--sysconfdir=/usr/local/etc/squid' ::)


: squid -v
Squid Cache: Version 3.1.20
configure options:  '--with-default-user=squid' '--bindir=/usr/pbi/squid-i386/sbin' '--sbindir=/usr/pbi/squid-i386/sbin' '--datadir=/usr/pbi/squid-i386/etc/squid' '--libexecdir=/usr/pbi/squid-i386/libexec/squid' '--localstatedir=/var/squid' '--sysconfdir=/usr/pbi/squid-i386/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--disable-translation' '--enable-auth=basic digest negotiate ntlm' '--enable-basic-auth-helpers=DB NCSA PAM MSNT SMB squid_radius_auth YP' '--enable-digest-auth-helpers=password' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group' '--enable-ntlm-auth-helpers=smb_lm' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-storeio=ufs diskd aufs' '--enable-disk-io=AIO Blocking DiskDaemon DiskThreads' '--disable-ecap' '--disable-loadable-modules' '--enable-kqueue' '--prefix=/usr/pbi/squid-i386' '--mandir=/usr/pbi/squid-i386/man' '--infodir=/usr/pbi/squid-i386/info/' '--build=i386-portbld-freebsd8.1' 'build_alias=i386-portbld-freebsd8.1' 'CC=cc' 'CFLAGS=-O2 -pipe  -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS=' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -fno-strict-aliasing' 'CPP=cpp' --with-squid=/usr/wrkdirprefix/usr/ports/www/squid31/work/squid-3.1.20 --enable-ltdl-convenience

jimp · Jun 19, 2012, 6:58 PM

No, the config and startup script should be manually pointing it to the right place, we're not hacking up configure options, we're trying to keep the builds automated. :-)
(And it should really be /var/etc/squid not /usr/local/etc/squid …)

# grep WITH /pbi-build/modules/www/squid31/pbi.conf      
MAKEOPTS="WITHOUT_X11=true WITH_SQUID_KERB_AUTH=true WITH_SQUID_LDAP_AUTH=true WITH_SQUID_NIS_AUTH=true WITH_SQUID_SASL_AUTH=true WITH_SQUID_IPV6=true WITH_SQUID_DELAY_POOLS=true WITH_SQUID_SNMP=true WITH_SQUID_SSL=true WITH_SQUID_SSL_CRTD=true WITH_SQUID_PINGER=true WITHOUT_SQUID_DNS_HELPER=true WITH_SQUID_HTCP=true WITH_SQUID_VIA_DB=true WITH_SQUID_CACHE_DIGESTS=true WITHOUT_SQUID_WCCP=true WITH_SQUID_WCCPV2=true WITHOUT_SQUID_STRICT_HTTP=true WITH_SQUID_IDENT=true WITH_SQUID_REFERER_LOG=true WITH_SQUID_USERAGENT_LOG=true WITH_SQUID_ARP_ACL=true WITH_SQUID_IPFW=true WITH_SQUID_PF=true WITHOUT_SQUID_IPFILTER=true WITH_SQUID_FOLLOW_XFF=true WITHOUT_SQUID_ECAP=true WITHOUT_SQUID_ICAP=true WITHOUT_SQUID_ESI=true WITH_SQUID_AUFS=true WITHOUT_SQUID_COSS=true WITHOUT_SQUID_KQUEUE=true WITH_SQUID_LARGEFILE=true WITHOUT_SQUID_STACKTRACES=true WITHOUT_SQUID_DEBUG=true"

tgbauer · Jun 19, 2012, 7:07 PM

I just tried to install the latest squid3 (3.1.20 pkg 2.0.5_2) on pfSense 2.0.1-RELEASE (i386) after uninstalling the prior version.
When squid was started it would exit with the following error:
/libexec/ld-elf.so.1: Shared object "libmd5.so.1" not found, required by "squid"

pkg_add -r libwww
failed because freeBSD 8.1 has been moved to "Archive"
I was able to install the package using:

pkg_add -r http://ftp2.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.1-release/www/libwww-5.4.0_4.tbz

I'm sure others may have a better way to get around this problem, but this is what worked for me.

Cino · Jun 19, 2012, 7:10 PM

I agree with you… I may try to hack my install to see if I can get it to point to /var/etc/squid but not sure on how to have it create the folders and such(other then by hand)... Never really looked at the installation part of the inc & xml files.

Anything in the build log saying its missing something or errors?

Take a look at this post, http://forum.pfsense.org/index.php/topic,44735.msg252767.html#msg252767

I did a stare and compare and it looks like there are options not being built.

jimp · Jun 20, 2012, 12:50 PM

OK, I just uploaded a new set of squid3 binaries, can someone upgrade and see if the options are there now?

I also added libwww as a manual dependency to install for 2.0.x so it should hopefully also fix the libmd5 bit.

Cino · Jun 20, 2012, 1:15 PM

@jimp:

OK, I just uploaded a new set of squid3 binaries, can someone upgrade and see if the options are there now?

I also added libwww as a manual dependency to install for 2.0.x so it should hopefully also fix the libmd5 bit.

I'll give it a shot in a few minutes… Btw, should keep posting my findings on this topic or here http://forum.pfsense.org/index.php/topic,50493.0.html ? Don't like having more then 1 thread on the same issue. I posted here because this was the official topic for squid3

EDIT: I gave it try, same issue.. Options are not there... I have noticed that I can do a re-install with squid. It doesn't bring down the pbi file. I have to uninstall then install for it to download the pbi package.

jimp · Jun 20, 2012, 2:13 PM

Other thread is probably better, I lost track of which thread it was and there were similar posts in each, other has a more accurate subject and relevance.

nutt318 · Jun 20, 2012, 4:05 PM

Can anyone provide some help with the Reverse Proxy?

My posts might have been overlooked in a couple pages back.

Does anyone else have the Reverse Proxy working?

jimp · Jun 20, 2012, 4:13 PM

nutt318 - perhaps you should start a new thread with a separate subject, it will get more attention that way.

nesense · Jun 24, 2012, 4:32 PM

Hi :), been testing squid3 on embedded and full version 2.0.1 and I noticed its not caching items, /var/squid/cache dir remains the same size (works fine on full but not on nanobsd)

BTW, I think COSS filesystem is very useful, specially for SSD/flash storage and embedded in general, lowers wear on cells, its best used mixed with AUFS, with COSS caching smaller objects and the rest for AUFS.

Thank you :)