Squid3 - New GUI with sync, normal and reverse proxy

marcelloc

i'll update my ports and check

compile options that are not checked:

SQUID_DNS_HELPER
SQUID_WCCP
SQUID_STRICT_HTTP
SQUID_IPFILTER
SQUID_ECAP
SQUID_YCAP
SQUID_ESI
SQUID_COSS
SQUID_KQUEUE
SQUID_STACKTRACES
SQUID_DEBUG

updated options filemore /var/db/ports/squid31/options


# This file is auto-generated by 'make config'.
# Options for squid-3.1.20
_OPTIONS_READ=squid-3.1.20
_FILE_COMPLETE_OPTIONS_LIST=SQUID_KERB_AUTH SQUID_LDAP_AUTH SQUID_NIS_AUTH SQUID_SASL_AUTH SQUID_IPV6 SQUID_DELAY_POOLS SQUID_SNMP SQUID_SSL SQUID_SSL_CRTD SQUID_PINGER SQUID_DNS_HELPER SQUID_HTCP SQUID_VIA_DB SQUID_CACHE_DIGESTS SQUID_WCCP SQUID_WCCPV2 SQUID_STRICT_HTTP SQUID_IDENT SQUID_REFERER_LOG SQUID_USERAGENT_LOG SQUID_ARP_ACL SQUID_IPFW SQUID_PF SQUID_IPFILTER SQUID_FOLLOW_XFF SQUID_ECAP SQUID_ICAP SQUID_ESI SQUID_AUFS SQUID_COSS SQUID_KQUEUE SQUID_LARGEFILE SQUID_STACKTRACES SQUID_DEBUG
OPTIONS_FILE_SET+=SQUID_KERB_AUTH
OPTIONS_FILE_SET+=SQUID_LDAP_AUTH
OPTIONS_FILE_SET+=SQUID_NIS_AUTH
OPTIONS_FILE_SET+=SQUID_SASL_AUTH
OPTIONS_FILE_SET+=SQUID_IPV6
OPTIONS_FILE_SET+=SQUID_DELAY_POOLS
OPTIONS_FILE_SET+=SQUID_SNMP
OPTIONS_FILE_SET+=SQUID_SSL
OPTIONS_FILE_SET+=SQUID_SSL_CRTD
OPTIONS_FILE_SET+=SQUID_PINGER
OPTIONS_FILE_UNSET+=SQUID_DNS_HELPER
OPTIONS_FILE_SET+=SQUID_HTCP
OPTIONS_FILE_SET+=SQUID_VIA_DB
OPTIONS_FILE_SET+=SQUID_CACHE_DIGESTS
OPTIONS_FILE_UNSET+=SQUID_WCCP
OPTIONS_FILE_SET+=SQUID_WCCPV2
OPTIONS_FILE_UNSET+=SQUID_STRICT_HTTP
OPTIONS_FILE_SET+=SQUID_IDENT
OPTIONS_FILE_SET+=SQUID_REFERER_LOG
OPTIONS_FILE_SET+=SQUID_USERAGENT_LOG
OPTIONS_FILE_SET+=SQUID_ARP_ACL
OPTIONS_FILE_SET+=SQUID_IPFW
OPTIONS_FILE_SET+=SQUID_PF
OPTIONS_FILE_UNSET+=SQUID_IPFILTER
OPTIONS_FILE_SET+=SQUID_FOLLOW_XFF
OPTIONS_FILE_UNSET+=SQUID_ECAP
OPTIONS_FILE_UNSET+=SQUID_ICAP
OPTIONS_FILE_UNSET+=SQUID_ESI
OPTIONS_FILE_SET+=SQUID_AUFS
OPTIONS_FILE_UNSET+=SQUID_COSS
OPTIONS_FILE_UNSET+=SQUID_KQUEUE
OPTIONS_FILE_SET+=SQUID_LARGEFILE
OPTIONS_FILE_UNSET+=SQUID_STACKTRACES
OPTIONS_FILE_UNSET+=SQUID_DEBUG

jimp

ok I think that lines up with what I have on there now (close enough :-)

has anyone tried the PBI in the last couple hours? The new one should be up now, at least for i386. I thought I uploaded another amd64 also that should be fixed.

Cino

@jimp:

ok I think that lines up with what I have on there now (close enough :-)

has anyone tried the PBI in the last couple hours? The new one should be up now, at least for i386. I thought I uploaded another amd64 also that should be fixed.

I just installed it and i'm getting the same errors:


: /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:17 unrecognized: 'sslcrtd_children'
2012/06/19 13:49:45| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/19 13:49:45| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/19 13:49:45| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/19 13:49:45| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/19 13:49:45| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/19 13:49:45| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:73 unrecognized: 'delay_pools'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:74 unrecognized: 'delay_class'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:75 unrecognized: 'delay_parameters'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:76 unrecognized: 'delay_initial_bucket_level'
2012/06/19 13:49:45| cache_cf.cc(381) parseOneConfigFile: squid.conf:77 unrecognized: 'delay_access'

Still able to get squid3 to run with a few hand edits of squid.inc

jimp

Sure you reinstalled it all the way? Is it really 3.1.20?

The options for WITH_SQUID_SSL_CRTD and WITH_SQUID_DELAY_POOLS are present and set on the build config.

podilarius

I just tried to reload squid3 and I cannot get to any web sites. I am not getting any errors on startup any longer but I am am getting:

The following error was encountered while trying to retrieve the URL: /

Invalid URL

Some aspect of the requested URL is incorrect.

Some possible problems are:

Missing or incorrect access protocol (should be http:// or similar)

Missing hostname

Illegal double-escape in the URL-Path

Illegal character in hostname; underscores are not allowed.

Your cache administrator is webmaster.

I get this on google and yahoo with squid3 installed. I have not had this problem in the past. I see a new binary, so I will try that with a gitsync to see if any new changes will fix that.

Cino

its 3.1.20… I don't see it as a configured option ???

For a band-aid, option '--sysconfdir=/usr/pbi/squid-i386/etc/squid' should be '--sysconfdir=/usr/local/etc/squid' ::)


: squid -v
Squid Cache: Version 3.1.20
configure options:  '--with-default-user=squid' '--bindir=/usr/pbi/squid-i386/sbin' '--sbindir=/usr/pbi/squid-i386/sbin' '--datadir=/usr/pbi/squid-i386/etc/squid' '--libexecdir=/usr/pbi/squid-i386/libexec/squid' '--localstatedir=/var/squid' '--sysconfdir=/usr/pbi/squid-i386/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--disable-translation' '--enable-auth=basic digest negotiate ntlm' '--enable-basic-auth-helpers=DB NCSA PAM MSNT SMB squid_radius_auth YP' '--enable-digest-auth-helpers=password' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group' '--enable-ntlm-auth-helpers=smb_lm' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-storeio=ufs diskd aufs' '--enable-disk-io=AIO Blocking DiskDaemon DiskThreads' '--disable-ecap' '--disable-loadable-modules' '--enable-kqueue' '--prefix=/usr/pbi/squid-i386' '--mandir=/usr/pbi/squid-i386/man' '--infodir=/usr/pbi/squid-i386/info/' '--build=i386-portbld-freebsd8.1' 'build_alias=i386-portbld-freebsd8.1' 'CC=cc' 'CFLAGS=-O2 -pipe  -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS=' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -fno-strict-aliasing' 'CPP=cpp' --with-squid=/usr/wrkdirprefix/usr/ports/www/squid31/work/squid-3.1.20 --enable-ltdl-convenience

jimp

No, the config and startup script should be manually pointing it to the right place, we're not hacking up configure options, we're trying to keep the builds automated. :-)
(And it should really be /var/etc/squid not /usr/local/etc/squid …)

# grep WITH /pbi-build/modules/www/squid31/pbi.conf      
MAKEOPTS="WITHOUT_X11=true WITH_SQUID_KERB_AUTH=true WITH_SQUID_LDAP_AUTH=true WITH_SQUID_NIS_AUTH=true WITH_SQUID_SASL_AUTH=true WITH_SQUID_IPV6=true WITH_SQUID_DELAY_POOLS=true WITH_SQUID_SNMP=true WITH_SQUID_SSL=true WITH_SQUID_SSL_CRTD=true WITH_SQUID_PINGER=true WITHOUT_SQUID_DNS_HELPER=true WITH_SQUID_HTCP=true WITH_SQUID_VIA_DB=true WITH_SQUID_CACHE_DIGESTS=true WITHOUT_SQUID_WCCP=true WITH_SQUID_WCCPV2=true WITHOUT_SQUID_STRICT_HTTP=true WITH_SQUID_IDENT=true WITH_SQUID_REFERER_LOG=true WITH_SQUID_USERAGENT_LOG=true WITH_SQUID_ARP_ACL=true WITH_SQUID_IPFW=true WITH_SQUID_PF=true WITHOUT_SQUID_IPFILTER=true WITH_SQUID_FOLLOW_XFF=true WITHOUT_SQUID_ECAP=true WITHOUT_SQUID_ICAP=true WITHOUT_SQUID_ESI=true WITH_SQUID_AUFS=true WITHOUT_SQUID_COSS=true WITHOUT_SQUID_KQUEUE=true WITH_SQUID_LARGEFILE=true WITHOUT_SQUID_STACKTRACES=true WITHOUT_SQUID_DEBUG=true"

tgbauer

I just tried to install the latest squid3 (3.1.20 pkg 2.0.5_2) on pfSense 2.0.1-RELEASE (i386) after uninstalling the prior version.
When squid was started it would exit with the following error:
/libexec/ld-elf.so.1: Shared object "libmd5.so.1" not found, required by "squid"

pkg_add -r libwww
failed because freeBSD 8.1 has been moved to "Archive"
I was able to install the package using:

pkg_add -r http://ftp2.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.1-release/www/libwww-5.4.0_4.tbz

I'm sure others may have a better way to get around this problem, but this is what worked for me.

Cino

I agree with you… I may try to hack my install to see if I can get it to point to /var/etc/squid but not sure on how to have it create the folders and such(other then by hand)... Never really looked at the installation part of the inc & xml files.

Anything in the build log saying its missing something or errors?

Take a look at this post, http://forum.pfsense.org/index.php/topic,44735.msg252767.html#msg252767

I did a stare and compare and it looks like there are options not being built.

jimp

OK, I just uploaded a new set of squid3 binaries, can someone upgrade and see if the options are there now?

I also added libwww as a manual dependency to install for 2.0.x so it should hopefully also fix the libmd5 bit.

Cino

@jimp:

OK, I just uploaded a new set of squid3 binaries, can someone upgrade and see if the options are there now?

I also added libwww as a manual dependency to install for 2.0.x so it should hopefully also fix the libmd5 bit.

I'll give it a shot in a few minutes… Btw, should keep posting my findings on this topic or here http://forum.pfsense.org/index.php/topic,50493.0.html ? Don't like having more then 1 thread on the same issue. I posted here because this was the official topic for squid3

EDIT: I gave it try, same issue.. Options are not there... I have noticed that I can do a re-install with squid. It doesn't bring down the pbi file. I have to uninstall then install for it to download the pbi package.

jimp

Other thread is probably better, I lost track of which thread it was and there were similar posts in each, other has a more accurate subject and relevance.

nutt318

Can anyone provide some help with the Reverse Proxy?

My posts might have been overlooked in a couple pages back.

Does anyone else have the Reverse Proxy working?

jimp

nutt318 - perhaps you should start a new thread with a separate subject, it will get more attention that way.

nesense

Hi :), been testing squid3 on embedded and full version 2.0.1 and I noticed its not caching items, /var/squid/cache dir remains the same size (works fine on full but not on nanobsd)

BTW, I think COSS filesystem is very useful, specially for SSD/flash storage and embedded in general, lowers wear on cells, its best used mixed with AUFS, with COSS caching smaller objects and the rest for AUFS.

Thank you :)

jvorhees

Hello,
and first thanks for the great work on this mandatory feature in pfsense (also mandatory in serious network :p)

It seems however that we are plenty to have the problem that the reverse setup of squid send traffic always to the default site defined on general tab (https talking…)
I've tried with empty and * on URI field, no luck.

My setup is:

WORLD ---- PFSENSE/SQUID ---- extranet1.mydomain.com
|
-- extranet2.mydomain.com

I've also tried to replace url_regex with dstdomain, squid won't start.

In the log i've:
… X.X.X.X TCP_MISS/404 588 GET https://extranet2.mydomain.com/ - FIRST_UP_PARENT/Extranet2 text/html

The only thing i could do to make it work is to change default site to send traffic to other host (so i think firewalling and basic syntax for my squid setup is OK)

Any ideas ?

Thanks again for the all work !!!

marcelloc

I did some update to the package without version change, please try to reinstall or uninstall/reinstall and test again.

att,
Marcello Coutinho

dhipo

i want work on squidguard sync too .. can you send me way to do this …like squid ?

marcelloc

@dhipo:

i want work on squidguard sync too .. can you send me way to do this …like squid ?

You will need to code it on squidguard inc file and create/or use squid_sync xml file.

If you don't know how pfsense gui works, it will be not that easy.

take a look on ipguard package files, it's a small package with sync code.

att,
Marcello Coutinho

dhipo

obrigado Marcello..

onde vc está no Br ?

marcelloc

@dhipo:

onde vc está no Br ?

Take a look on my profile and on portuguese forum ;)

_igor_

seems that squid doesn't accept any entry in "custom options, i get errors:

php: /pkg_edit.php: The command '/usr/local/sbin/squid -k reconfigure -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was 'FATAL: Bungled squid.conf line 83: http_port 8080 transparent; Squid Cache (Version 3.1.20): Terminated abnormally. CPU Usage: 0.018 seconds = 0.009 user + 0.009 sys Maximum Resident Size: 5176 KB Page faults with physical i/o: 0'

I tried with this setup here: http://forum.pfsense.org/index.php/topic,42413.0.html.

I stripped the ident-entries, because they don't work too :(

marcelloc

@_igor_:

seems that squid doesn't accept any entry in "custom options, i get errors:
the output was 'FATAL: Bungled squid.conf line 83: http_port 8080 transparent;

Is it your custom option? http_port 8080 transparent?

_igor_

yes it is. But other custom options generate the same kind of errors, so i suppose, custom options don't work here. :(

marcelloc

@_igor_:

yes it is.

try ** http_port 8080 intercept** on squid 3.1

_igor_

None of any entry works. All entries result in a "bungled squid config". It doesnt work. :(

marcelloc

what do you have on squid.conf file after applying your config?

The squid gui includes http_port config.

_igor_

Here we go:

# Do not edit manually !
http_port 192.168.1.1:8080
icp_port 7

pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language de
icon_directory /usr/local/etc/squid/icons
visible_hostname pfsense
cache_mgr hier@da.de
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 0
shutdown_lifetime 3 seconds
uri_whitespace encode

acl dynamic urlpath_regex cgi-bin \?
cache deny dynamic
cache_mem 8 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir ufs /var/squid/cache 1000000 16 256
minimum_object_size 0 KB
maximum_object_size 4 KB
offline_mode offcache_swap_low 90
cache_swap_high 95

# No redirector configured

#Remote proxies

# Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 1025-65535 
acl sslports port 443 563  
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT

acl allowed_subnets src 192.168.1.0/24
acl whitelist dstdom_regex -i "/var/squid/acl/whitelist.acl"
http_access allow manager localhost

# Allow external cache managers
acl ext_manager src 127.0.0.1
acl ext_manager src 192.168.1.1
acl ext_manager src 
http_access allow manager ext_manager

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings

# Package Integration
redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
redirector_bypass on
redirect_children 3

# Custom options
# http_port 8080 intercept;
ident_lookup_access allow all;
ident_timeout 3 seconds;

# Always allow access to whitelist domains
http_access allow whitelist
# Setup allowed acls
http_access allow allowed_subnets
# Default block all to be sure
http_access deny allsrc

and here the log-entry:

php: /pkg_edit.php: The command '/usr/local/sbin/squid -k reconfigure -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was '2012/07/18 16:32:18| aclParseAclList: ACL name 'all;' not found. FATAL: Bungled squid.conf line 84: ident_lookup_access allow all; Squid Cache (Version 3.1.20): Terminated abnormally. CPU Usage: 0.018 seconds = 0.018 user + 0.000 sys Maximum Resident Size: 4112 KB Page faults with physical i/o: 0'

Do you need more info?

marcelloc

Custom options

http_port 8080 intercept;

ident_lookup_access allow all;
ident_timeout 3 seconds;

Note that only your custom options has ";" at the end.

Remove it and test again.

_igor_

duh! That was it. Thx for your help!!

nesense

Hello, just reinstalled squid3 and it is still not caching. logs give TCP_MISS/200

thank you

al_reidy

my HD died in the router. on a fresh reinstall i can install the squid3 package but i get this in the system log.

php: /pkg_mgr_install.php: XML error: Mismatched tag at line 348 in /usr/local/pkg/squid.xml

the package doesnt have any menu item options either.

corrupt package?

al_reidy

the line in the xml refers to :

<default_value>en</default_value>

as part of

<field><fielddescr>Language</fielddescr>
<fieldname>error_language</fieldname>
<description>Select the language in which the proxy server will display error messages to users.</description>
<type>select</type>
<default_value>en</default_value></field>

podilarius

a pull request has been submited by phil that should fix that. We are waiting on a dev to pull in that request.

marcelloc

@al_reidy:

the line in the xml refers to :

<default_value>en</default_value>

It's fixed now, wait 15 minutes and reinstall

att,
Marcello Coutinho

podilarius

Thanks!

phil.davis

Apologies to all - my little "fix" for the default language had a bit of extra cut-and-paste somehow. I noticed it soon after it got committed, but unfortunately it took quite a while for anyone with the necessary privs to commit the better version. I'll stare a lot harder at seemingly mindless fixes in future!

al_reidy

@phil.davis:

Apologies to all - my little "fix" for the default language had a bit of extra cut-and-paste somehow. I noticed it soon after it got committed, but unfortunately it took quite a while for anyone with the necessary privs to commit the better version. I'll stare a lot harder at seemingly mindless fixes in future!

Its working now. Thank you all for your help. ;D

nutt318

I would like to report a bug in the Squid Reverse Proxy. Under the HTTPS settings the default port used is 443 if you leave the feild empty. However if you look at squid.conf in /usr/local/etc/squid the listening port is still 80. So you have to manually put in 443 for the listening port.

Even though I found this bug I'm still having issues with getting the HTTPS reverse proxy working correctly.

I'm getting a squid error page saying Access Denied. Access control configuration prevents your request from being allowed at this time.

Not sure of the problem but it looks like others are having problems with the reverse proxy on HTTPS as well.
http://forum.pfsense.org/index.php/topic,51945.0.html

dhatz

Folks, is there any reason why in proxy_monitor.sh you're using a series of pipes (btw both awk seem redundant)

NUM_PROCS=`ps auxw | grep "[s]quid -D"|awk '{print $2}'| wc -l | awk '{ print $1 }'`

rather than a simpler pgrep -f "process" | wc -l ?[/s]