Squid 3.3.4 package for pfsense with ssl filtering



  • Hi all,

    first devel version of squid 3.3 for pfsense is out.

    Main changes

    • Update squid to latest stable version 3.3.4

    • Per interface proxy mode

    • Per interface transparent proxy mode

    • Per interface ssl proxy mode

    • Antivirus integration via i-cap

    • Option to include squidguard denied logs on squid log too (requires this option and a small update on sgerror.php)

    know bugs

    • Clamav integration via i-cap is not working(loops until crash)

    att,
    Marcello Coutinho

    moderator edit:  removed links no longer necessary triggering malware alerts for some users.



  • Awesome! I know many are looking forward to this.  I have a few questions:

    Does the ssl proxy mode work in transparent, normal, or both?
    Is this package something we can use with DG to try out the ssl filtering or is there more work needed on DG before that should work?
    How are the certificates set up?  I know the pfsense box should act as a certificate authority and all the clients must trust it.  So is the CA cert automatically generated and how do you get it to the clients?  Or can you use an existing PKI to generate one for it?



  • just tried to install it and it failed :-(
    2.1-BETA1 (i386) built on Fri May 10 16:28:23 EDT 2013

    
    Beginning package installation for squid3-dev .
    Downloading package configuration file... done.
    Saving updated package information... done.
    Downloading squid3-dev and its dependencies... 
    Checking for package installation... 
     Downloading http://files.pfsense.org/packages/8/All/squid-3.3.4-i386.pbi ...  (extracting)
    Loading package configuration... done.
    Configuring package components...
    Additional files... sqpmon.sh failed.
    Backing up libraries... 
    Removing package...
    Starting package deletion for squid-3.3.4-i386...
    
    


  • @wheelz:

    Does the ssl proxy mode work in transparent, normal, or both?

    both.

    @wheelz:

    Is this package something we can use with DG to try out the ssl filtering or is there more work needed on DG before that should work?

    with ssl, mybe not as squid need always direct directive.
    if dansguardian pass it to squid and then ssl is filtered, then it works.

    @wheelz:

    How are the certificates set up?  I know the pfsense box should act as a certificate authority and all the clients must trust it.  So is the CA cert automatically generated and how do

    Default web configurator certificate can be used.

    @wheelz:

    you get it to the clients?  Or can you use an existing PKI to generate one for it?

    you may need to export crt to clients to avoid most of ssl client check erros



  • @Cino:

    Additional files… sqpmon.sh failed.

    Fixed. retry in 15 minutes  :)



  • @marcelloc:

    with ssl, mybe not as squid need always direct directive.
    if dansguardian pass it to squid and then ssl is filtered, then it works.

    I'm not sure I understand what you mean.  Should I take that to mean, try it and see?  :)



  • Hi Marcello,

    this are really amazing features. Transparent SSL filtering and an up to date anti-virus feature ist great!

    Thank you very much for the very very hard work on that! I hope I will find some time to test all the festures in the near future  ;D



  • On pfsense2.1 snapshot from first week of may 2013 I uninstalled squid 3.1 and installed squid 3.3.4

    I noticed that the squid service was not starting anymore and there were 2 new services not starting as well.

    the other 2 services are:
    Icap inteface for squid and clamav integration
    Clamav Antivirus

    Also i noticed that after installing squid 3.3.4 my old config was still there…...handy but not sure if was a very clean installation.

    so i installed in a new machine the latest 2.1 snapshot from today may 13th and installed the latest squid 3.3.4
    and the very same issue is still there, squid service is still not starting even in an absolutely clean install.

    what should i do to make it work?



  • quetzalcoatl,

    @marcelloc:

    know bugs

    • Clamav integration via i-cap is not working(loops until crash)

    It is a known bug that I'm sure he is working on.



  • thanks, and i supposed that those new 2 services still need to be fixed.
    I don't even need the antivirus and stuff anyways.
    I just need the squid3.3 service to start but it does not.



  • @quetzalcoatl:

    I just need the squid3.3 service to start but it does not.

    What you get on squid logs? I've did a clean install and service is up with antivirus disabled.

    squid -NsXY is a good way to find what is not working.

    Did you saved config after package installing 3.3?



  • it installs now but its missing a shared object.. I haven't had a chance to to see if its on the box and just needs to be linked.

    
    php: : The command '/usr/pbi/squid-i386/sbin/squid -f /usr/pbi/squid-i386/etc/squid/squid.conf' returned exit code '1', the output was '/libexec/ld-elf.so.1: Shared object "libgssapi.so.10" not found, required by "squid"'
    
    


  • @marcelloc:

    @quetzalcoatl:

    I just need the squid3.3 service to start but it does not.

    What you get on squid logs? I've did a clean install and service is up with antivirus disabled.

    squid -NsXY is a good way to find what is not working.

    Did you saved config after package installing 3.3?

    i don't even know where to look for squid logs.
    also as soon as i install squid 3.3 the icap and clamd services get installed by themselves and i don't know how to remove them (if i have to).

    if i go to the pfsense console and hit 8 to get into the shell, then i write "squid -NsXY"
    it says "/libexec/lds-elf.so.1: Shared object "libgssapi.so.10" not found, required by "squid".

    I did not save my squid config before installing 3.3 but the old config was still there.

    Right now i have 2 virtual machines, 1 with pfsense 2.1 with reinstalled squid 3.3 from 3.1, and another one with the today's pfsense snapshot and clean install of squid 3.3.

    I see "/libexec/lds-elf.so.1: Shared object "libgssapi.so.10" not found, required by "squid" in both virtual machines.



  • I hope the file libgssapi.so.10 will be added to the next pfsense snapshots or within the next release of squid3.3

    I have been trying all evening to get this file into that /usr/local/lib folder
    but i did not succeed.

    How do i copy or move a file from the internet to a certain folder inside pfsense?
    Using the GUI i can upload a file but i don't know how to place it in /usr/local/lib



  • @quetzalcoatl:

    How do i copy or move a file from the internet to a certain folder?

    using console/ssh

    • cd /usr/local/lib

    • fetch url_for_libs

    Download all libs from my ldd folder.



  • Thanks to you now I'm a little less ignorant about pfsense and squid! :P

    I did copy several files from http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/ to /usr/local/lib

    until when i do squid -NsXY it doesn't ask me any more for missing files but now squid still service doesn't start and when i do squid -NsXY the shell shows me a bunch of:
    "Acl.cc(353) ~ACL: ACL::~ACL: ´"

    and i don't know how to see above these lines because there are too many.

    I don't know how to scroll text lines inside the shell.
    (tons of ignorance here! sorry)



  • Hi marcelloc,

    I tried it with my normal configuration (pfSense -> DG -> Squid3 -> Internet) and couldn't get it to work.

    DG on pfSense:8080 with Squid as parent
    Squid3 on 127.0.0.1:3128



  • @Legion:

    DG on pfSense:8080 with Squid as parent
    Squid3 on 127.0.0.1:3128

    Check if squid is running, on log you sent I can see only warnings.

    squid -NsXY on console can show you squid startup error or /var/squid/logs/cache.log



  • Hi, I am having problems using squid 3.3
    I can not start the service ..
    Returns the following error ..

    May 14 14:49:28	php: /pkg_edit.php: Starting Squid
    May 14 14:49:28	squid: Bungled squid.conf line 45: offline_mode offcache_swap_low 90
    May 14 14:49:28	php: /pkg_edit.php: The command '/usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was '2013/05/14 14:49:28| Warning: empty ACL: acl localnet src FATAL: Bungled squid.conf line 45: offline_mode offcache_swap_low 90 Squid Cache (Version 3.3.4): Terminated abnormally. CPU Usage: 0.021 seconds = 0.021 user + 0.000 sys Maximum Resident Size: 26512 KB Page faults with physical i/o: 0'
    May 14 14:49:38	check_reload_status: Reloading filter
    


  • I've been trying to get my "Squid3-dev"+"Squidguard" config running (no ssl & no antivirus active, not yet anyway)

    Copied the libs and chmod 755, used WinSCP.

    First warning:

    php: /pkg_edit.php: The command '/usr/pbi/squid-amd64/sbin/squid -f /usr/pbi/squid-amd64/etc/squid/squid.conf' returned exit code '1', the output was '2013/05/14 19:27:02| Warning: empty ACL: acl localnet src FATAL: Bungled squid.conf line 33: offline_mode offcache_swap_low 90 Squid Cache (Version 3.3.4): Terminated abnormally. CPU Usage: 0.038 seconds = 0.030 user + 0.008 sys Maximum Resident Size: 39824 KB Page faults with physical i/o: 0

    Squid complains "localnet" was not set correctly. I changed the code to work.
    I hardcoded my localnet into "squid.inc" so the squid.conf line would read "acl localnet src 192.168.0.1/24"
    So that was a quick and very dirty temporary fix. (but hey it works…)

    
    	if ($settings['allow_interface'] == 'on') {
    		$src = '';
    		foreach ($real_ifaces as $iface) {
    			list($ip, $mask) = $iface;
    			$ip = long2ip(ip2long($ip) & ip2long($mask));
    			$mask = 32-log((ip2long($mask) ^ ip2long('255.255.255.255'))+1,2);
    			$src .= " $ip/$mask";
    		}
    		$conf .= "# Allow local network(s) on interface(s)\n";
    		$conf .= "acl localnet src 192.168.0.0/24\n";
    		$valid_acls[] = 'localnet';
    
    

    Second warning:

    php: /pkg_edit.php: The command '/usr/pbi/squid-amd64/sbin/squid -f /usr/pbi/squid-amd64/etc/squid/squid.conf' returned exit code '1', the output was '2013/05/14 19:29:35| WARNING: Netmasks are deprecated. Please use CIDR masks instead. 2013/05/14 19:29:35| WARNING: IPv4 netmasks are particularly nasty when used to compare IPv6 to IPv4 ranges. 2013/05/14 19:29:35| WARNING: For now we will assume you meant to write /24 FATAL: Bungled squid.conf line 33: offline_mode offcache_swap_low 90 Squid Cache (Version 3.3.4): Terminated abnormally. CPU Usage: 0.038 seconds = 0.031 user + 0.008 sys Maximum Resident Size: 38832 KB Page faults with physical i/o: 0

    Hmmm.. "offline_mode offcache_swap_low 90" not good…
    So checked "squid.inc" again, added extra linefeed between "offline_mode" and "EOD".

    cache_mem $memory_cache_size MB
    maximum_object_size_in_memory {$max_objsize_in_mem} KB
    memory_replacement_policy {$memory_policy}
    cache_replacement_policy {$cache_policy}
    $disk_cache_opts
    minimum_object_size {$min_objsize} KB
    maximum_object_size {$max_objsize}
    offline_mode {$offline_mode}
    
    EOD;
    

    Changes for "squidguard_configurator.inc" (replaced redirector commands, for squid 3.3 compatibility)

    # ------------------------------------------------------------------------------
    # squid config options
    # ------------------------------------------------------------------------------
    define('REDIRECTOR_OPTIONS_REM',   '# squidGuard options');
    define('REDIRECTOR_PROGRAM_OPT',   'url_rewrite_program');
    define('REDIRECT_BYPASS_OPT',      'url_rewrite_bypass');
    define('REDIRECT_CHILDREN_OPT',    'url_rewrite_children');
    define('REDIRECTOR_PROCESS_COUNT', '5'); # redirector processes count will started
    
    

    Clear old settings from the "Proxy server" page, then save on the "proxy filter" page again so the new redirector commands are used.



  • I've pushed a fix for some of these issues

    on 2.0.x, if you got squid running but no port listening, you may try squid 3.3.4 from my repo compiled without ipv6.

    amd64
    http://e-sac.siteseguro.ws/packages/amd64/8/All/squid-3.3.4.tbz

    i386
    http://e-sac.siteseguro.ws/packages/8/All/squid-3.3.4.tbz

    Some lib replacements may require a reboot to avoid squid crashes.



  • Great package, thank you!
    On my i386 test KVM I had to copy over the libs and also reinstall squid3-dev afterwards. It would not start because of:

    squid[88922]: execvp failed: (2) No such file or directory
    

    Now it's working and I am testing it.



  • @athurdent:

    It would not start because of:

    squid[88922]: execvp failed: (2) No such file or directory
    

    Now it's working and I am testing it.

    I'm confused, is it working or not ???



  • @marcelloc:

    @Legion:

    DG on pfSense:8080 with Squid as parent
    Squid3 on 127.0.0.1:3128

    Check if squid is running, on log you sent I can see only warnings.

    squid -NsXY on console can show you squid startup error or /var/squid/logs/cache.log

    Nope, wasn't running. I tried again and got:

    Noticeable points:

    • I had all my subnets on the ACL allow list (don't remember why)
    • I had pfsense.org and pfsense.com in ACL allow lists, from problems I had once accessing new package information
    • I didn't uninstall Squid3 first. But when I noticed that I still had it, I tried to uninstall it and reinstall Squid3-dev and it still didn't work
    • I get multiple dansguardian[23423]: Error connecting to proxy messages in system.log and no internet connectivity at all
    • no transparent http or https checked, Squid3-dev listening on localhost:3128 only, NAT rules to redirect 80 to DG, then DG has Squid as parent.

    Maybe Squid3-dev works best without DG underneath?



  • @Legion:

    Noticeable points:

    • I had all my subnets on the ACL allow list (don't remember why)
    • I had pfsense.org and pfsense.com in ACL allow lists, from problems I had once accessing new package information
    • I didn't uninstall Squid3 first. But when I noticed that I still had it, I tried to uninstall it and reinstall Squid3-dev and it still didn't work
    • I get multiple dansguardian[23423]: Error connecting to proxy messages in system.log and no internet connectivity at all
    • no transparent http or https checked, Squid3-dev listening on localhost:3128 only, NAT rules to redirect 80 to DG, then DG has Squid as parent.

    Maybe Squid3-dev works best without DG underneath?

    uninstall both and then reinstall squid3-dev.

    I've pushed yesterday some fixes to conf generator.

    I think it's better to test squid itself and then go to dansguardian integration.

    Leave localhost unchecked, it's automatically inserted when using transparent mode. I'll include this warning on gui to prevent some errors.



  • @marcelloc:

    @athurdent:

    It would not start because of:

    squid[88922]: execvp failed: (2) No such file or directory
    

    Now it's working and I am testing it.

    I'm confused, is it working or not ???

    Sorry for not being clearer. After copying the libs it would not start until I reinstalled it. Just wanted to let others know that maybe a reinstall is needed after putting the libs in place.
    It works fine now and I have to dig a little further into ssl filtering ;)



  • @marcelloc:

    @wheelz:

    How are the certificates set up?  I know the pfsense box should act as a certificate authority and all the clients must trust it.  So is the CA cert automatically generated and how do

    Default web configurator certificate can be used.

    I deleted my original default certificates some time ago and set up my own CA using pfSense. What kind of certificate do I need to create for SSL interception to work? I tried generating a CA certificate signed by my CA but Squid does not like it.
    I always get

    squid: No valid signing SSL certificate configured for http_port 192.168.x.4:3128
    

    also tried using a server certificate, does not work either, same error. Any hints for me?



  • @athurdent:

    also tried using a server certificate, does not work either, same error. Any hints for me?

    I'm using a server ceritificate signed by created ca.

    webconfigurator in some cases may work too.

    Check on cache.log if squid is not crashing while trying to intercept ssl.



  • I've found another missing file:

    ERROR: auth_param basic program /usr/local/libexec/squid/msnt_auth: (2) No such file or directory
    FATAL: auth_param basic program /usr/local/libexec/squid/msnt_auth: (2) No such file or directory Squid Cache (Version 3.3.4)

    (there IS a file called basic_msnt_auth)

    I get this error, if I try to activate the NT Domain authentication. By the way, there is another helper called ntlm_smb_lm_auth. Wouldn't that be the better choice for windows?



  • @Fehler20:

    I've found another missing file:

    ERROR: auth_param basic program /usr/local/libexec/squid/msnt_auth: (2) No such file or directory
    FATAL: auth_param basic program /usr/local/libexec/squid/msnt_auth: (2) No such file or directory Squid Cache (Version 3.3.4)

    (there IS a file called basic_msnt_auth)

    I get this error, if I try to activate the NT Domain authentication. By the way, there is another helper called ntlm_smb_lm_auth. Wouldn't that be the better choice for windows?

    The only tested NTLM authentication for pfsense that I am aware of is outlined in my thread:  http://forum.pfsense.org/index.php/topic,58700.0.html.  That was using an earlier version of squid though.  If you find other auth plugins that work or expand on this, I'd really like to know the details of it.  If so, please post it in that thread to put it all in one place for everyone's benefit.



  • It seems to be that there was some renaming done. The LDAP plugin is broken, too (I havn't tested RADIUS).
    @wheelz: You are correct :), I've tried ntlm_smb_lm_auth today and it does not seem to work either with IE and FF. There are some strings exchanged between the Domain Controller and squid, but SSO with NTLM does not work neither entering the creditals manually.

    Btw could you add another option to the webinterface: I gives the ability to add custom caching rules.

    /usr/local/pkg/squid.inc line 983

    //custom Options
    $conf.=sq_text_area_decode($settings['custcache']);
    $conf.='
    ';

    and

    /usr/local/pkg/squid_cache.xml line 138ff.

    <field><fielddescr>Custom Cache Options</fielddescr>
    <fieldname>custcache</fieldname>
    <description>Specify custom Cache rules here.</description>
    <type>textarea</type>
    <cols>50</cols>
    <rows>5</rows>
    <encoding>base64</encoding></field>

    Edit: I have found a new problem/mistake. The cache function only works if I manually add this custom option:

    cache allow ALL

    Maybe this has to be added to the default configuration.



  • @Fehler20:

    It seems to be that there was some renaming done. The LDAP plugin is broken, too (I havn't tested RADIUS).

    @Fehler20:

    Btw could you add another option to the webinterface: I gives the ability to add custom caching rules.

    Can't it be done on custom options?

    @Fehler20:

    Edit: I have found a new Problem/mistake. The cache funktion only works if I manually add this custom option:

    cache allow ALL

    Maybe this has to be added to the default configuration.

    I'll check it.



  • Can't it be done on custom options?

    Yes, but it is a little bit confusing, if there is an option. It causes every following configuration change to fail because the config file is corrupted.



  • @Fehler20:

    Can't it be done on custom options?

    Yes, but it is a little bit confusing, if there is an option. It causes every following configuration change to fail because the config file is corrupted.

    custom_options need one cmd per line instead of old  ";" from squid2.

    Is this the way you are doing?



  • Quote from: Fehler20 on Today at 11:42:58 am

    Quote

    Can't it be done on custom options?

    Yes, but it is a little bit confusing, if there is an option. It causes every following configuration change to fail because the config file is corrupted.

    custom_options need one cmd per line instead of old  ";" from squid2.

    Is this the way you are doing?

    Yes, my config is running!

    I just noticed that if you use an authentcation method from the authentication tab this causes the following error:

    php: /pkg_edit.php: The command '/usr/local/sbin/squid -k reconfigure -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was '2013/05/16 19:07:32| ERROR: auth_param basic program /usr/local/libexec/squid/msnt_auth: (2) No such file or directory FATAL: auth_param basic program /usr/local/libexec/squid/msnt_auth: (2) No such file or directory Squid Cache (Version 3.3.4): Terminated abnormally. CPU Usage: 0.012 seconds = 0.008 user + 0.004 sys Maximum Resident Size: 37344 KB Page faults with physical i/o: 0'



  • @Fehler20:

    I just noticed that if you use an authentcation method from the authentication tab this causes the following error:

    I'm fixing it and checking other config changes. I'll push new gui version today.



  • pkg version 2.1.1 is out.

    main changes

    • Fixed auth plugins filenames

    • Included more ssl_crt checks

    • Included custom refresh_pattern field for dynamic content on cache tab

    • Included missing cache allow all on squid.inc(this may fix no cache hits issue with dynamic content enabled.)



  • •Included custom refresh_pattern field for dynamic content on cache tab

    Little Problem here: you have to insert a new line after the custom caching options. If not the configuration becomes corrupted.
    Besides this caching (of dynamic content) works for me now. Thank you!



  • @Fehler20:

    Little Problem here: you have to insert a new line after the custom caching options. If not the configuration becomes corrupted.

    Test inserting an extra <enter>on your custom options.</enter>



  • On the i-cap for AV feature… If I am already using Dan's Guardian with the ClamAV options, would there be any reason to switch to squid using i-cap when it is working?  Or is that mainly geared for people who are using squid by itself?