Squid, SquidGuard, Lightsquid status on 2.3


  • Rebel Alliance Developer Netgate

    I fixed up some issues in the GUI for squid, squidGuard, and Lightsquid though some yet remain.

    • In squid, the monitor page has been converted to Bootstrap. The others pages are XML and do not need conversion, but there were some rendering issues with form field text that should be resolved in current snapshots.
    • In squidGaurd, the blacklist and log page to Bootstrap and I made a few other relevant adjustments to other items to be Bootstrap-friendly. Still could use some tweaks but it's generally functional at least. As with squid, there were some rendering issues that require being on a new snapshot to solve.
    • In lightsquid, there is less progress. The change to nginx introduces a problem here because lightsquid wants to use perl CGI, which nginx does not support, so we'll need to figure out how to best solve that. I fixed some rendering issues with the sqstat page but it also seems to have a more fundamental issue in that it doesn't seem to be properly polling squid.

  • Rebel Alliance Developer Netgate

    Looks like some of the changes aren't in the latest snap yet, but it should be up soon. In the meantime an update and then gitsync should do it.


  • Administrator

    @jimp:

    I fixed up some issues in the GUI for squid, squidGuard, and Lightsquid though some yet remain.

    • In squid, the monitor page has been converted to Bootstrap. The others pages are XML and do not need conversion, but there were some rendering issues with form field text that should be resolved in current snapshots.
    • In squidGaurd, the blacklist and log page to Bootstrap and I made a few other relevant adjustments to other items to be Bootstrap-friendly. Still could use some tweaks but it's generally functional at least. As with squid, there were some rendering issues that require being on a new snapshot to solve.
    • In lightsquid, there is less progress. The change to nginx introduces a problem here because lightsquid wants to use perl CGI, which nginx does not support, so we'll need to figure out how to best solve that. I fixed some rendering issues with the sqstat page but it also seems to have a more fundamental issue in that it doesn't seem to be properly polling squid.

    It' not clear to me that we need 3 packages for squid, but then, I'm not a squid fan.

    Given that we have nginx, and Lua why not just run the BSD licensed Ledge https://github.com/pintsized/ledge


  • Rebel Alliance Developer Netgate

    Squid is the base proxy, squidGuard just does URL filtering in combination with squid (it is not itself a proxy) and lightsquid parses the logs to produce access reports.

    I'd love to see something native in nginx+lua but all I've seen so far with nginx+lua like Ledge or OpenResty is a reverse proxy (like haproxy, varnish, and the "squid-reverse" parts) but I haven't seen anything like a client forward proxy to arbitrary remote servers with access controls like squid+squidGuard does.



  • Hi!
    Tested squid and it works (looks) great now.
    Lightsquid as you mentioned is useless right now.
    Reports are not working (cgi) realtime data also not working.

    Btw, will sarg be ported? Its handy for reports.


  • Rebel Alliance Developer Netgate

    Sarg is unlikely to come back. It has been a huge source of trouble. It rarely works as expected and more often than not results in the reports filling up people's disks (either by size or by running it out of inodes with tons of tiny files). If it does come back, we'd need some assurance that the package has been improved sufficiently that it wouldn't be a source of problems.



  • SquidGuard isn't filtering, have the targets set like I normally would, have applied, have saved, have done everything, even custom blacklists are not working.

    I have netflix on a blacklist, and traffic passes right through, nothing showing anywhere that SquidGuard isn't working, aside from it… not working...  Services shows it as running.







  • Rebel Alliance Developer Netgate

    Hmm, interesting. Granted I didn't try it extensively, it did filter for me. I got the error redirect page and all.

    You might start a fresh thread for that. I was only focusing on the GUI issues. Someone else may have better insight on the backend part of the code.


  • Rebel Alliance Developer Netgate

    @C0RR0SIVE:

    SquidGuard isn't filtering, have the targets set like I normally would, have applied, have saved, have done everything, even custom blacklists are not working.

    I have netflix on a blacklist, and traffic passes right through, nothing showing anywhere that SquidGuard isn't working, aside from it… not working...  Services shows it as running.

    I think I found the problem here. Update the squidGuard package to version 1.12 when it comes out here in a few minutes and then try it again.



  • I can confirm that SquidGuard v1.12 on the current version of 2.3 is working in terms of the target lists and blocking/whitelisting.

    Only other thing I can personally see is just a slightly annoying issue while using the pfsense dark theme setting, when looking at the target rules, you get white text on an almost white bar for every other category.  Though, it seems the black one isn't the default :)

    Thanks for all the hard work jimp!


  • Rebel Alliance Developer Netgate

    I fixed some more issues in squid today and have a few notes for those who may be upgrading from 2.2.x or earlier to 2.3 and having problems:

    1. Make sure that the most current version of the squid package is loaded (>= 0.4.12)

    2. Clean up leftover PBI messes:

    find / -type l -print0 | xargs -0 ls -l | egrep '(squid|perl|pbi)'
    

    Remove any symlinks still pointing to PBI dirs, especially things like perl, lightsquid, perl5, etc.

    For example:

    lrwxr-xr-x  1 root   wheel  39 May  7  2015 /usr/bin/perl -> /usr/pbi/lightsquid-i386/local/bin/perl
    lrwxr-xr-x  1 root   wheel  45 May  7  2015 /usr/local/etc/lightsquid -> /usr/pbi/lightsquid-i386/local/etc/lightsquid
    lrwxr-xr-x  1 root   wheel  40 May  7  2015 /usr/local/lib/perl5 -> /usr/pbi/lightsquid-i386/local/lib/perl5
    lrwxr-xr-x  1 root   wheel  45 Nov  5 10:32 /usr/local/www/lightsquid -> /usr/pbi/lightsquid-i386/local/www/lightsquid
    

    3. Blow away the cache:

    mv /var/squid/cache /var/squid/cache.old
    squid -z
    rm -rf /var/squid/cache.old
    


  • Coming from a Windows background I don't understand the pbi comment and symlinks. Any chance this fix can be automated?


  • Rebel Alliance Developer Netgate

    Not in the package. It fails before it gets to a point where the package can run any code.


  • Administrator

    I've pushed a fix to make sure all symlinks pointing to /usr/pbi are removed when upgrade from 2.2 to 2.3. It'll be available on next snapshots


  • Rebel Alliance Developer Netgate

    And if you're already on 2.3, you can use the command from that commit to clean up manually:

    find / -type l -lname '/usr/pbi/*' -delete
    


  • And I was just going to ask this :)



  • @jimp:

    I fixed some more issues in squid today and have a few notes for those who may be upgrading from 2.2.x or earlier to 2.3 and having problems:

    1. Make sure that the most current version of the squid package is loaded (>= 0.4.12)

    2. Clean up leftover PBI messes:

    find / -type l -print0 | xargs -0 ls -l | egrep '(squid|perl|pbi)'
    

    Remove any symlinks still pointing to PBI dirs, especially things like perl, lightsquid, perl5, etc.

    For example:

    lrwxr-xr-x  1 root   wheel  39 May  7  2015 /usr/bin/perl -> /usr/pbi/lightsquid-i386/local/bin/perl
    lrwxr-xr-x  1 root   wheel  45 May  7  2015 /usr/local/etc/lightsquid -> /usr/pbi/lightsquid-i386/local/etc/lightsquid
    lrwxr-xr-x  1 root   wheel  40 May  7  2015 /usr/local/lib/perl5 -> /usr/pbi/lightsquid-i386/local/lib/perl5
    lrwxr-xr-x  1 root   wheel  45 Nov  5 10:32 /usr/local/www/lightsquid -> /usr/pbi/lightsquid-i386/local/www/lightsquid
    

    3. Blow away the cache:

    mv /var/squid/cache /var/squid/cache.old
    squid -z
    rm -rf /var/squid/cache.old
    
    
    /pkg_edit.php: The command '/usr/local/sbin/squid -z -f /usr/local/etc/squid/squid.conf' returned exit code '1', the output was 'FATAL: getpwnam failed to find userid for effective user 'squid' Squid Cache (Version 3.5.12): Terminated abnormally. CPU Usage: 0.018 seconds = 0.018 user + 0.000 sys Maximum Resident Size: 50000 KB Page faults with physical i/o: 0'
    

  • Rebel Alliance Developer Netgate

    That's a new one. Uninstall the package and install it again. The pkg code adds that user on install, or it's supposed to anyhow



  • It mostly worked for me. I was able to install freeradius but squid returned an error. Did you catch it in my other post?



  • @jimp:

    That's a new one. Uninstall the package and install it again. The pkg code adds that user on install, or it's supposed to anyhow

    work



  • squidguard error warning on instalation

    >>> Installing pfSense-pkg-squidGuard... 
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up-to-date.
    Updating pfSense repository catalogue...
    pfSense repository is up-to-date.
    All repositories are up-to-date.
    The following 3 package(s) will be affected (of 0 checked):
    
    New packages to be INSTALLED:
    	pfSense-pkg-squidGuard: 1.12 [pfSense]
    	squidGuard: 1.4_15 [pfSense]
    	db5: 5.3.28_3 [pfSense]
    
    The process will require 15 MiB more space.
    1 MiB to be downloaded.
    Fetching pfSense-pkg-squidGuard-1.12.txz: ..... done
    Fetching squidGuard-1.4_15.txz: .... done
    Fetching db5-5.3.28_3.txz: .......... done
    Checking integrity... done (0 conflicting)
    [1/3] Installing db5-5.3.28_3...
    [1/3] Extracting db5-5.3.28_3: .......... done
    [2/3] Installing squidGuard-1.4_15...
    [2/3] Extracting squidGuard-1.4_15: ..... done
    [3/3] Installing pfSense-pkg-squidGuard-1.12...
    [3/3] Extracting pfSense-pkg-squidGuard-1.12: .......... done
    Saving updated package information...
    done.
    Loading package configuration... done.
    Configuring package components...
    Loading package instructions...
    
    Warning: file_put_contents(/usr/local/etc/squidGuard/squidguard_conf.xml): failed to open stream: No such file or directory in /usr/local/pkg/squidguard.inc on line 1045
    
    Call Stack:
        0.0004     228704   1\. {main}() /etc/rc.packages:0
        0.1839   10561792   2\. install_package_xml() /etc/rc.packages:77
        0.4223   11047992   3\. require_once('/usr/local/pkg/squidguard.inc') /etc/inc/pkg-utils.inc:702
        0.4543   12883648   4\. convert_pfxml_to_sgxml() /usr/local/pkg/squidguard.inc:100
        0.4574   12913928   5\. file_put_contents() /usr/local/pkg/squidguard.inc:1045
    
    Custom commands...
    Executing custom_php_install_command()...done.
    Executing custom_php_resync_config_command()...done.
    Menu items... done.
    Services... done.
    Writing configuration... done.
    Please visit Services - SquidGuard Proxy Filter - Target Categories and set up at least one category there before enabling SquidGuard. See https://forum.pfsense.org/index.php?topic=94312.0 for details.Message from squidGuard-1.4_15:
    ===================================================================
     In order to activate squidGuard you have to edit squid.conf
     To the contain "url_rewrite_program /usr/local/bin/squidGuard"
     and create a configuration file for squidGuard.
    
     Sample blacklists have been installed in /usr/local/share/examples/squidGuard.
    
     A sample configuration file has beeen installed in
     /usr/local/etc/squid/squidGuard.conf.sample.
    
     You need to edit the configuration and compile the blacklist
     you choose to use with:
     squidGuard -d -C all
    
     Please bear in mind that this is just a sample configuration file
     and for any real world usage you need to download or create your
     own updated blacklists and create your own configuration file.
    
     Check documentation here:
    
     http://www.squidguard.org/Doc/
    
     To activate the changes do a /usr/local/sbin/squid -k reconfigure
    ===================================================================
    Message from pfSense-pkg-squidGuard-1.12:
    Please visit Services - SquidGuard Proxy Filter - Target Categories and set up at least one category there before enabling SquidGuard. See https://forum.pfsense.org/index.php?topic=94312.0 for details.
    >>> Cleaning up cache... done.
    Success
    

  • Rebel Alliance Developer Netgate

    @whitexp:

    squidguard error warning on instalation

    I'll push a fix for that, looks easy enough to correct.



  • error on lightsquid

    Jan 21 16:31:22	php-fpm	28398	/rc.start_packages: [lightsquid] Error: Could not load default '/usr/local/etc/lightsquid/lightsquid.cfg.dist' configuration file.
    Jan 21 16:31:22	php-fpm	28398	/rc.start_packages: [lightsquid] Error: Could not create '/usr/local/etc/lightsquid/lightsquid.cfg' configuration file.
    Jan 21 16:31:22	php-fpm	28398	/rc.start_packages: [lightsquid] Removing old cronjobs...
    

  • Rebel Alliance Developer Netgate

    Lightsquid is broken in many ways (see the earlier posts in the thread) – no hope of it working util we fix up nginx for CGI.



  • Fired up a test vm with a clean 2.3 install, squid appears to be working as long as I disable clamav and c-icap.
    I seriously lack time lately, after a (really) quick check it doesn't seem to build the .sock file and it probably misses something else.
    I think you guys are well-aware of it but I'll be happy to provide more info as soon as I can if needed.

    Thank you once again for the awesome job you keep doing.
    See ya!


  • Rebel Alliance Developer Netgate

    We've made no attempt to test or work on clamav or c-icap, just the base functions of the forward proxy currently.



  • When ever i turn on Transparent HTTP Proxy  i couldn't browse any website , but there is no problem if i use explicit  proxy  .

    Any one have the same issue ???



  • I can't use transparent or adding it to my system direct. They both fail. Looks like a few of us having the problem but no cause or solution yet to my knowledge.



  • Clean install, restore configs, problem remains.



  • @jimp:

    Lightsquid is broken in many ways (see the earlier posts in the thread) – no hope of it working util we fix up nginx for CGI.

    @jimp:

    We've made no attempt to test or work on clamav or c-icap, just the base functions of the forward proxy currently.

    What does this mean? It's fixed in 2.3 final right?



  • @seanelias:

    When ever i turn on Transparent HTTP Proxy  i couldn't browse any website , but there is no problem if i use explicit  proxy  .

    Any one have the same issue ???

    I can confirm that, the transparent proxy does not work


  • Rebel Alliance Developer Netgate

    @Valex:

    @jimp:

    Lightsquid is broken in many ways (see the earlier posts in the thread) – no hope of it working util we fix up nginx for CGI.

    @jimp:

    We've made no attempt to test or work on clamav or c-icap, just the base functions of the forward proxy currently.

    What does this mean? It's fixed in 2.3 final right?

    The package version has no relation to 2.3 "final", it could be changed before or after release. Hopefully, before. It just means it isn't working or isn't tested now. The functions I stated we didn't work on may work fine, we just haven't tested them because they're not functions of squid we intend to "officially" support at this time.

    Others are free to test and submit fixes as needed if they want, though.



  • Reading threads from older versions of pfSense, it sounds like this is a known issue, but I wanted to report that I am also having issues with the transparent proxy and getting "ERR_EMPTY_RESPONSE" on Chrome. However, my observation has been that everything works fine when the DNS response is still hanging around from the last time I visited the page (read: squid was off). To me, this might suggest that Squid is either having a hard time talking to the dns server (isn't pointed a the local machine/general DNS settings), squid isn't giving dns enough time to respond (timeout setting too short), or there's some conflicting setting between squid and dns.



  • There's a workaround for the transparent proxy issue in https://redmine.pfsense.org/issues/5869

    chgrp squid /dev/pf
    

    Proper fix to come.



  • Work around confirmed working on my end! Squidguard also functioning now that squid is responding. Thank you!!!

    Now I await the return of Lightsquid for reporting but for now the important part of content filtering has returned!



  • Also pleased to report that enabling antivirus is working too! I attempted to download the eicar.txt file and it was blocked with a virus warning as it should be!

    Only thing left I notice is on the realtime log page.

    Message
    WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.sock: No such file or directory
    Database updated (6435493 signatures) from db.us.clamav.net (IP: 194.8.197.22)
    

    Everything else looks good to my knowledge.



  • Work around did not survive a reboot. Reapplied and things work again. Just an FYI.


  • Rebel Alliance Developer Netgate

    That's expected, it's just a temp workaround.


  • Rebel Alliance Developer Netgate

    New revision of the squid package is up now for testing. If you did the chgrp work around you need to change it back to 'proxy' or reboot before upgrading the package.



  • Rebooted, updated squid package. Seems to be ok. Content filtering and virus scanning are working.


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy