Squid Returned to Packages *** PLEASE TEST ***



  • Thanks to Fernando and databeestje, the squid package has been updated to 2.6 and returned to the package list.  It is not perfect and definitely needs testing.  This is the opportunity for everyone who has been asking about this package to test and publish your results.  Please download the latest squid package, install and report your results to this thread, be they positive or negative.

    *** PLEASE DO NOT POST FEATURE REQUESTS TO THIS THREAD ***



  • My pfSense install just bombed on me with transparent proxy enabled. Back to the drawing board!



  • Can you furnish us with usable details?



  • No go, the service won't start.



  • Here's what I figured out so far.

    • It will start when running /usr/local/etc/rc.d/squid.sh start manually.

    • It does not restart via the webconfigurator

    • It sometimes restarts by itself:
      Dec 19 11:21:48 Squid_Alarm[96105]: Squid has exited. Reconfiguring filter.
      Dec 19 11:21:48 Squid_Alarm[96107]: Attempting restart…

    • In transparent mode the firewall redirects all http to 3128, but squid listens on 80, so nothing happens. When transparent is off, squid listens on 3128

    • The throttling does not work, the maximum is stays 10MB. This is my delay_parameters from squid.conf when overall limit is 20kb and per host is 16kb
      delay_parameters 1 20480/10485760 16384/10485760. Online speedtests all return more than these limits when going through squid

    I hope this is of value.
    Regards,
    Nicki



  • Package has been removed again.  This damn thing is cursed.



  • hmm, who is Fernado and databeestje?
    I tought I was the one who updated the package.

    First, you don't have to start squid manually… I think you, if you change something it will restart squid automaticly.
    Even you may not start squid via the manual way, because some things has to be done first.

    I think the alarm was because there was no cache dir made and maybe you started it twice.

    hmm... no wait...
    I looked at the CVS page and I saw there are still the old scripts on the site and in the package list are also the old ones.

    I think you need to copy the eddited files I mailed to you first.. (see other thread)



  • I did experience that when it did start it took a few minutes for it to start. It would start right up on a reboot … that is with ntop, snort, nntpd, miniupnpd, and a few other services running as well ... all of that would run fine ... i just couldn't get it to go out to through the proxy. It would always tell me that it was restricted no matter if i had that option on the first tab checked or not, also with the CIDR network defined in the allowed section as well.



  • @Umberto:

    hmm, who is Fernado and databeestje?
    I tought I was the one who updated the package.

    First, you don't have to start squid manually… I think you, if you change something it will restart squid automaticly.
    Even you may not start squid via the manual way, because some things has to be done first.

    I think the alarm was because there was no cache dir made and maybe you started it twice.

    hmm... no wait...
    I looked at the CVS page and I saw there are still the old scripts on the site and in the package list are also the old ones.

    I think you need to copy the eddited files I mailed to you first.. (see other thread)

    The author for the CVS fields is the one who actually commits the files. Its not going to show your name as the author.

    The squid.inc file was updated 21 hours ago. Does this include the changes you made? I'm on the dev list and received a copy of squid_diffs.tar.gz. The archive is corrupt and only squid_inc.diff is accessible with winrar. On me fedora box it tells me its not in gzip format.

    I did a tar xf on your archive and it looks like it was not in gzip format. Should have only had the .tar extension.

    tar: Skipping to next header
    tar: Archive contains obsolescent base-64 headers
    tar: Read 2095 bytes from squid_diffs.tar.gz
    tar: Error exit delayed from previous errors

    Not sure what you used to create this with. Still only can get part of the squid_inc.diff file

    Out of curiosity I just installed squid on a clean pfSense image in vmware. It still doesn't work. Looks like major changes are needed to get this working. It is even causing kernel panics.

    Although maybe this is due to not having all the changed files.

    Dec 19 21:55:50 php: : SQUID is installed but not started. Not installing redirect rules.
    Dec 19 21:55:50 php: : SQUID is installed but not started. Not installing redirect rules.
    Dec 19 21:55:49 check_reload_status: reloading filter
    Dec 19 21:55:41 Squid_Alarm[38834]: Reconfiguring filter…
    Dec 19 21:55:38 squid[38778]: Squid Parent: child process 38781 started
    Dec 19 21:55:38 Squid_Alarm[38775]: Attempting restart…
    Dec 19 21:55:38 Squid_Alarm[38773]: Squid has exited. Reconfiguring filter.
    Dec 19 21:55:38 php: /pkg_mgr_install.php: / does not exist. Creating.
    Dec 19 21:55:38 check_reload_status: reloading filter
    Dec 19 21:55:35 squid[38710]: Squid Parent: child process 38712 started
    Dec 19 21:55:35 Squid_Alarm[38705]: Reconfiguring filter…
    Dec 19 21:55:33 squid[38685]: Squid Parent: child process 38688 exited due to signal 15
    Dec 19 21:55:32 squid[38685]: Squid Parent: child process 38688 started
    Dec 19 21:55:32 Squid_Alarm[38682]: Attempting restart…
    Dec 19 21:55:32 Squid_Alarm[38680]: Squid has exited. Reconfiguring filter.
    Dec 19 21:55:20 kernel: pid 38639 (squid), uid 0: exited on signal 6 (core dumped)
    Dec 19 21:55:20 php: /pkg_mgr_install.php: / does not exist. Creating.
    Dec 19 21:55:20 squid: Unable to open configuration file: /usr/local/etc/squid/squid.conf: (2) No such file or directory
    Dec 19 21:55:20 Squid_Alarm[38637]: Attempting restart…
    Dec 19 21:55:20 Squid_Alarm[38628]: Squid has exited. Reconfiguring filter.
    Dec 19 21:55:19 kernel: pid 38603 (squid), uid 0: exited on signal 6 (core dumped)
    Dec 19 21:55:19 squid: Unable to open configuration file: /usr/local/etc/squid/squid.conf: (2) No such file or directory
    Dec 19 21:54:11 php: /pkg_mgr_install.php: Beginning package installation for squid.



  • Well it looks like some one has altered the script in the wrong way (it could be that the diff file wasn't good)
    But I see some changes I made, but not all… and some code I have deleted is placed back, so that doesn't work indeed

    I'll post my versions as txt files...
    just rename them (squid_inc.txt -> squid.inc, squid_xml.txt -> squid.xml, squid_cache.txt) and place them in the /usr/local/pkg dir (just overwrite)

    Note: the copyright message is deleted as I tought that should have be done (I don't see them in the CVS aswell)

    squid_cache_xml.txt
    squid_xml.txt
    squid_inc.txt



  • @Umberto:

    Well it looks like some one has altered the script in the wrong way (it could be that the diff file wasn't good)
    But I see some changes I made, but not all… and some code I have deleted is placed back, so that doesn't work indeed

    I'll post my versions as txt files...
    just rename them (squid_inc.txt -> squid.inc, squid_xml.txt -> squid.xml, squid_cache.txt) and place them in the /usr/local/pkg dir (just overwrite)

    Note: the copyright message is deleted as I tought that should have be done (I don't see them in the CVS aswell)

    Nobody has even looked at your patches yet.  Be patient.



  • No, but I was just having a quick look at the inc file and it wasn't the one I have made.

    But I have attached the ones I have made to my previous post. (the first attachment were the wrong files, my mistake, now they are the ones I have)



  • @Umberto:

    No, but I was just having a quick look at the inc file and it wasn't the one I have made.

    But I have attached the ones I have made to my previous post. (the first attachment were the wrong files, my mistake, now they are the ones I have)

    Those updated files do not work either. I just replace all mine with those. Went to the squid settings, resaved the page and rebooted the box.

    I don't have the time to look into this now. My advice to you is to make sure you test out all the possible configuration options. It doesn't even work on my box with the default settings. The package should be fool proof.

    I made modifications on miniupnpd and built the imspector package. No matter what settings the user chooses it is impossible to break the package. I know that squid is more difficult due to the numerous config pages, but the same thought should be put into it.

    The correct and more time consuming fix is better than the short hack to get it working.

    Dec 20 00:23:32 squid[677]: Exiting due to repeated, frequent failures
    Dec 20 00:23:32 squid[677]: Squid Parent: child process 768 exited with status 1
    Dec 20 00:23:32 squid[677]: Squid Parent: child process 768 started
    Dec 20 00:23:29 squid[677]: Squid Parent: child process 748 exited with status 1
    Dec 20 00:23:29 squid[677]: Squid Parent: child process 748 started
    Dec 20 00:23:26 squid[677]: Squid Parent: child process 745 exited with status 1
    Dec 20 00:23:26 squid[677]: Squid Parent: child process 745 started
    Dec 20 00:23:23 squid[677]: Squid Parent: child process 710 exited with status 1
    Dec 20 00:23:23 squid[677]: Squid Parent: child process 710 started
    Dec 20 00:23:20 check_reload_status: check_reload_status is starting
    Dec 20 00:23:20 squid[677]: Squid Parent: child process 684 exited due to signal 6
    Dec 20 00:23:20 kernel: pid 684 (squid), uid 62: exited on signal 6
    Dec 20 00:23:20 (squid): Cannot open HTTP Port
    Dec 20 00:23:19 squid[677]: Squid Parent: child process 684 started
    Dec 20 00:23:19 check_reload_status: check_reload_status is starting
    Dec 20 00:23:19 squid[663]: Squid Parent: child process 665 started
    Dec 20 00:23:17 squid[612]: Squid Parent: child process 616 exited due to signal 15
    Dec 20 00:23:10 squid[612]: Squid Parent: child process 616 started
    Dec 20 00:23:10 php: : / does not exist. Creating.
    Dec 20 00:23:10 Squid_Alarm[607]: Attempting restart…
    Dec 20 00:23:10 Squid_Alarm[605]: Squid has exited. Reconfiguring filter.
    Dec 20 00:23:04 php: : SQUID is installed but not started. Not installing redirect rules.



  • I just looked at the the changes been made after my version… and I want to explain some things (to the author)

    I saw you replaced this :
    /*
    mwexec("/usr/local/sbin/squid -k shutdown");
    sleep (5);
    mwexec("killall -9 squid");
    sleep(1);
    mwexec_bg("/usr/local/sbin/squid -D");
    */
    with this:
    restart_service('squid');

    That doesn't work, because squid will be started with the shell script in the background.
    The shellscript sends a shutdown to squid waits 15 seconds and then kills it.
    What happens, squid stops and starts again and then get killed again after the 15 seconds delay.
    I choosed to restart squid within the script because it will get a hold on the process until squid has been restarted and after that the new rules will be made, wich is my next point.

    I saw you added this again:
    if($squid_conf['proxy_port'])
    $port = $squid_conf['proxy_port'];
    else
    $port = "3128";

    why? you only want to nat if it's in transparent mode, otherwise it's only a service listening on a port, so the nat will only be made with a transparent proxy in other words, $port will always be port 80.

    This line does say it in php code
    if (($squid_conf['transparent_proxy'] != 'on') || ($squid_conf['allow_interface'] != 'on')) {
    return;
    }

    btw, don't see this as be bitching or something like that, I think you have made a very nice script!

    I only added some extra option as requested to store the cache an logs on an other place, and in the upcomming day I want to make 1 update, so that the script deletes the old cache dir if you configure another one.
    If you guys want that aswell, I'll share it again.



  • @rsw686:

    @Umberto:

    No, but I was just having a quick look at the inc file and it wasn't the one I have made.

    But I have attached the ones I have made to my previous post. (the first attachment were the wrong files, my mistake, now they are the ones I have)

    Those updated files do not work either. I just replace all mine with those. Went to the squid settings, resaved the page and rebooted the box.

    I don't have the time to look into this now. My advice to you is to make sure you test out all the possible configuration options. It doesn't even work on my box with the default settings. The package should be fool proof.

    I made modifications on miniupnpd and built the imspector package. No matter what settings the user chooses it is impossible to break the package. I know that squid is more difficult due to the numerous config pages, but the same thought should be put into it.

    The correct and more time consuming fix is better than the short hack to get it working.

    Dec 20 00:23:32 squid[677]: Exiting due to repeated, frequent failures
    Dec 20 00:23:32 squid[677]: Squid Parent: child process 768 exited with status 1
    Dec 20 00:23:32 squid[677]: Squid Parent: child process 768 started
    Dec 20 00:23:29 squid[677]: Squid Parent: child process 748 exited with status 1
    Dec 20 00:23:29 squid[677]: Squid Parent: child process 748 started
    Dec 20 00:23:26 squid[677]: Squid Parent: child process 745 exited with status 1
    Dec 20 00:23:26 squid[677]: Squid Parent: child process 745 started
    Dec 20 00:23:23 squid[677]: Squid Parent: child process 710 exited with status 1
    Dec 20 00:23:23 squid[677]: Squid Parent: child process 710 started
    Dec 20 00:23:20 check_reload_status: check_reload_status is starting
    Dec 20 00:23:20 squid[677]: Squid Parent: child process 684 exited due to signal 6
    Dec 20 00:23:20 kernel: pid 684 (squid), uid 62: exited on signal 6
    Dec 20 00:23:20 (squid): Cannot open HTTP Port
    Dec 20 00:23:19 squid[677]: Squid Parent: child process 684 started
    Dec 20 00:23:19 check_reload_status: check_reload_status is starting
    Dec 20 00:23:19 squid[663]: Squid Parent: child process 665 started
    Dec 20 00:23:17 squid[612]: Squid Parent: child process 616 exited due to signal 15
    Dec 20 00:23:10 squid[612]: Squid Parent: child process 616 started
    Dec 20 00:23:10 php: : / does not exist. Creating.
    Dec 20 00:23:10 Squid_Alarm[607]: Attempting restart…
    Dec 20 00:23:10 Squid_Alarm[605]: Squid has exited. Reconfiguring filter.
    Dec 20 00:23:04 php: : SQUID is installed but not started. Not installing redirect rules.

    The cache or log dir isn't set, I have set a default value for it, but you need to change the scripts, go to the gui, see if the cache dir and log dir are set right (this because the you just have replaced the files and don't installed it as a fresh package) and then hit save…

    Ooh and btw, be sure you have the latest squid version 2.6.STABLE5
    It won't work with older versions



  • Well, I had too little time yesterday to properly test it. I was not even considering it done.

    Also, that restart sequence is allready gone and replaced with a squid -k reconfigure. That works so much better.
    If squid is dead for some other (unlikely) reason the monitor script will restart it.

    I removed the habit to kill and start the proxy_monitor script in a few places. It seems silly.
    Also it appears to start 2 of those on boot.

    The port thing I already adjusted after the earlier post.

    Still debugging. It's started, it's listening on port 80, it has a rdr and pass rule to port 80.
    But nothing is going through.

    2006/12/19 20:35:50| Accepting transparently proxied HTTP connections at 192.168.12.253, port 80, FD 9.
    2006/12/19 20:35:50| WCCP Disabled.

    root    933  0.0  1.1  4516  2724  ??  Is    8:35PM  0:00.01 /usr/local/sbin/squid -D
    proxy    935  0.0  3.1  8628  7260  ??  S    8:35PM  0:02.11 (squid) -D (squid)

    Setup Squid transparent proxy redirect

    rdr on ural0 proto tcp from any to !(ural0) port 80 -> (ural0) port 80

    Setup squid pass rules for transparent proxy

    pass in quick on ural0 proto tcp from any to !(ural0) port 80 flags S/SA keep state

    Slightly baffled why nothing is showing up in the logs.



  • @Umberto:

    The cache or log dir isn't set, I have set a default value for it, but you need to change the scripts, go to the gui, see if the cache dir and log dir are set right (this because the you just have replaced the files and don't installed it as a fresh package) and then hit save…

    Yes they are set /var/squid/log and /var/squid/cache. Although the script doesn't seem to make the directories. From looking at the log I'm assuming

    Dec 20 00:23:10    php: : / does not exist. Creating.

    was it trying to create the directory, but what create /? Somethings not right there.

    I just created the directories by hand and reboot the box. Still doesn't work. Says it cannot open HTTP port like it did before and then squid exits due to frequent failures.



  • Let me commit a newer squid.inc from my box into CVS



  • @ Databeesje: Indeed the -k option is better, but I didn't know if there were some problems with it in early editions, so I kept it like the way it was.
    I must say the -k shutdown option takes some time to actually stop, sometimes up till a minute, I don't know what happens if you start it when the old one is still running.

    Strange that the data is not going thru the proxy..
    I used this page http://www.ericgiguere.com/tools/http-header-viewer.html to see if my proxy is running, and on my box it worked great.. with the proxy turned on you could see the header is changing.

    @rsw686: the script does create the dirs and even doe a -z to rebuild the cache
    I seems the that the dir isn't passed thru correctly…



  • @databeestje:

    Let me commit a newer squid.inc from my box into CVS

    I like your -k reconfigure  haven't tought about that!

    I saw you left out the abbility to change dirs (I think it's step 2 if you want to add that feature to list in the first place)

    I'm going to change my version with the -k reconigure aswell…



  • I have added a shutdown time of 3 seconds a number of commits back. By default it waits 30 seconds. Which is silly.

    Since squid binds and unbinds configurations seamlessly it seems appropriate. And bringing down the squid for a config change is not common. I use the reconfigure option as much as I can at work. Because it is non disruptive.

    Furthermore I think we squid use squid check to see if the config is OK and squid is running.

    Could be something wrong on my box though. Not sure. it's a bit of a testbox.



  • Well, I can't test right now, running a new install  ::)
    The config was screwed so I resetted to factory default, and after that I wouldn't let me installe the squid package again…

    and I think I have 3 versions of squid installed on that box now, so it will be time..



  • In terms of the throttling not working as I expected, I believe that squid.inc line 544 should be changed from

    delay_parameters 1 $overall/$threshold $perhost/$threshold

    to

    delay_parameters 1 $overall/$overall $perhost/$perhost

    $threshold is 10MB, we don't even have 10MB on our link, we have 1MB, so the guys will never get throttled with the current squid.inc

    From http://www.visolve.com/squid/squid24s1/glossary.php#Classes:

    Each of these parameters is specified as restore / maximum - restore being the bytes per second restored to the bucket, and maximum being the amount of bytes that can be in the bucket at any time. It is important to remember that they are in bytes per second, not bits. To specify that a parameter is unlimited, use a -1.

    If we wish to limit any parameter in bits per second, divide this amount by 8, and use the value for both the restore and the maximum. For example, to restrict the entire proxy to 64kbps, use:

    delay_parameters 1 8000/8000



  • I don't mind changing that, but are you certain that actually works. I have zero experience with squid delay buckets.



  • Yes, I'm running it like that on a Smoothwall at the moment, and it does work.



  • @nicki
    I haven't looked at throteling yet, but I'll change it… thanks..

    I have also an update for the latest CVS
    In the restart script I would change
                   log_error("Reloading Squid for configuration sync");
                   mwexec("/usr/local/sbin/squid -k reconfigure");

    in to:

    if (!is_service_running('squid')) {
                   log_error("Starting Squid");
                   mwexec_bg("/usr/local/sbin/squid -D");
                   mwexec_bg("/usr/local/etc/rc.d/proxy_monitor.sh");
           } else {
                   log_error("Reloading Squid for configuration sync");
                   mwexec("/usr/local/sbin/squid -k reconfigure");
           }

    because when you install squid fresh, it won't start.
    So the first time you configure squid, it will start aswell...

    Btw, after my clean install it runs perfect.

    I replaced the 3 files with my own version because I like to change my cache dir...
    But I think it can be back in the list



  • Newer squid.inc with fixes.
    Cache location added to XML



  • @databeestje:

    Newer squid.inc with fixes.
    Cache location added to XML

    Good job!!!! Looks to be working fine. The service starts without errors. I can't really test it further without reconfiguring my box to use the vmware pfSense so I'll let others do that.



  • Uhm, I'm sorry to say, but the cache dir won't work.. there is something else there have to be changed with processing the conf file..

    I've added my version (I cahnged the restart with the -k reconfigure as you did, but added a service check… start when it doesn't run and reconfigure when it runs)

    And the log dir is also configurable (for if you want to process the log)

    Update: I have added the acl rules update as databeestje has made...
    But I'm not sure what you ment with that update? you still can't run squid and the gui on the same port.
    It won't allow you to do, because there is a check, but if you would force it on the same port it won't work either...

    squid_inc.txt



  • the port 80 check is broken. I'm fixing that now. The service check is in.

    The cache location allready made it in. You have log location changes too?



  • @databeestje:

    the port 80 check is broken. I'm fixing that now. The service check is in.

    The cache location allready made it in. You have log location changes too?

    I don't know what's wrong with it, but thanks for fixing it.

    Yeah the logdir works the same as the cache dir…

    But in your version in:
    function squid_resync_cache

    there is
    $cachedir = SQUID_CACHEDIR;
    that's wrong because that's still the fixed cache dir...
    it should be
    $cachedir =($settings['harddisk_cache_location'] ? $settings['harddisk_cache_location'] : '/var/squid/cache');

    same as with
    $logdir_cache = SQUID_LOGDIR . '/cache.log';
    $logdir_access = ($settings['log_enabled'] == 'on' ? SQUID_LOGDIR . '/access.log' : '/dev/null');

    that should be
                $logdir = ($settings['log_dir'] ? $settings['log_dir'] : '/var/squid/log');

    $logdir_cache = $logdir . '/cache.log';
    $logdir_access = ($settings['log_enabled'] == 'on' ? $logdir . '/access.log' : '/dev/null');

    I tought that's all, but I'm not 100% sure, otherwise check my included file in my previous post…



  • port 80 check fixed, added log dir option to xml.



  • Thanks to all those who contributed so far, my installation now at least starts up fine, and everything looks good!

    Will test the throttling later today, got to get some sleep first.



  • I have made it available for the public again, and the 1st succes is in. I'm off to bed.

    I think the log directory and cache directory creation is ok now and it should also populate the cache directory after changing.

    Authentication really needs testing.



  • well just 1 small bug left. i dunno if this is a bug or intentional but the squid package cant work in transparent mode if the webgui is on the port 80 so have to change the port of the webgui. if this is intentional i think it's better to have a better notice to ask user to change the webgui port. anyways great job on the package it's working well now



  • i seen another bug again. seems like after installing the package u need to change the default cache location to another location first then change back to original or else it wont create the location and squid will fail to start with (squid): Failed to verify one of the swap directories, Check cache.log for details. Run 'squid -z' to create swap directories if needed, or if running Squid for the first time. and the blacklist is not working. seems like sullrich is kinda right this thing is cursed



  • The throttling now behaves as expected. Thanks to all involved.



  • @ivanjong:

    well just 1 small bug left. i dunno if this is a bug or intentional but the squid package cant work in transparent mode if the webgui is on the port 80 so have to change the port of the webgui. if this is intentional i think it's better to have a better notice to ask user to change the webgui port. anyways great job on the package it's working well now

    No that's not a bug, you will lock yourself out of the webgui if you do that.
    There can't be both services on the same port.

    About creating a cache dir.
    I'll do a reinstall to be sure, but the problem is, if you had a squid version installed allready, it could give problems, because the cache dir is allready there but for another version
    I'll take a look at it, maybe we can cange the install procedure that it will remove the old default cahce dir if it's there during install.



  • nope mine is on fresh instalation of pfsense latest snapshot. it would not run unless i change the cache dir and change back and the blocklist is not working as well. anyways thanks for all the hardwork put into it. seems like the new squid breaks a lot of things. i'll try to test more features of the package for everyone



  • I have some other update…

    in the resync functions after
    log_error($disk_cache_location." does not exist.  Creating.");
    there must be mwexec("/usr/local/sbin/squid -k kill");

    otherwise it won't make the new cahce dir

    Ok the kill option is not that nice, but who cares? we start with a new cache dir... so it's not a problem to shut it down this way.
    I tried to do it nice, with the shutdown function, but that takes ages

    This will be the new code:
           if(!is_dir($disk_cache_location.'01/')) {
                   log_error($disk_cache_location." does not exist.  Creating.");
                   mwexec("/usr/local/sbin/squid -k kill");

    But I'll give a new update later on.
    I want to delete the old cache dir aswell, but that ain't as easy as I hope it would be.

    P.s This peace of code fixes the problem of having to setup the cache dir twice


Log in to reply