Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Squid Package Tuning

    Scheduled Pinned Locked Moved pfSense Packages
    29 Posts 8 Posters 17.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • _
      _igor_
      last edited by

      and this one for Apple, including developer-downloads…

      refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*.(pkg|dmg) 4320 100% 43200 reload-into-ims;

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Added that one, too. Thanks!

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • N
          Nachtfalke
          last edited by

          Hi,

          thanks for your great job.

          Perhaps somebody could help me with updates for Kaspersky for Windows Workstations Updates ?
          If I update on client, it connects to one of this sites:
          http://dnl-01.geo.kaspersky.com
          …
          http://dnl-19.geo.kaspersky.com

          And McAfee seems to connect to:
          update.nai.com

          But I don't know, which extensions will be downloaded :(

          Thanks for your help!

          1 Reply Last reply Reply Quote 0
          • _
            _igor_
            last edited by

            Best thing is looking at your squid-logs. There you have the full path of that update-files. Other way could be a manual download of the respective files via Webbrowser. The rest is copy/paste.

            1 Reply Last reply Reply Quote 0
            • D
              DigitalJer
              last edited by

              SWEET, this is one of my favourite features of pfsense + Squid.

              Nice work, thanks for sharing!

              –------------------------------------------------
              2.4.3-RELEASE (amd64)
              built on Mon Mar 26 18:02:04 CDT 2018
              FreeBSD 11.1-RELEASE-p7
              VM in ESXi 5.5
              1 x 1000baseTX (WAN)
              1 x 1000baseTX (LAN)

              1 Reply Last reply Reply Quote 0
              • S
                sam0t
                last edited by

                Came across this this informative post while searching for troubleshooting guidance with squid and Windows Updates. The problem being that the Windows Updates are not caching, otherwise the Squid is working as intended.

                As suggested in this post, I have copy/pasted the following entry to Squid > Services > Proxy server > General settings > Custom Settings -box

                refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims;

                However it does not seem to be working and when I was removing the cache folder and executing the command "squid -k" I get the following error:

                refreshAddToList: Invalid regular expression '([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp)': empty (sub)expression

                Any ideas about this?

                1 Reply Last reply Reply Quote 0
                • S
                  serangku
                  last edited by

                  same here … squid -k, report error with that pattern

                  some one must validate again with that refresh pattern ...
                  or ... simple ignore since squid still running ?

                  thanks ...

                  1 Reply Last reply Reply Quote 0
                  • N
                    Nachtfalke
                    last edited by

                    Hi,

                    I am using the following pattern and there is no error in syslog when rebooting or reloading squid:

                    refresh_pattern ([^.]+.|)avg.com/.*\.(bin) 4320 100% 43200 reload-into-ims;
refresh_pattern ([^.]+.|)spywareblaster.net/.*\.(dtb) 4320 100% 4320 reload-into-ims;
refresh_pattern ([^.]+.|)symantecliveupdate.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims;
refresh_pattern ([^.]+.|)avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims;
refresh_pattern ([^.]+.|)adobe.com/.*\.(exe|msi) 4320 100% 43200 reload-into-ims;
refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*\.(pkg|dmg) 8640 100% 86400 reload-into-ims;
refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp) 8640 100% 86400 reload-into-ims;
refresh_pattern ([^.]+.|)ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb) 8640 100% 86400 reload-into-ims;
range_offset_limit -1;
                    1 Reply Last reply Reply Quote 0
                    • _
                      _igor_
                      last edited by

                      I had this errors only with squid3, since some days. With squid 2.7 and squid3 also before never had such problems.

                      1 Reply Last reply Reply Quote 0
                      • N
                        Nachtfalke
                        last edited by

                        I am using squid 2.7

                        1 Reply Last reply Reply Quote 0
                        • S
                          serangku
                          last edited by

                          yes, on sys log gui there is no error with that pattern …
                          please try on pfsense shell with command squid -k rotate or parse

                          
[2.0-BETA5][root@server.localan.vvt]/root(11): squid -k parse
2011/02/08 09:15:39| pattern.conf line 1: refresh_pattern ([^.]+.|)avg.com/.*\.(bin) 4320 100% 43200 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)avg.com/.*\.(bin)': empty (sub)expression
2011/02/08 09:15:39| pattern.conf line 2: refresh_pattern ([^.]+.|)spywareblaster.net/.*\.(dtb) 4320 100% 4320 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)spywareblaster.net/.*\.(dtb)': empty (sub)expression
2011/02/08 09:15:39| pattern.conf line 3: refresh_pattern ([^.]+.|)symantecliveupdate.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)symantecliveupdate.com/.*\.(zip|exe)': empty (sub)expression
2011/02/08 09:15:39| pattern.conf line 4: refresh_pattern ([^.]+.|)avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)avast.com/.*\.(vpu|vpaa)': empty (sub)expression
2011/02/08 09:15:39| pattern.conf line 5: refresh_pattern ([^.]+.|)adobe.com/.*\.(exe|msi) 4320 100% 43200 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)adobe.com/.*\.(exe|msi)': empty (sub)expression
2011/02/08 09:15:39| pattern.conf line 6: refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*\.(pkg|dmg) 8640 100% 86400 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)(download|adcdownload).(apple.|)com/.*\.(pkg|dmg)': empty (sub)expression
2011/02/08 09:15:39| pattern.conf line 7: refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp) 8640 100% 86400 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp)': empty (sub)expression
2011/02/08 09:15:39| pattern.conf line 8: refresh_pattern ([^.]+.|)ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb) 8640 100% 86400 reload-into-ims
2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb)': empty (sub)expression

                          or is simple to ignore it …
                          if ... when it works it works beautifully!

                          thanks

                          1 Reply Last reply Reply Quote 0
                          • N
                            Nachtfalke
                            last edited by

                            Hi,

                            you are right. If I do squid -k rotate than there is the following output

                            2011/02/08 07:50:54| squid.conf line 74: refresh_pattern ([^.]+.|)avg.com/.*\.(b                                   in) 4320 100% 43200 reload-into-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   avg.com/.*\.(bin)': empty (sub)expression
2011/02/08 07:50:54| squid.conf line 75: refresh_pattern ([^.]+.|)spywareblaster                                   .net/.*\.(dtb) 4320 100% 4320 reload-into-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   spywareblaster.net/.*\.(dtb)': empty (sub)expression
2011/02/08 07:50:54| squid.conf line 76: refresh_pattern ([^.]+.|)symantecliveup                                   date.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   symantecliveupdate.com/.*\.(zip|exe)': empty (sub)expression
2011/02/08 07:50:54| squid.conf line 77: refresh_pattern ([^.]+.|)avast.com/.*\.                                   (vpu|vpaa) 4320 100% 43200 reload-into-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   avast.com/.*\.(vpu|vpaa)': empty (sub)expression
2011/02/08 07:50:54| squid.conf line 78: refresh_pattern ([^.]+.|)adobe.com/.*\.                                   (exe|msi) 4320 100% 43200 reload-into-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   adobe.com/.*\.(exe|msi)': empty (sub)expression
2011/02/08 07:50:54| squid.conf line 79: refresh_pattern ([^.]+.|)(download|adcd                                   ownload).(apple.|)com/.*\.(pkg|dmg) 8640 100% 86400 reload-into-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   (download|adcdownload).(apple.|)com/.*\.(pkg|dmg)': empty (sub)expression
2011/02/08 07:50:54| squid.conf line 80: refresh_pattern ([^.]+.|)(download|(win                                   dows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp) 8640 100% 86400 reload-int                                   o-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   (download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp)': empty (sub                                   )expression
2011/02/08 07:50:54| squid.conf line 81: refresh_pattern ([^.]+.|)ubuntu.com/.*\                                   .(tar|bz|bz2|gpg|gz|zip|deb) 8640 100% 86400 reload-into-ims
2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb)': empty (sub)expression

                            But it seems to work and to cache updates from windows and from adobe (this are the two I tested yesterday).

                            1 Reply Last reply Reply Quote 0
                            • S
                              sam0t
                              last edited by

                              I got bit older system at hands, Pfsense 1.2-RELEASE with the following squid packages installed:

                              squid Network 2.6.21_11
                              squidGuard Current: 1.2.0_1 pkg v.1.5 , Installed: 1.2.0_1-2

                              Maybe this is the reason for Windows Updates not caching.

                              1 Reply Last reply Reply Quote 0
                              • S
                                serangku
                                last edited by

                                @Nachtfalke:

                                Hi,

                                you are right. If I do squid -k rotate than there is the following output

                                But it seems to work and to cache updates from windows and from adobe (this are the two I tested yesterday).

                                also when checking on cachemanager …
                                that pattern won't show on report .... unless this "([^.]+.|)" deleted, it's invalid regex
                                then pattern will show on report
                                well … it seem need validate on wiki from squid experience ...

                                thanks

                                1 Reply Last reply Reply Quote 0
                                • K
                                  khan
                                  last edited by

                                  Dear serangku
                                  Look at this i m trying to solve this too

                                  http://forum.pfsense.org/index.php/topic,33158.0.html

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    serangku
                                    last edited by

                                    i'll try tomorrow …

                                    thanks for sharing

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      scooterdouglas
                                      last edited by

                                      @serangku:

                                      also when checking on cachemanager …
                                      that pattern won't show on report .... unless this "([^.]+.|)" deleted, it's invalid regex
                                      then pattern will show on report
                                      well … it seem need validate on wiki from squid experience ...

                                      I'm sorry that everyone is having difficulty, I have only tried the regex on a 2-3 machines and everything appeared to be OK.  I running Squid 2.7 with pfSense 1.2.3 and I'm not getting any errors when I run squid -k rotate.  But that is not to say the regex is "faulty".  I had hoped that more people would have tested it before Jim had updated the wiki page, I think it might be time for myself to try a fresh install of version 1.2.3 or maybe try version 2 beta to see what is wrong.

                                      Thanks.

                                      1 Reply Last reply Reply Quote 0
                                      • N
                                        Nachtfalke
                                        last edited by

                                        Hi,

                                        in the past we talked about caching Kaspersky updates:
                                        I tried this:

                                        
refresh_pattern -i .*kaspersky\.com/.*\.(.*) 1440 100% 1440 reload-into-ims override-expire override-lastmode;
refresh_pattern -i .*kaspersky-labs\.com/.*\.(.*) 1440 100% 1440 reload-into-ims override-expire override-lastmode;

                                        I think this is working just fine for the "Kaspersky Offline Updater Tool"
                                        http://support.kaspersky.com/updater?level=2

                                        Need more testing if it works with the "normal" updater function of the client software.
                                        Perhaps someone else will find some time for this.

                                        Further I found out, that if we use

                                        range_offset_limit -1

                                        squid is still downloading files even if the user has left a website or canceld some download.
                                        To avoid this, I am using this:

                                        
quick_abort_min 0 KB;
quick_abort_max 0 KB;
quick_abort_pct 100;

                                        For windows updates I am using this without any errors in syslog and with success:

                                        
refresh_pattern -i .*microsoft\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 reload-into-ims;
refresh_pattern -i .*windowsupdate\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 reload-into-ims;

                                        To avoid, that a user is getting still a "Error 404" if a website wasn't available for a short time and then ist up again because of the high time for negative dns I am using this:

                                        
negative_ttl 5 second;
negative_dns_ttl 5 second;

                                        I am using this all in squid "Custom options".
                                        It would be really nice if someone else could test and verify this or post some other usefull squid options.
                                        For me it was hard to find these parameters and so I wnat to share them and get some feedback.

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.