Squid Package Tuning



  • Hi,

    I've been playing around with Squid Caches, what to keep and what to discard, however my regular expression (regex) is rusty (almost non-existent).

    I've read the Squid Packing Tuning guid from here:  http://doc.pfsense.org/index.php/Squid_Package_Tuning and noticed the following two lines:

    refresh_pattern windowsupdate.com/..(cab|exe) 4320 100% 43200 reload-into-ims;
    refresh_pattern au.download.windowsupdate.com/.
    .(cab|exe) 4320 100% 43200 reload-into-ims;

    Could this not be reduced to one line with the following regex command:

    refresh_pattern ([^.]+.|)windowsupdate.com/.*.(cab|exe) 4320 100% 43200 reload-into-ims;

    And while we're at it, how about changing:

    (cab|exe)

    to the pick up Office Updates as well:

    (cab|exe|msi|msp)

    Just a thought (or a brain fart ;D)!

    Scott.


  • Rebel Alliance Developer Netgate

    If you've tested that and it's known to work, I'll update the docs. There's no reason additional regexes couldn't be used to simplify it, afaik.



  • Jim,

    Thanks, I've done some limited testing, I only have a couple of PCs with Windows to test with!  I've modified the expression to this:

    refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims;

    Checking my Squid access.log, I could see plenty of hits (both TCP_MEM_HIT/200 and TCP_REFRESH_HIT/200), but will need to wait till "Patch Tuesday" (14th December) to check further.

    Scott.


  • Rebel Alliance Developer Netgate

    Sounds good, update the thread then and let us know how it went. I'll update the docs if it worked.



  • Hi,

    it is not a question pointing exactly to the topic, but could you tell what *.MSP files are ?



  • Nachtfalke,

    I would say your question is relevant to the topic.

    Files with the .MSP extension are Windows Installer Patch files, they are mainly used for updating the Microsoft Office suites.

    Scott.



  • Hi,

    thanks for your reply. I'm using pfsense2 and squid at the moment only in a testing environment and I used the "old" regex patters with cab and exe, later added msi because of your post above and it works fine.
    I will try it with the MSP extension, too, but I cannot remeber, that ever downloaded MSP files…I just downloaded service packs for my office suites as .EXE.

    Nevertheless, thanks for your explanation

    PS: Isn't there a tool, where I could transform a "normal" expression like
    http://.microsoft.com/ to a regexp ?

    Thanks



  • All,

    It looks as if the regex is working.

    This is my main XP machine, which has Office 2003 on it, as you can see lots of misses as this is the first box to use Windows update:

    
    1292365669.857    101 192.168.3.199 TCP_MISS/200           405 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2423089-x86-enu_94cce8eb153f98affd81951f2123b74f14a59925.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365671.711   1850 192.168.3.199 TCP_MISS/200        515349 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2423089-x86-enu_94cce8eb153f98affd81951f2123b74f14a59925.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365672.277     35 192.168.3.199 TCP_MISS/200           406 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-express-enu_277a9af7b64b8d5776c3b4398c4e1ce88b7ba0dd.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365673.237    955 192.168.3.199 TCP_MISS/200        262026 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-express-enu_277a9af7b64b8d5776c3b4398c4e1ce88b7ba0dd.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365682.846     39 192.168.3.199 TCP_MISS/200           407 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-enu_9fc5b54a23314ea57d995f4536edb09a0751db82.psf - DEFAULT_PARENT/havp application/octet-stream
    1292365683.663    775 192.168.3.199 TCP_MISS/206        194230 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-enu_9fc5b54a23314ea57d995f4536edb09a0751db82.psf - DEFAULT_PARENT/havp multipart/byteranges
    1292365685.395    110 192.168.3.199 TCP_MISS/200           404 HEAD http://download.windowsupdate.com/msdownload/update/software/uprl/2010/11/windowsxp-kb2443685-x86-enu_d5ce3060d0317ae2ba8b37f43377004ba3cb5a2b.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365687.238   1838 192.168.3.199 TCP_MISS/200        513812 GET  http://download.windowsupdate.com/msdownload/update/software/uprl/2010/11/windowsxp-kb2443685-x86-enu_d5ce3060d0317ae2ba8b37f43377004ba3cb5a2b.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365687.767     48 192.168.3.199 TCP_MISS/200           405 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2440591-x86-enu_4ecb42986b6e12ccc9e83c8bdda536f94f6d8488.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365689.619   1847 192.168.3.199 TCP_MISS/200        515861 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2440591-x86-enu_4ecb42986b6e12ccc9e83c8bdda536f94f6d8488.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365690.097     35 192.168.3.199 TCP_MISS/200           405 HEAD http://download.windowsupdate.com/msdownload/update/software/crup/2010/11/windowsxp-kb2467659-x86-enu_921027bf773a08190462844c45ad682a6d0d03d7.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365691.882   1779 192.168.3.199 TCP_MISS/200        495893 GET  http://download.windowsupdate.com/msdownload/update/software/crup/2010/11/windowsxp-kb2467659-x86-enu_921027bf773a08190462844c45ad682a6d0d03d7.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365692.368     37 192.168.3.199 TCP_MISS/200           407 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/ie8-windowsxp-kb2416400-x86-enu_84550c3ba8570c64937a5f154745abf7a80c533d.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365732.463  40090 192.168.3.199 TCP_MISS/200      10479895 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/ie8-windowsxp-kb2416400-x86-enu_84550c3ba8570c64937a5f154745abf7a80c533d.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365736.932     34 192.168.3.199 TCP_MISS/200           407 HEAD http://download.windowsupdate.com/msdownload/update/software/uprl/2010/12/windows-kb890830-v3.14-delta_e74875038e701d2e1388e20fb8bca1d43a8f9563.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365753.236  16259 192.168.3.199 TCP_MISS/200       4579167 GET  http://download.windowsupdate.com/msdownload/update/software/uprl/2010/12/windows-kb890830-v3.14-delta_e74875038e701d2e1388e20fb8bca1d43a8f9563.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365753.506     39 192.168.3.199 TCP_MISS/200           406 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-express-enu_31c4c9074a8b5e4ec33ce33f93af3b81d245dc94.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365754.487    939 192.168.3.199 TCP_MISS/200        244426 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-express-enu_31c4c9074a8b5e4ec33ce33f93af3b81d245dc94.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365755.948     38 192.168.3.199 TCP_MISS/200           406 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-enu_9868146acbe92b31aecec3be7732ca5b2dc8ae77.psf - DEFAULT_PARENT/havp application/octet-stream
    1292365756.317    364 192.168.3.199 TCP_MISS/206         72986 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-enu_9868146acbe92b31aecec3be7732ca5b2dc8ae77.psf - DEFAULT_PARENT/havp multipart/byteranges
    1292365804.754     36 192.168.3.199 TCP_MISS/200           406 HEAD http://download.windowsupdate.com/msdownload/update/software/defu/2010/12/am_delta_patch1_fd1798cd440388fc04aacd9d63d23f7298d6a56d.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365806.411   1616 192.168.3.199 TCP_MISS/200        457510 GET  http://download.windowsupdate.com/msdownload/update/software/defu/2010/12/am_delta_patch1_fd1798cd440388fc04aacd9d63d23f7298d6a56d.exe - DEFAULT_PARENT/havp application/octet-stream
    1292365757.800    102 192.168.3.199 TCP_MISS/200           405 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/12/gpfilt_f86338f746ba4aa86b88a6823f4444caae326e46.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365760.378   2572 192.168.3.199 TCP_MISS/200        729535 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/12/gpfilt_f86338f746ba4aa86b88a6823f4444caae326e46.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365760.502     35 192.168.3.199 TCP_MISS/200           406 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/12/mspub_09d2531c6cd96a26d2a83c1a0ff8c9bc91aeeef2.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365772.090  11547 192.168.3.199 TCP_MISS/200       2912623 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/12/mspub_09d2531c6cd96a26d2a83c1a0ff8c9bc91aeeef2.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365772.270     36 192.168.3.199 TCP_MISS/200           405 HEAD http://download.windowsupdate.com/msdownload/update/software/crup/2010/12/outlook_49ecfe899e24ee3a825a61a1d72b6b6ba081bfc2.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365790.045  17770 192.168.3.199 TCP_MISS/200       5084434 GET  http://download.windowsupdate.com/msdownload/update/software/crup/2010/12/outlook_49ecfe899e24ee3a825a61a1d72b6b6ba081bfc2.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365790.216     35 192.168.3.199 TCP_MISS/200           406 HEAD http://download.windowsupdate.com/msdownload/update/software/crup/2010/12/outlfltr_e353aad09af692bcd477d82a7e1dedf146333e12.cab - DEFAULT_PARENT/havp application/octet-stream
    1292365804.603  14380 192.168.3.199 TCP_MISS/200       3551773 GET  http://download.windowsupdate.com/msdownload/update/software/crup/2010/12/outlfltr_e353aad09af692bcd477d82a7e1dedf146333e12.cab - DEFAULT_PARENT/havp application/octet-stream
    
    

    Now this is my (old) laptop:

    
    1292365986.419     11 192.168.3.197 TCP_HIT/200            413 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2423089-x86-enu_94cce8eb153f98affd81951f2123b74f14a59925.exe - NONE/- application/octet-stream
    1292365987.846   1350 192.168.3.197 TCP_HIT/200         515357 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2423089-x86-enu_94cce8eb153f98affd81951f2123b74f14a59925.exe - NONE/- application/octet-stream
    1292365991.217     11 192.168.3.197 TCP_HIT/200            414 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-express-enu_277a9af7b64b8d5776c3b4398c4e1ce88b7ba0dd.cab - NONE/- application/octet-stream
    1292365991.920    676 192.168.3.197 TCP_HIT/200         262034 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-express-enu_277a9af7b64b8d5776c3b4398c4e1ce88b7ba0dd.cab - NONE/- application/octet-stream
    1292366002.587      0 192.168.3.197 TCP_MEM_HIT/200        415 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-enu_9fc5b54a23314ea57d995f4536edb09a0751db82.psf - NONE/- application/octet-stream
    1292366003.398    748 192.168.3.197 TCP_MISS/206        194230 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-enu_9fc5b54a23314ea57d995f4536edb09a0751db82.psf - DEFAULT_PARENT/havp multipart/byteranges
    1292366009.603     11 192.168.3.197 TCP_HIT/200            412 HEAD http://download.windowsupdate.com/msdownload/update/software/uprl/2010/11/windowsxp-kb2443685-x86-enu_d5ce3060d0317ae2ba8b37f43377004ba3cb5a2b.exe - NONE/- application/octet-stream
    1292366010.985   1339 192.168.3.197 TCP_HIT/200         513820 GET  http://download.windowsupdate.com/msdownload/update/software/uprl/2010/11/windowsxp-kb2443685-x86-enu_d5ce3060d0317ae2ba8b37f43377004ba3cb5a2b.exe - NONE/- application/octet-stream
    1292366014.465     10 192.168.3.197 TCP_HIT/200            413 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2440591-x86-enu_4ecb42986b6e12ccc9e83c8bdda536f94f6d8488.exe - NONE/- application/octet-stream
    1292366015.631   1115 192.168.3.197 TCP_HIT/200         515869 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2440591-x86-enu_4ecb42986b6e12ccc9e83c8bdda536f94f6d8488.exe - NONE/- application/octet-stream
    1292366019.214     10 192.168.3.197 TCP_HIT/200            413 HEAD http://download.windowsupdate.com/msdownload/update/software/crup/2010/11/windowsxp-kb2467659-x86-enu_921027bf773a08190462844c45ad682a6d0d03d7.exe - NONE/- application/octet-stream
    1292366020.592   1292 192.168.3.197 TCP_HIT/200         495901 GET  http://download.windowsupdate.com/msdownload/update/software/crup/2010/11/windowsxp-kb2467659-x86-enu_921027bf773a08190462844c45ad682a6d0d03d7.exe - NONE/- application/octet-stream
    1292366023.936     11 192.168.3.197 TCP_HIT/200            415 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/ie8-windowsxp-kb2416400-x86-enu_84550c3ba8570c64937a5f154745abf7a80c533d.exe - NONE/- application/octet-stream
    1292366042.305  18320 192.168.3.197 TCP_HIT/200       10479903 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/ie8-windowsxp-kb2416400-x86-enu_84550c3ba8570c64937a5f154745abf7a80c533d.exe - NONE/- application/octet-stream
    1292366055.405     11 192.168.3.197 TCP_HIT/200            415 HEAD http://download.windowsupdate.com/msdownload/update/software/uprl/2010/12/windows-kb890830-v3.14-delta_e74875038e701d2e1388e20fb8bca1d43a8f9563.exe - NONE/- application/octet-stream
    1292366065.523   9985 192.168.3.197 TCP_HIT/200        4579175 GET  http://download.windowsupdate.com/msdownload/update/software/uprl/2010/12/windows-kb890830-v3.14-delta_e74875038e701d2e1388e20fb8bca1d43a8f9563.exe - NONE/- application/octet-stream
    1292366067.905     11 192.168.3.197 TCP_HIT/200            414 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-express-enu_31c4c9074a8b5e4ec33ce33f93af3b81d245dc94.cab - NONE/- application/octet-stream
    1292366068.572    610 192.168.3.197 TCP_HIT/200         244434 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-express-enu_31c4c9074a8b5e4ec33ce33f93af3b81d245dc94.cab - NONE/- application/octet-stream
    1292366073.061      0 192.168.3.197 TCP_MEM_HIT/200        414 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-enu_9868146acbe92b31aecec3be7732ca5b2dc8ae77.psf - NONE/- application/octet-stream
    1292366073.430    342 192.168.3.197 TCP_MISS/206         72986 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-enu_9868146acbe92b31aecec3be7732ca5b2dc8ae77.psf - DEFAULT_PARENT/havp multipart/byteranges
    1292366078.192     98 192.168.3.197 TCP_MISS/200           407 HEAD http://download.windowsupdate.com/msdownload/update/software/defu/2010/12/am_delta_ecfd87fd2d14882a4ce6d5822c2da25b171c50ea.exe - DEFAULT_PARENT/havp application/octet-stream
    1292366091.758  13537 192.168.3.197 TCP_MISS/200       3988775 GET  http://download.windowsupdate.com/msdownload/update/software/defu/2010/12/am_delta_ecfd87fd2d14882a4ce6d5822c2da25b171c50ea.exe - DEFAULT_PARENT/havp application/octet-stream
    
    

    And finally my (very old) XP box that I keep updated just in case:

    
    1292366376.917     11 192.168.3.196 TCP_HIT/200            413 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2423089-x86-enu_94cce8eb153f98affd81951f2123b74f14a59925.exe - NONE/- application/octet-stream
    1292366378.236   1302 192.168.3.196 TCP_HIT/200         515357 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2423089-x86-enu_94cce8eb153f98affd81951f2123b74f14a59925.exe - NONE/- application/octet-stream
    1292366381.193     11 192.168.3.196 TCP_HIT/200            414 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-express-enu_277a9af7b64b8d5776c3b4398c4e1ce88b7ba0dd.cab - NONE/- application/octet-stream
    1292366381.679    477 192.168.3.196 TCP_HIT/200         262034 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-express-enu_277a9af7b64b8d5776c3b4398c4e1ce88b7ba0dd.cab - NONE/- application/octet-stream
    1292366397.496      0 192.168.3.196 TCP_MEM_HIT/200        415 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-enu_9fc5b54a23314ea57d995f4536edb09a0751db82.psf - NONE/- application/octet-stream
    1292366398.291    781 192.168.3.196 TCP_MISS/206        194230 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2436673-x86-enu_9fc5b54a23314ea57d995f4536edb09a0751db82.psf - DEFAULT_PARENT/havp multipart/byteranges
    1292366403.734     11 192.168.3.196 TCP_HIT/200            412 HEAD http://download.windowsupdate.com/msdownload/update/software/uprl/2010/11/windowsxp-kb2443685-x86-enu_d5ce3060d0317ae2ba8b37f43377004ba3cb5a2b.exe - NONE/- application/octet-stream
    1292366404.598    852 192.168.3.196 TCP_HIT/200         513820 GET  http://download.windowsupdate.com/msdownload/update/software/uprl/2010/11/windowsxp-kb2443685-x86-enu_d5ce3060d0317ae2ba8b37f43377004ba3cb5a2b.exe - NONE/- application/octet-stream
    1292366407.613     11 192.168.3.196 TCP_HIT/200            413 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2440591-x86-enu_4ecb42986b6e12ccc9e83c8bdda536f94f6d8488.exe - NONE/- application/octet-stream
    1292366408.577    949 192.168.3.196 TCP_HIT/200         515869 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2440591-x86-enu_4ecb42986b6e12ccc9e83c8bdda536f94f6d8488.exe - NONE/- application/octet-stream
    1292366411.623     11 192.168.3.196 TCP_HIT/200            413 HEAD http://download.windowsupdate.com/msdownload/update/software/crup/2010/11/windowsxp-kb2467659-x86-enu_921027bf773a08190462844c45ad682a6d0d03d7.exe - NONE/- application/octet-stream
    1292366412.904   1261 192.168.3.196 TCP_HIT/200         495901 GET  http://download.windowsupdate.com/msdownload/update/software/crup/2010/11/windowsxp-kb2467659-x86-enu_921027bf773a08190462844c45ad682a6d0d03d7.exe - NONE/- application/octet-stream
    1292366373.177    129 192.168.3.196 TCP_REFRESH_MISS/200   405 HEAD http://download.windowsupdate.com/msdownload/update/software/updt/2010/11/ie8-windowsxp-kb2447568-x86-enu_6db7985fc12f63020fc905ffa507d7366fe182c4.exe - DEFAULT_PARENT/havp application/octet-stream
    1292366374.075    799 192.168.3.196 TCP_REFRESH_HIT/200 498964 GET  http://download.windowsupdate.com/msdownload/update/software/updt/2010/11/ie8-windowsxp-kb2447568-x86-enu_6db7985fc12f63020fc905ffa507d7366fe182c4.exe - DEFAULT_PARENT/havp application/octet-stream
    1292366415.862     10 192.168.3.196 TCP_HIT/200            415 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/ie8-windowsxp-kb2416400-x86-enu_84550c3ba8570c64937a5f154745abf7a80c533d.exe - NONE/- application/octet-stream
    1292366436.737  20856 192.168.3.196 TCP_HIT/200       10479903 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/ie8-windowsxp-kb2416400-x86-enu_84550c3ba8570c64937a5f154745abf7a80c533d.exe - NONE/- application/octet-stream
    1292366446.862     11 192.168.3.196 TCP_HIT/200            415 HEAD http://download.windowsupdate.com/msdownload/update/software/uprl/2010/12/windows-kb890830-v3.14-delta_e74875038e701d2e1388e20fb8bca1d43a8f9563.exe - NONE/- application/octet-stream
    1292366455.849   8967 192.168.3.196 TCP_HIT/200        4579175 GET  http://download.windowsupdate.com/msdownload/update/software/uprl/2010/12/windows-kb890830-v3.14-delta_e74875038e701d2e1388e20fb8bca1d43a8f9563.exe - NONE/- application/octet-stream
    1292366457.686     11 192.168.3.196 TCP_HIT/200            414 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-express-enu_31c4c9074a8b5e4ec33ce33f93af3b81d245dc94.cab - NONE/- application/octet-stream
    1292366458.285    588 192.168.3.196 TCP_HIT/200         244434 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-express-enu_31c4c9074a8b5e4ec33ce33f93af3b81d245dc94.cab - NONE/- application/octet-stream
    1292366463.064      0 192.168.3.196 TCP_MEM_HIT/200        414 HEAD http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-enu_9868146acbe92b31aecec3be7732ca5b2dc8ae77.psf - NONE/- application/octet-stream
    1292366463.449    370 192.168.3.196 TCP_MISS/206         72986 GET  http://download.windowsupdate.com/msdownload/update/software/secu/2010/11/windowsxp-kb2296199-x86-enu_9868146acbe92b31aecec3be7732ca5b2dc8ae77.psf - DEFAULT_PARENT/havp multipart/byteranges
    1292366467.666     11 192.168.3.196 TCP_HIT/200            415 HEAD http://download.windowsupdate.com/msdownload/update/software/defu/2010/12/am_delta_ecfd87fd2d14882a4ce6d5822c2da25b171c50ea.exe - NONE/- application/octet-stream
    1292366475.638   7963 192.168.3.196 TCP_HIT/200        3988783 GET  http://download.windowsupdate.com/msdownload/update/software/defu/2010/12/am_delta_ecfd87fd2d14882a4ce6d5822c2da25b171c50ea.exe - NONE/- application/octet-stream
    
    

    Just need some more people to test it!

    ;)


  • Rebel Alliance Developer Netgate

    Sounds good, I went ahead and updated the doc wiki article.



  • Jim,

    One last thing, near the bottom of the Wiki page, there is an update for AVG, here are a couple more updates that could be applied:

    refresh_pattern ([^.]+.|)avg.com/..(bin) 4320 100% 43200 reload-into-ims;
    refresh_pattern ([^.]+.|)spywareblaster.net/.
    .(dtb) 4320 100% 64800 reload-into-ims;
    refresh_pattern ([^.]+.|)symantecliveupdate.com/..(zip|exe) 43200 100% 43200 reload-into-ims
    refresh_pattern ([^.]+.|)avast.com/.
    .(vpu|vpaa) 4320 100% 43200 reload-into-ims

    I put the SpywareBlaster update to 45 days as it is not a regular update as the others

    Scott.


  • Rebel Alliance Developer Netgate

    I added those to the page, thanks!



  • and this one for Apple, including developer-downloads…

    refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*.(pkg|dmg) 4320 100% 43200 reload-into-ims;


  • Rebel Alliance Developer Netgate

    Added that one, too. Thanks!



  • Hi,

    thanks for your great job.

    Perhaps somebody could help me with updates for Kaspersky for Windows Workstations Updates ?
    If I update on client, it connects to one of this sites:
    http://dnl-01.geo.kaspersky.com

    http://dnl-19.geo.kaspersky.com

    And McAfee seems to connect to:
    update.nai.com

    But I don't know, which extensions will be downloaded :(

    Thanks for your help!



  • Best thing is looking at your squid-logs. There you have the full path of that update-files. Other way could be a manual download of the respective files via Webbrowser. The rest is copy/paste.



  • SWEET, this is one of my favourite features of pfsense + Squid.

    Nice work, thanks for sharing!



  • Came across this this informative post while searching for troubleshooting guidance with squid and Windows Updates. The problem being that the Windows Updates are not caching, otherwise the Squid is working as intended.

    As suggested in this post, I have copy/pasted the following entry to Squid > Services > Proxy server > General settings > Custom Settings -box

    refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims;

    However it does not seem to be working and when I was removing the cache folder and executing the command "squid -k" I get the following error:

    refreshAddToList: Invalid regular expression '([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*.(cab|exe|msi|msp)': empty (sub)expression

    Any ideas about this?



  • same here … squid -k, report error with that pattern

    some one must validate again with that refresh pattern ...
    or ... simple ignore since squid still running ?

    thanks ...



  • Hi,

    I am using the following pattern and there is no error in syslog when rebooting or reloading squid:

    refresh_pattern ([^.]+.|)avg.com/.*\.(bin) 4320 100% 43200 reload-into-ims;
    refresh_pattern ([^.]+.|)spywareblaster.net/.*\.(dtb) 4320 100% 4320 reload-into-ims;
    refresh_pattern ([^.]+.|)symantecliveupdate.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims;
    refresh_pattern ([^.]+.|)avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims;
    refresh_pattern ([^.]+.|)adobe.com/.*\.(exe|msi) 4320 100% 43200 reload-into-ims;
    refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*\.(pkg|dmg) 8640 100% 86400 reload-into-ims;
    refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp) 8640 100% 86400 reload-into-ims;
    refresh_pattern ([^.]+.|)ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb) 8640 100% 86400 reload-into-ims;
    range_offset_limit -1;
    


  • I had this errors only with squid3, since some days. With squid 2.7 and squid3 also before never had such problems.



  • I am using squid 2.7



  • yes, on sys log gui there is no error with that pattern …
    please try on pfsense shell with command squid -k rotate or parse

    
    [2.0-BETA5][root@server.localan.vvt]/root(11): squid -k parse
    2011/02/08 09:15:39| pattern.conf line 1: refresh_pattern ([^.]+.|)avg.com/.*\.(bin) 4320 100% 43200 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)avg.com/.*\.(bin)': empty (sub)expression
    2011/02/08 09:15:39| pattern.conf line 2: refresh_pattern ([^.]+.|)spywareblaster.net/.*\.(dtb) 4320 100% 4320 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)spywareblaster.net/.*\.(dtb)': empty (sub)expression
    2011/02/08 09:15:39| pattern.conf line 3: refresh_pattern ([^.]+.|)symantecliveupdate.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)symantecliveupdate.com/.*\.(zip|exe)': empty (sub)expression
    2011/02/08 09:15:39| pattern.conf line 4: refresh_pattern ([^.]+.|)avast.com/.*\.(vpu|vpaa) 4320 100% 43200 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)avast.com/.*\.(vpu|vpaa)': empty (sub)expression
    2011/02/08 09:15:39| pattern.conf line 5: refresh_pattern ([^.]+.|)adobe.com/.*\.(exe|msi) 4320 100% 43200 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)adobe.com/.*\.(exe|msi)': empty (sub)expression
    2011/02/08 09:15:39| pattern.conf line 6: refresh_pattern ([^.]+.|)(download|adcdownload).(apple.|)com/.*\.(pkg|dmg) 8640 100% 86400 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)(download|adcdownload).(apple.|)com/.*\.(pkg|dmg)': empty (sub)expression
    2011/02/08 09:15:39| pattern.conf line 7: refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp) 8640 100% 86400 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp)': empty (sub)expression
    2011/02/08 09:15:39| pattern.conf line 8: refresh_pattern ([^.]+.|)ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb) 8640 100% 86400 reload-into-ims
    2011/02/08 09:15:39| parse_refreshpattern: Invalid regular expression '([^.]+.|)ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb)': empty (sub)expression
    
    

    or is simple to ignore it …
    if ... when it works it works beautifully!

    thanks



  • Hi,

    you are right. If I do squid -k rotate than there is the following output

    2011/02/08 07:50:54| squid.conf line 74: refresh_pattern ([^.]+.|)avg.com/.*\.(b                                   in) 4320 100% 43200 reload-into-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   avg.com/.*\.(bin)': empty (sub)expression
    2011/02/08 07:50:54| squid.conf line 75: refresh_pattern ([^.]+.|)spywareblaster                                   .net/.*\.(dtb) 4320 100% 4320 reload-into-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   spywareblaster.net/.*\.(dtb)': empty (sub)expression
    2011/02/08 07:50:54| squid.conf line 76: refresh_pattern ([^.]+.|)symantecliveup                                   date.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   symantecliveupdate.com/.*\.(zip|exe)': empty (sub)expression
    2011/02/08 07:50:54| squid.conf line 77: refresh_pattern ([^.]+.|)avast.com/.*\.                                   (vpu|vpaa) 4320 100% 43200 reload-into-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   avast.com/.*\.(vpu|vpaa)': empty (sub)expression
    2011/02/08 07:50:54| squid.conf line 78: refresh_pattern ([^.]+.|)adobe.com/.*\.                                   (exe|msi) 4320 100% 43200 reload-into-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   adobe.com/.*\.(exe|msi)': empty (sub)expression
    2011/02/08 07:50:54| squid.conf line 79: refresh_pattern ([^.]+.|)(download|adcd                                   ownload).(apple.|)com/.*\.(pkg|dmg) 8640 100% 86400 reload-into-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   (download|adcdownload).(apple.|)com/.*\.(pkg|dmg)': empty (sub)expression
    2011/02/08 07:50:54| squid.conf line 80: refresh_pattern ([^.]+.|)(download|(win                                   dows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp) 8640 100% 86400 reload-int                                   o-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   (download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp)': empty (sub                                   )expression
    2011/02/08 07:50:54| squid.conf line 81: refresh_pattern ([^.]+.|)ubuntu.com/.*\                                   .(tar|bz|bz2|gpg|gz|zip|deb) 8640 100% 86400 reload-into-ims
    2011/02/08 07:50:54| parse_refreshpattern: Invalid regular expression '([^.]+.|)                                   ubuntu.com/.*\.(tar|bz|bz2|gpg|gz|zip|deb)': empty (sub)expression
    
    

    But it seems to work and to cache updates from windows and from adobe (this are the two I tested yesterday).



  • I got bit older system at hands, Pfsense 1.2-RELEASE with the following squid packages installed:

    squid Network 2.6.21_11
    squidGuard Current: 1.2.0_1 pkg v.1.5 , Installed: 1.2.0_1-2

    Maybe this is the reason for Windows Updates not caching.



  • @Nachtfalke:

    Hi,

    you are right. If I do squid -k rotate than there is the following output

    But it seems to work and to cache updates from windows and from adobe (this are the two I tested yesterday).

    also when checking on cachemanager …
    that pattern won't show on report .... unless this "([^.]+.|)" deleted, it's invalid regex
    then pattern will show on report
    well … it seem need validate on wiki from squid experience ...

    thanks



  • Dear serangku
    Look at this i m trying to solve this too

    http://forum.pfsense.org/index.php/topic,33158.0.html



  • i'll try tomorrow …

    thanks for sharing



  • @serangku:

    also when checking on cachemanager …
    that pattern won't show on report .... unless this "([^.]+.|)" deleted, it's invalid regex
    then pattern will show on report
    well … it seem need validate on wiki from squid experience ...

    I'm sorry that everyone is having difficulty, I have only tried the regex on a 2-3 machines and everything appeared to be OK.  I running Squid 2.7 with pfSense 1.2.3 and I'm not getting any errors when I run squid -k rotate.  But that is not to say the regex is "faulty".  I had hoped that more people would have tested it before Jim had updated the wiki page, I think it might be time for myself to try a fresh install of version 1.2.3 or maybe try version 2 beta to see what is wrong.

    Thanks.



  • Hi,

    in the past we talked about caching Kaspersky updates:
    I tried this:

    
    refresh_pattern -i .*kaspersky\.com/.*\.(.*) 1440 100% 1440 reload-into-ims override-expire override-lastmode;
    refresh_pattern -i .*kaspersky-labs\.com/.*\.(.*) 1440 100% 1440 reload-into-ims override-expire override-lastmode;
    
    

    I think this is working just fine for the "Kaspersky Offline Updater Tool"
    http://support.kaspersky.com/updater?level=2

    Need more testing if it works with the "normal" updater function of the client software.
    Perhaps someone else will find some time for this.

    Further I found out, that if we use

    range_offset_limit -1
    

    squid is still downloading files even if the user has left a website or canceld some download.
    To avoid this, I am using this:

    
    quick_abort_min 0 KB;
    quick_abort_max 0 KB;
    quick_abort_pct 100;
    
    

    For windows updates I am using this without any errors in syslog and with success:

    
    refresh_pattern -i .*microsoft\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 reload-into-ims;
    refresh_pattern -i .*windowsupdate\.com/.*\.(cab|exe|msi|msp) 129600 100% 129600 reload-into-ims;
    
    

    To avoid, that a user is getting still a "Error 404" if a website wasn't available for a short time and then ist up again because of the high time for negative dns I am using this:

    
    negative_ttl 5 second;
    negative_dns_ttl 5 second;
    
    

    I am using this all in squid "Custom options".
    It would be really nice if someone else could test and verify this or post some other usefull squid options.
    For me it was hard to find these parameters and so I wnat to share them and get some feedback.


Log in to reply