Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Improve Custom refresh pattern

    Scheduled Pinned Locked Moved Cache/Proxy
    111 Posts 27 Posters 50.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JonathanLeeJ
      JonathanLee @High_Voltage
      last edited by

      @high_voltage thanks for the reply,

      Yes this Netgate 2100 max firewall is running Squid over pfSense with custom refresh patterns and Squid guard. I also reset the disk cache and have this installed WPAD for on and off the wifi with smartphone I even added SSL certificates to everything. I forced all traffic into the proxy port 3128-3129 everything works Hulu, Disney plus, Amazon video. I watch XFILES alot lately over Hulu with this config. What's weird I can take that link and manually download the update over the browser, but somehow the system won't download the update. It's almost paused just gets to 0. My Raspberry Pi was doing this also, I changed the mirrors to only use https downloads under some settings and that now updates fine. Again Windows 10 pro is not using https for the anti virus signature updates right now. I am still new to this refresh options. When you force traffic over the proxy port 80 is still working for everything else I can access http over web url use. It can see Squid guard blocks and will display viruses on clam AV tests. This thing is a tank. This last configuration fix will make it work perfectly. Why does Microsoft use http for updates? Most of the internet moved to HTTPS. It's weird right? I am about to factory default it and try again. Nat I tested with a port redirect also. I tested using just transparent. Everything works except the Windows 10 pro updates. It worked last night however Microsoft started using https for a couple hours.

      Make sure to upvote

      High_VoltageH 1 Reply Last reply Reply Quote 0
      • High_VoltageH
        High_Voltage @JonathanLee
        last edited by

        @jonathanlee ill be totally honest, part of why i recently disabled squid was for similar issues, for whay ever reason, linux updates broke recently when using squid, not sure why but this definitely seems to be an issue for some reason right now.

        JonathanLeeJ 1 Reply Last reply Reply Quote 0
        • JonathanLeeJ
          JonathanLee @High_Voltage
          last edited by

          @high_voltage I got my Raspberry Pi to work with a different mirror I edited sources to one that allowed https. When I run apt-get update it uses a different mirror now I use the constant com's mirror.

          Edit this file
          /etc/apt/sources.list

          Add a https source from the update mirrors for example in Raspberry Pi Linux I changed it to a https source.

          Screenshot_20220116-072528.png

          Screenshot_20220116-072615.png

          Check out other countries some are almost all https like Germany.

          Make sure to upvote

          JonathanLeeJ High_VoltageH 2 Replies Last reply Reply Quote 0
          • JonathanLeeJ
            JonathanLee @JonathanLee
            last edited by

            @jonathanlee it has got to work the same for refreshers for other Linux flavors also.

            Make sure to upvote

            JonathanLeeJ 1 Reply Last reply Reply Quote 0
            • JonathanLeeJ
              JonathanLee @JonathanLee
              last edited by

              @jonathanlee that was the only way to get Linux updates to work with Squid for me, it was doing the same thing as Windows updates, Squid would show a http and when you looked at Squid guard's live connections it would only show 0.

              Make sure to upvote

              JonathanLeeJ 1 Reply Last reply Reply Quote 0
              • JonathanLeeJ
                JonathanLee @JonathanLee
                last edited by

                @jonathanlee

                Made new post with this specific issue.

                https://forum.netgate.com/topic/169166/warning-possible-bypass-attempt-found-multiple-slashes-where-only-one-is-expected-http-dl-delivery-mp-microsoft-com-filestreamingservice-files/3?_=1642466910316

                Make sure to upvote

                1 Reply Last reply Reply Quote 0
                • JonathanLeeJ
                  JonathanLee @KOM
                  last edited by

                  @kom

                  Here it is, per your request, a Windows 10 update cached and delivered to another machine. Notice the HIT

                  1643495363814-hit.jpeg

                  (IMAGE: Windows dynamic refresh patterns to work recently)

                  Make sure to upvote

                  KOMK 1 Reply Last reply Reply Quote 0
                  • High_VoltageH
                    High_Voltage @JonathanLee
                    last edited by High_Voltage

                    @jonathanlee For whatever reason, it's worth noting I literally only just discovered 2 weeks ago that apparently a good chunk of my problems were due to transparent squid and clam AV, having clamAV set to scan all mode was causing random issues I cannot even begin to pinpoint. Setting it to scan Web only fixed everything, but having it set to scan all mode for whatever reason would cause apt packages To fail at trying to receive header information. Even http connections failed due to this.

                    JonathanLeeJ 1 Reply Last reply Reply Quote 1
                    • JonathanLeeJ
                      JonathanLee @High_Voltage
                      last edited by JonathanLee

                      @high_voltage I think it is the same as if you were to do a ClamAV scan on Kali Linux. So many packages and tools come up as issues when they are in fact only Pen Testing tools. In PFsense Curl, and many other items are included in packages and may scan as false positives also as they are not on a client machine however part of a firewall. It should have a scan Squid Cache option, that is what should be scanned right? Think about the number of items stored in the content accelerator that could be invasive. Why does squid not include scan local cache as an option?

                      Make sure to upvote

                      High_VoltageH 1 Reply Last reply Reply Quote 0
                      • High_VoltageH
                        High_Voltage @JonathanLee
                        last edited by

                        @jonathanlee no, i mean it broke traffic entirely.

                        JonathanLeeJ 1 Reply Last reply Reply Quote 0
                        • JonathanLeeJ
                          JonathanLee @High_Voltage
                          last edited by JonathanLee

                          @high_voltage wow that's different. I had issues where I needed to clear the cache before the traffic would flow again, almost like a container was in the cache.

                          Make sure to upvote

                          1 Reply Last reply Reply Quote 0
                          • KOMK
                            KOM @JonathanLee
                            last edited by

                            @jonathanlee Huh? last time I posted in this thread was 4 years ago.

                            JonathanLeeJ 1 Reply Last reply Reply Quote 0
                            • JonathanLeeJ
                              JonathanLee @KOM
                              last edited by

                              @kom sorry I thought you wanted to see a Windows 10 update run that was cached.

                              Make sure to upvote

                              KOMK 1 Reply Last reply Reply Quote 0
                              • KOMK
                                KOM @JonathanLee
                                last edited by

                                @jonathanlee Perhaps four years ago I did. I don't remember since it's been four years. I don't even use squid anymore. It's completely useless other than as a base for squidguard URL filtering.

                                JonathanLeeJ 1 Reply Last reply Reply Quote 0
                                • JonathanLeeJ
                                  JonathanLee @KOM
                                  last edited by JonathanLee

                                  @kom I respectfully disagree with "useless", I use it for HTTPS cache anti-virus scanning of HTTPS websites and HTTP. Dynamic caching, URL filtering, and blocking. Don't get me wrong it is rather complicated to understand, however the vast abilities that it has to customize a network environment by need is what sets it apart. It can do many things. It is just a challenge to learn. It has also protected my system from many hidden issues that Clam AV stops and reports with HTTPS alongside pup detection as well as generates clear reports. It's Mirrored Analytics down to a granular level.

                                  Make sure to upvote

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    dmalick @kivimart
                                    last edited by

                                    @kivimart is it working for squid version - 4.45

                                    A 1 Reply Last reply Reply Quote 0
                                    • A
                                      aGeekhere @dmalick
                                      last edited by

                                      @dmalick You can use the latest here https://github.com/mmd123/squid-cache-dynamic_refresh-list

                                      Yes it works with the latest squid

                                      Never Fear, A Geek is Here!

                                      D 1 Reply Last reply Reply Quote 1
                                      • D
                                        dmalick @aGeekhere
                                        last edited by

                                        @ageekhere it is working thanks

                                        JonathanLeeJ 1 Reply Last reply Reply Quote 1
                                        • JonathanLeeJ
                                          JonathanLee @dmalick
                                          last edited by

                                          @dmalick keep in mind that sometimes if changes in a website are very small it will still load old information if you use ssl intercept and have it set up to cache https. I have had a issue with a time card that would not load a new line because the watermark was to low to require a refresh, however on a different machine it would see the new time card. Just be aware that things are still a work in progress.

                                          Make sure to upvote

                                          D 2 Replies Last reply Reply Quote 1
                                          • D
                                            dmalick @JonathanLee
                                            last edited by

                                            This post is deleted!
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.