Troubleshooting sites that wont load using Squid bump

[michmoor]

@JonathanLee I figured it out. It was much more simple than that. I needed to select the other interfaces for SSL inspection. Only 2 out of the 4 were selected.

So at this point the custom SSL configuration is the way to go. Im bumping some networks and splicing others. Maximum flexability here so thank you for pointing the way.

Now i am getting a LightSquid report with all the IPs. I am not at the point that i want to use a blacklist as thats too much. I would prefer a paid URL block list if i could find one.
I am using SquidGuard to customize a very specific domain i want to block for certain users/IPs so thats great. PFblocker cannot be customized in this way so this is really good.

[JonathanLee]

@michmoor what a cool puzzle right? Squid is amazing

[michmoor]

@JonathanLee This was a great puzzle thanks to you. The downside is that I cannot use Authentication while transparent mode is enabled but thats Ok.

You should teach a squid class at the university :)

Final configuration is this.

[JonathanLee]

@michmoor your URL splice list will have a sites that cannot be SSL intercepted with bumping. If they are bank websites, Facebook messenger, etc. Just think ethically. After your set up watch how many viruses Clam finds. It's scary when it's running to see how many issues there really are.

[JonathanLee]

@michmoor you can use authentication, I think I seen something about that. Look at the first Squid URL I sent in a prior post here the info is on that website. It has a custom config to do that. I have never done it. My primary goal is pure cyber security issues thus getting Clam AV to block and log issues so I can report them.

[JonathanLee]

@michmoor

https://wiki.squid-cache.org/Features/Authentication

You can do custom authentication also.

"Simply said, it’s not possible to authenticate users using proxy authentication schemes when running in interception or transparent modes. See SquidFaq/InterceptionProxy for details on why."

Yes you are correct

[michmoor]

@JonathanLee said in Troubleshooting sites that wont load using Squid bump:

your URL splice list will have a sites that cannot be SSL intercepted with bumping. If they are bank websites, Facebook messenger, etc. Just think ethically. After your set up watch how many viruses Clam finds. It's scary when it's running to see how many issues there really are.

I havent enabled ClamAV. Not sure if its worth it. When i last did enable it i got lots of false positives. So for now just saving my cpu cycles and letting the endpoint protection on that.

As far as bank websites and FB messenger is concerned there is no way to actually inspect the data once its decrypted, right? So i can see the full URL but i cant see the payload so sites like banks or facebook getting bumped i dont think its a problem. pfSense doesnt have a way to inspect the payload after its decrypted.

[JonathanLee]

@michmoor Some websites won't work unless they are splice also. I don't know if you have ever used or pen tested with "Burp Suite" you can see a lot of information when sites are not set to splice. For antivirus in the proxy it's a dream as it can inspect everything for bugs I have caught hundreds of items. Again, if your not using ClamAV I don't see the reason. These are just some of the sites I have set to splice always. I trust them it works better with them on splice, some will not even work unless they are set to splice. Streaming for example is faster on splice over intercept. Yes the firewall does not provide any tools outside of the antivirus scans. I still splice a lot.

My Splice File Example:

#Sites to be spliced
update.microsoft.com
update.microsoft.com.akadns.net
delivery.mp.microsoft.com
apple.com
cdn-apple.com
icloud.com
icloud-content.com
itunes.com
mzstatic.com
play.google.com
android.com
google-analytics.com
googleusercontent.com
gstatic.com
^((gvt)([0-9])).com
ggpht.com
dl.google.com
dl-ssl.google.com
android.clients.google.com
^(((clients)[0-9])|accounts).google.(com|us)
connectivitycheck.android.com
^((alt[0-9]-mtalk.)|(mtalk.)|(mtalk-(staging|dev).))google.com
android.clients.google.com
device-provisioning.googleapis.com
connectivitycheck.gstatic.com
play.google.com
omahaproxy.appspot.com
payments.google.com
googleapis.com
notifications.google.com
mservice.bankofamerica.com
bankofamerica.com
mcafee.com
kaspersky.com
^(pki|(crl|ocsp).pki).google.com
payments.google.com
ogs.google.com
googleapis.com
androidmanagement.googleapis.com
appldnld.apple.com.edgesuite.net
entrust.net
digicert.com
apple-cloudkit.com
apple-livephotoskit.com
gc.apple.com
icloud-content.com
olui2m.fs.ml.com
ml.com
^.*zoom.us
^.teams.microsoft.com
statics.teams.cdn.office.net
^.(outlook.)(office365|office).com
edge-chat.facebook.com
internet.speedpay.com
^.*hulustream.com
cws-hulu.conviva.com
^.*hulu.com
^.*dssott.com
prod-ripcut-delivery.disney-plus.net
^(disney.(content|connections)).edge.bamgrid.com
disney.my.sentry.io
^.*amazonvideo.com
unagi-na.amazon.com
events.data.microsoft.com
^(apis|((t|c)[0-9])).rbxcdn.com
^.*roblox.com
^.*xboxlive.com
^.*tubi.io
^.*tubitv.com
caauthservice.state.gov
mohela.com
www.whitehouse.gov

(antivirus running)

My url splice list is on a as need basis and evolves over time. For me it's basically if I use it alot and trust it add it or if it has a ethical dilemma add it, anything else inspect it check it for bugs. Tubi is a new steaming service, so I just added that for example.

[michmoor]

@JonathanLee To be fair those PUA Trojan are in more likelyhood a false positive. There are quite a few sites i visit that are safe sites but it detects PUAs. Your milage may vary but i just needed Squid to audit the sites being visited and using custom url filters especially for the Kids network ive set up. I can attach time limits along with my url filter using SquidGuard which is exactly what i need.
Hopefully one day we can have squid forward to Suricata to do scanning much like ClamAV

[JonathanLee]

@michmoor I think Squid can be set up to forward logs using syslog and setting it up to forward it where you want. Configure syslog to forward it off to 3rd party analysis software after you have the paths.

Ref:
https://wiki.squid-cache.org/SquidFaq/SquidLogs

Also just setup pfSense to do that for you with other logs at the same time,

I attempted Security Onion in a VM to use for passing log analytics, I got it working for a short while however, it is a monster image and uses a lot of resources to just run on a laptop on top of other images at the same time.

[JonathanLee]

@michmoor I use this blacklist:

http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz

I have a 2100-MAX so I use this list t's not as big as the main version. Works great if you go to their website you can also report items for their lists.