Troubleshooting sites that wont load using Squid bump

[JonathanLee]

@michmoor lightsquid does. Should be a great project/puzzle to work on. My issue was Xbox not having options for proxy use like all the other game systems. So I used DHCP option 252 to configure the proxy and set it for transparent use.

[michmoor]

@JonathanLee So ive set DHCP Option 252 but it doesnt seem to be working at all.

I have LAN1 - Thats set for SSL Intercept. I got certificates installed on all my client machines. Working
I have LAN2 - This is Wifi. I got transparent mode set up. Ive set DHCP Option 252. I still dont see SSL/TLS being intercepted and seeing the reports in LightSquid.

My iPhone is getting the option. Nothing in LightSquid on port 443

[JonathanLee]

@michmoor you have wpad files set up also? Just to confirm,

The Squid proxy must have both options enabled and you will have to use advanced config to direct what vlan subnets need splice and what needs intercept. I have only done it with specific IP addresses, again you must add the subnets to Squid for use also.

(both options enabled)

(add in other vlan subnets)

[JonathanLee]

@JonathanLee

https://wiki.squid-cache.org/SquidFaq/SquidAcl

Use this webpage with advanced config and set up the subnets under Custom Options ssl/mitm area notice mine has spice only set for some and others are not they are mitm

(option 252 I set up in dhcp server for auto proxy config you must allow access to the firewalls port however with use of this)

https://192.168.1.1:8080/wpad.dat
I am using 8080 for firewall access so it must be set that way under the dhcp option 252 and access control lists

[JonathanLee]

@JonathanLee

Make sure your ACLs are configured also to force proxy use

If you want to take it further block DoH requests also with Squidguard once you get it going

[michmoor]

@JonathanLee You are a true gent. Thanks so much. I’m not using a custom configs but that looks like the next step.

[michmoor]

@JonathanLee So what do you use WPAD for? My understanding is to inform clients that are set up to find a proxy server, where it is. These are not for your clients that are set up for bump, right?

edit. Came up with my custom config using you as a guide. I dont have SSL transparent on as im using proxy authentication on certain VLANs.
So for my iPads and XBOXs do i have to find proxy configuration in the software?

acl splice_only src 192.168.14.0/24
acl splice_only src 192.168.23.0/24
acl NoSSLCIntercept ssl::server_name_regex -i "/usr/local/pkg/url.nobump"
ssl_bump splice splice_only
ssl_bump splice NoSSLCIntercept
ssl_bump bump all

[JonathanLee]

@michmoor great configuration. I use WPAD for automatic proxy configuration. Some laptops go back and forth from work or others to the university so I have WPAD set to auto configure when they get back on our home network. This way I do not have to manually set them to the proxy each and everytime they come home versus out in the world. To answer your question about the Xbox and iPad I splice them as you can not load a certificate into Xbox but you can on the iPad. The iPad I just have to set to splice so it runs and still does URL blocking, it's my wife's and it works better. The only ones I have set to intercept are ones I use for programing and cyber security researching. DHCP option all it does is serve the WPAD even when it's not asked this is for when the laptop is not set to auto for proxy, it will still configure to the proxy this way. It's like a double version. If the laptop is set to auto it configures and if not it auto configures the proxy too. Stuff has to work without my manually changing proxy settings.

[michmoor]

@JonathanLee That is the set up I have as well. I have at least 3x machines across two vlans that are set up for automatic proxy detection as they move in and out the network.
The problem I have now is that I do not see my Wireless VLAN in the Proxy Reports. I should be seeing way more devices in the 192.168.14.0/24 network but i dont. Any suggestions?

[JonathanLee]

@michmoor so do you have another subnet your not splicing like a laptop you SSL intercept with certificates loaded? That would be what falls into bump.

[JonathanLee]

@michmoor try to name it splice_only_A and splice_only_B I have not done this with different subnets yet. I think I read something on Squids website about best practices with different networks. Have you tried peek step1? You have to peek at it before it knows to splice or bump right?

ssl_bump peek step1 ---> it has to peek to make a bump decision for the urls it needs to inspect

ssl_bump splice splice_only
ssl_bump splice NoSSLCIntercept
ssl_bump bump all

It falls through this when it runs. First it splices the IPs it needs to next it has to peek to see the URLs to make a decision if it needs to splice that connection after if it matches it splices them. All others it bumps the bumps those machines must have certificates installed.

https://wiki.squid-cache.org/Features/SslPeekAndSplice

[michmoor]

@JonathanLee I figured it out. It was much more simple than that. I needed to select the other interfaces for SSL inspection. Only 2 out of the 4 were selected.

So at this point the custom SSL configuration is the way to go. Im bumping some networks and splicing others. Maximum flexability here so thank you for pointing the way.

Now i am getting a LightSquid report with all the IPs. I am not at the point that i want to use a blacklist as thats too much. I would prefer a paid URL block list if i could find one.
I am using SquidGuard to customize a very specific domain i want to block for certain users/IPs so thats great. PFblocker cannot be customized in this way so this is really good.

[JonathanLee]

@michmoor what a cool puzzle right? Squid is amazing

[michmoor]

@JonathanLee This was a great puzzle thanks to you. The downside is that I cannot use Authentication while transparent mode is enabled but thats Ok.

You should teach a squid class at the university :)

Final configuration is this.

[JonathanLee]

@michmoor your URL splice list will have a sites that cannot be SSL intercepted with bumping. If they are bank websites, Facebook messenger, etc. Just think ethically. After your set up watch how many viruses Clam finds. It's scary when it's running to see how many issues there really are.

[JonathanLee]

@michmoor you can use authentication, I think I seen something about that. Look at the first Squid URL I sent in a prior post here the info is on that website. It has a custom config to do that. I have never done it. My primary goal is pure cyber security issues thus getting Clam AV to block and log issues so I can report them.

[JonathanLee]

@michmoor

https://wiki.squid-cache.org/Features/Authentication

You can do custom authentication also.

"Simply said, it’s not possible to authenticate users using proxy authentication schemes when running in interception or transparent modes. See SquidFaq/InterceptionProxy for details on why."

Yes you are correct

[michmoor]

@JonathanLee said in Troubleshooting sites that wont load using Squid bump:

your URL splice list will have a sites that cannot be SSL intercepted with bumping. If they are bank websites, Facebook messenger, etc. Just think ethically. After your set up watch how many viruses Clam finds. It's scary when it's running to see how many issues there really are.

I havent enabled ClamAV. Not sure if its worth it. When i last did enable it i got lots of false positives. So for now just saving my cpu cycles and letting the endpoint protection on that.

As far as bank websites and FB messenger is concerned there is no way to actually inspect the data once its decrypted, right? So i can see the full URL but i cant see the payload so sites like banks or facebook getting bumped i dont think its a problem. pfSense doesnt have a way to inspect the payload after its decrypted.

[JonathanLee]

@michmoor Some websites won't work unless they are splice also. I don't know if you have ever used or pen tested with "Burp Suite" you can see a lot of information when sites are not set to splice. For antivirus in the proxy it's a dream as it can inspect everything for bugs I have caught hundreds of items. Again, if your not using ClamAV I don't see the reason. These are just some of the sites I have set to splice always. I trust them it works better with them on splice, some will not even work unless they are set to splice. Streaming for example is faster on splice over intercept. Yes the firewall does not provide any tools outside of the antivirus scans. I still splice a lot.

My Splice File Example:

#Sites to be spliced
update.microsoft.com
update.microsoft.com.akadns.net
delivery.mp.microsoft.com
apple.com
cdn-apple.com
icloud.com
icloud-content.com
itunes.com
mzstatic.com
play.google.com
android.com
google-analytics.com
googleusercontent.com
gstatic.com
^((gvt)([0-9])).com
ggpht.com
dl.google.com
dl-ssl.google.com
android.clients.google.com
^(((clients)[0-9])|accounts).google.(com|us)
connectivitycheck.android.com
^((alt[0-9]-mtalk.)|(mtalk.)|(mtalk-(staging|dev).))google.com
android.clients.google.com
device-provisioning.googleapis.com
connectivitycheck.gstatic.com
play.google.com
omahaproxy.appspot.com
payments.google.com
googleapis.com
notifications.google.com
mservice.bankofamerica.com
bankofamerica.com
mcafee.com
kaspersky.com
^(pki|(crl|ocsp).pki).google.com
payments.google.com
ogs.google.com
googleapis.com
androidmanagement.googleapis.com
appldnld.apple.com.edgesuite.net
entrust.net
digicert.com
apple-cloudkit.com
apple-livephotoskit.com
gc.apple.com
icloud-content.com
olui2m.fs.ml.com
ml.com
^.*zoom.us
^.teams.microsoft.com
statics.teams.cdn.office.net
^.(outlook.)(office365|office).com
edge-chat.facebook.com
internet.speedpay.com
^.*hulustream.com
cws-hulu.conviva.com
^.*hulu.com
^.*dssott.com
prod-ripcut-delivery.disney-plus.net
^(disney.(content|connections)).edge.bamgrid.com
disney.my.sentry.io
^.*amazonvideo.com
unagi-na.amazon.com
events.data.microsoft.com
^(apis|((t|c)[0-9])).rbxcdn.com
^.*roblox.com
^.*xboxlive.com
^.*tubi.io
^.*tubitv.com
caauthservice.state.gov
mohela.com
www.whitehouse.gov

(antivirus running)

My url splice list is on a as need basis and evolves over time. For me it's basically if I use it alot and trust it add it or if it has a ethical dilemma add it, anything else inspect it check it for bugs. Tubi is a new steaming service, so I just added that for example.

[michmoor]

@JonathanLee To be fair those PUA Trojan are in more likelyhood a false positive. There are quite a few sites i visit that are safe sites but it detects PUAs. Your milage may vary but i just needed Squid to audit the sites being visited and using custom url filters especially for the Kids network ive set up. I can attach time limits along with my url filter using SquidGuard which is exactly what i need.
Hopefully one day we can have squid forward to Suricata to do scanning much like ClamAV