SquidGuard blacklist now bypassing

[Jeffrey_223]

Hi All

I followed this vid for my setup https://www.youtube.com/watch?v=5HGjfWHlmCI

Now it all seems to work till i turn on the blacklist with SSL filtering and then almost anything with https is blocked.
On the Common categories I've set default access to allow.

I've tried to create a whitelist and a deny list on the Target Categories but that's not working also not sure i've set that right.

And flicked between the categories on the Common list with no luck in figuring out why it's just blocking everything.
Nothing set in Group ACLs

All my machines have the CA cert from my pfsense and the transparent proxy is working fine.
I'm on version 2.6.0-RELEASE of PF sense
squid is 0.445-9
squidGuard 1.16.18-20

If i go to the log of SquidGuard the blacklist log has no data. Even though I'm pretty sure it's turned on.
Could anyone shed some light on this for me?

Thanks

[JonathanLee]

@jeffrey_223 have you added your default category and added the loopback? Also make sure squid allows the loopback on the ACL list.

[JonathanLee]

@jeffrey_223

Reference how to install the Squid certificate I had to generate it in the command line and load it into the Pfsense

This works for version 22.05 better when you load the certificate.

Check it out Ref: https://forum.it-monkey.net/index.php?topic=23.0

This site had the best walk through with setting this up outside of the advanced options.

[Jeffrey_223]

Thanks @JonathanLee
Your link was great.
I didn't set the loopback in the general settings.
I'll have to test that out.

I created the cert in pfsense and deployed it via Active Directory so all deployed to my workstations.

I've had to disable this for the moment because i couldn't get it to work. I'll be back on it in about 2 weeks.
I'll let you know how I go.

[JonathanLee]

@jeffrey_223 I actually had to create the certificate inside of squid in command line for this to work correctly. Once it was created import it into PfSense.

[Jeffrey_223]

Hey @jonathanlee
So i followed you guide mostly.
I know my cert works because I'm not getting errors with HTTPS and
I've set the loopback this time.
I have enabled Transparent because I want all my Private addresses to bypass and there are subnets i don't want the proxy to apply to.

Everything has access to my firewall for getting the proxy settings as we all on windows machine and i can see in the proxy setting that it's all there. So didn't create a WPAD.

It seem like the allow and deny list are not working correctly.
Everything is being blocked.

[JonathanLee]

@jeffrey_223 please mark the dummy list as whitelist.

I noticed yours is -

Do you have logs in squid and squid guard?

[Jeffrey_223]

@jonathanlee
It was working I blocked youtube and then facebook.
As soon as i made a change every started to get blocked.
And i can't seem to get the whitelist to work either.
Is there a way to bypass sites?
Should it be this fiddly, i fell like once i get this going I'll never be able to add or edit it cause it'll break?

From squidguard logs i have 'Filter GUI logs'

12.12.2022 20:26:12	[squid_reconfigure] Remove old redirector options from Squid config.
12.12.2022 20:26:12	[sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
12.12.2022 20:26:12	[sg_redirector_base_url] Select redirector base url (https://10.x.x.37:443/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
12.12.2022 20:26:12	[sg_create_config] Add Default
12.12.2022 20:26:12	[sg_create_config] Add rewrites: safesearch;
12.12.2022 20:26:12	[sg_create_config] Add destinations: Dummy; Blocked_Extra; ShotGrid_Allow; squidstatus;
12.12.2022 20:26:12	[sg_redirector_base_url] Select redirector base url (https://10.x.x.37:443/sgerror.php?url=403%20This%20site%20has%20been%20blocked%20by%20your%20Systems%20Admin%2C%20if%20you%20have%20a%20business%20reason%20to%20visit%20this%20page%20contact%20your%20supervisor.&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
12.12.2022 20:26:12	[sg_redirector_base_url] Select redirector base url (https://10.x.x.37:443/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
12.12.2022 20:26:12	[sg_create_config] Add sources: Nothing.
12.12.2022 20:26:12	[squidguard_rebuild_db] Start rebuild DB.
12.12.2022 20:26:02	[squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
12.12.2022 20:26:02	[sg_redirector_base_url] Select redirector base url (https://10.x.x.37:443/sgerror.php?url=403%20404&a=%a&n=%n&;i=%i&s=%s&t=%t&u=%u)
12.12.2022 20:26:02	[sg_create_simple_config] Added item 'squidstatus' = '/var/db/squidGuard/squidstatus'.
12.12.2022 20:26:02	[sg_create_simple_config] Added item 'ShotGrid_Allow' = '/var/db/squidGuard/ShotGrid_Allow'.
12.12.2022 20:26:02	[sg_create_simple_config] Added item 'Blocked_Extra' = '/var/db/squidGuard/Blocked_Extra'.
12.12.2022 20:26:02	[sg_create_simple_config] Warning Ignored empty item 'Dummy' = '/var/db/squidGuard/Dummy'.
12.12.2022 20:26:02	[sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
12.12.2022 20:26:02	[squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
12.12.2022 20:26:01	[sg_reconfigure_user_db] Add squidstatus domains '10.x.x.254 127.0.0.1 ';
12.12.2022 20:26:01	[sg_reconfigure_user_db] Add ShotGrid_Allow urls 'X.shotgunstudio.com/
12.12.2022 20:26:01	[sg_reconfigure_user_db] Add ShotGrid_Allow domains 'X.shotgunstudio.com launchdarkly.shotgrid.autodesk.com sg-software.ems.autodesk.com sg-sec.s3-accelerate.amazonaws.com sg-media-sydney.s3.amazonaws.com sg-media-sydney.s3-accelerate.amazonaws.com tank.shotgunstudio.com s3-proxy.shotgrid.autodesk.com s3-proxy.shotgunstudio.com api.amplitude.com autodesk-prod.okta.com autodesk.com oktacdn.com tiqcdn.com';
12.12.2022 20:26:01	[sg_reconfigure_user_db] Add Blocked_Extra urls 'cnn.com/
12.12.2022 20:26:01	[sg_reconfigure_user_db] Add Blocked_Extra domains 'facebook.com fb.com youtube.com';
12.12.2022 20:26:01	[sg_reconfigure_user_db] Add user entries
12.12.2022 20:26:01	[sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
12.12.2022 20:22:15	[squid_reconfigure] Add new redirector options to Squid config.
12.12.2022 20:22:15	[squid_reconfigure] Remove old redirector options from Squid config.
12.12.2022 20:22:15	[sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
12.12.2022 20:22:15	[sg_redirector_base_url] Select redirector base url (https://10.x.x.37:443/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
12.12.2022 20:22:15	[sg_create_config] Add Default
12.12.2022 20:22:15	[sg_create_config] Add rewrites: safesearch;
12.12.2022 20:22:15	[sg_create_config] Add destinations: Dummy; Blocked_Extra; ShotGrid_Allow; squidstatus;
12.12.2022 20:22:15	[sg_redirector_base_url] Select redirector base url (https://10.x.x.37:443/sgerror.php?url=403%20This%20site%20has%20been%20blocked%20by%20your%20Systems%20Admin%2C%20if%20you%20have%20a%20business%20reason%20to%20visit%20this%20page%20contact%20your%20supervisor.&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)

Then 'Filter logs'

12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:17	squidGuard stopped (1670837177.043)
12.12.2022 20:26:02	squidGuard stopped (1670837162.110)
12.12.2022 20:26:02	db update done
12.12.2022 20:26:02	squidGuard 1.4 started (1670837162.012)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.164)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.163)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.148)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.147)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.131)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.130)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.118)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.117)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.105)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.103)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.105)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.103)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.089)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.088)
12.12.2022 20:23:25	squidGuard ready for requests (1670837005.075)
12.12.2022 20:23:25	squidGuard 1.4 started (1670837005.073)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:23:19	squidGuard stopped (1670836999.295)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:21	squidGuard stopped (1670836941.003)
12.12.2022 20:22:20	squidGuard ready for requests (1670836940.993)
12.12.2022 20:22:20	squidGuard 1.4 started (1670836940.992)
12.12.2022 20:22:20	squidGuard ready for requests (1670836940.974)
12.12.2022 20:22:20	squidGuard 1.4 started (1670836940.972)
12.12.2022 20:22:20	squidGuard ready for requests (1670836940.966)
12.12.2022 20:22:20	squidGuard 1.4 started (1670836940.964)

[JonathanLee]

@jeffrey_223 any time I make changes I have to save, and apply. Once it takes the settings, I always restart the firewall. Squidguard integrates into the Squid proxy. So any changes require the reboot to fully work. It's not really meant to have on the fly changes, changes are meant for after hours. That's the only drawback, it is literally filtering the whole internet in real time and blocking specific items. It's meant to run without interrupting once it's configured. Doesn't mean changes can't be done, it just doesn't do them instantly. Glad you got it working.

(Image: Blacklist loaded)

http://dsi.ut-capitole.fr/blacklists/download/blacklists_for_pfsense_reducted.tar.gz

I use the above blacklist, that is why I have categories.

[JonathanLee]

@jeffrey_223

This is how I got my allow list to work

Example:

settings.win.data.microsoft.com login.live.com hulu-lr.hb-api.omtrdc.net www.zoo.com zoo.com au.download.windowsupdate.com zoom.us google.zoom.us cccconfer.zoom.us 192.168.1.3 api.onedrive.com smartscreen-prod.microsoft.com nav.smartscreen.microsoft.com browser.pipe.aria.microsoft.com nc-ring.msedge.net a-ring.msedge.net arc.msn.com cdn.onenote.net checkappexec.microsoft.com config.edge.skype.com config.teams.microsoft.com ctldl.windowsupdate.com emdl.ws.microsoft.com fe2cr.update.microsoft.com fe3cr.delivery.mp.microsoft.com slscr.update.microsoft.com evoke-windowsservices-tas.msedge.net fp.msedge.net fp-vp.azureedge.net g.live.com go.microsoft.com iecvlist.microsoft.com inference.location.live.net img-prod-cms-rt-microsoft-com.akamaized.net login.live.com logincdn.msauth.net manage.devcenter.microsoft.com maps.windows.com ocsp.digicert.com ocsp.msocsp.com mobile.pipe.aria.microsoft.com ris.api.iris.microsoft.com settings-win.data.microsoft.com spo-ring.msedge.net telecommand.telemetry.microsoft.com tile-service.weather.microsoft.com tsfe.trafficshaping.dsp.mp.microsoft.com v10.events.data.microsoft.com wdcp.microsoft.com www.bing.com www.msftconnecttest.com outlook.office365.com outlook.office.com storage.live.com skydrivesync.policies.live.net windows.policies.live.net activity.windows.com adl.windows.com spclient.wg.spotify.com cs.dds.microsoft.com msedge.b.tlu.dl.delivery.mp.microsoft.com liveupdate.symantecliveupdate.com tlu.dl.delivery.mp.microsoft.com geo.kaspersky.com displaycatalog.mp.microsoft.com hulu.com vod-akc-na-west-1.media.dssott.com disney.content.edge.bamgrid.com disney.my.sentry.io amazonvideo.com api.amazonvideo.com tv.apple.com cdn-apple.com 129.32.209.205 129.32.209.204 129.32.209.202 129.32.209.201 teams.microsoft.com statics.teams.cdn.office.net wpad.home.arpa proxy.pfsense.secure lee_family.home.arpa 192.168.1.1

Example URL list:

settings.win.data.microsoft.com/ login.live.com/ hulu-lr.hb-api.omtrdc.net/ www.zoo.com/ zoo.com/ au.download.windowsupdate.com/ zoom.us/ google.zoom.us/ cccconfer.zoom.us/ 192.168.1.3/ api.onedrive.com/ smartscreen-prod.microsoft.com/ nav.smartscreen.microsoft.com/ browser.pipe.aria.microsoft.com/ c-ring.msedge.net/ a-ring.msedge.net/ arc.msn.com/ cdn.onenote.net/ checkappexec.microsoft.com/ config.edge.skype.com/ config.teams.microsoft.com/ ctldl.windowsupdate.com/ emdl.ws.microsoft.com/ fe2cr.update.microsoft.com/ fe3cr.delivery.mp.microsoft.com/ slscr.update.microsoft.com/ evoke-windowsservices-tas.msedge.net/ fp.msedge.net/ fp-vp.azureedge.net/ g.live.com/ go.microsoft.com/ iecvlist.microsoft.com/ inference.location.live.net/ img-prod-cms-rt-microsoft-com.akamaized.net/ login.live.com/ logincdn.msauth.net/ manage.devcenter.microsoft.com/ maps.windows.com/ ocsp.digicert.com/ ocsp.msocsp.com/ mobile.pipe.aria.microsoft.com/ ris.api.iris.microsoft.com/settings-win.data.microsoft.com/spo-ring.msedge.net/ telecommand.telemetry.microsoft.com/ tile-service.weather.microsoft.com/ tsfe.trafficshaping.dsp.mp.microsoft.com/ v10.events.data.microsoft.com/ wdcp.microsoft.com/ www.bing.com/ www.msftconnecttest.com/ outlook.office365.com/ outlook.office.com/ storage.live.com/ skydrivesync.policies.live.net/ windows.policies.live.net/ activity.windows.com/ adl.windows.com/ spclient.wg.spotify.com/ cs.dds.microsoft.com/ msedge.b.tlu.dl.delivery.mp.microsoft.com/ liveupdate.symantecliveupdate.com/ tlu.dl.delivery.mp.microsoft.com/ geo.kaspersky.com/ displaycatalog.mp.microsoft.com/ hulu.com/ vod-akc-na-west-1.media.dssott.com/ disney.content.edge.bamgrid.com/ disney.my.sentry.io/ amazonvideo.com/ api.amazonvideo.com/ tv.apple.com/ cdn-apple.com/ 129.32.209.205/ 129.32.209.202/ 129.32.209.201/ 129.32.209.204/ teams.microsoft.com/ statics.teams.cdn.office.net/ wpad.home.arpa/ proxy.pfsense.secure/ lee_family.home.arpa/ 192.168.1.1/

Example Reg Expression:

(^.prod.do.dsp.mp.microsoft.com.|^.update.microsoft.com.|^.windowsupdate.com.|^.wns.windows.com.|^.dl.delivery.mp.microsoft.com.|^.storecatalogrevocation.storequality.microsoft.com.|^.arc.msn.com.|^.blob.core.windows.net.|^.displaycatalog.mp.microsoft.com.|^.licensing.mp.microsoft.com.|^.pti.store.microsoft.com.|^.watson.telemetry.microsoft.com.|^.au.download.windowsupdate.com.|^.msedge.b.tlu.dl.delivery.mp.microsoft.com.|^.liveupdate.symantecliveupdate.com.|^.delivery.mp.microsoft.com.|^.geo.kaspersky.com.|^.192.168.1.3.|^.displaycatalog.mp.microsoft.com.|^.hulu.com.|^.vod-akc-na-west-1.media.dssott.com.|^.disney.content.edge.bamgrid.com.|^.disney.my.sentry.io.|^.amazonvideo.com.|^.api.amazonvideo.com.|^.tv.apple.com.|^.cdn-apple.com.|^.teams.microsoft.com.|^.wpad.home.arpa.|^.proxy.pfsense.secure.|^.lee_family.home.arpa.|^.192.168.1.1.$)

[JonathanLee]

@jeffrey_223

Note I have them marked whitelist that means they are always allowed.

(image Dummy list showing loopbacks)

[JonathanLee]

@jeffrey_223

Make sure you make group ACLS this is just the IP addresses of specific requirements

Example loopback:

Example Protected Child Group ACL:

note the deny items.

Targets are what you want to filter approve or block they would show in the Target rule list

Example: I block docker rubygems prakdial, callrail.

[JonathanLee]

@jeffrey_223

What do you logs for this look like?

[JonathanLee]

@jeffrey_223

Check this location for ACLS

I have two subnets one for the xbox one for the rest notice whitelist .* it will let everything work unless the Squidgard blocks it.

You have approved access to your proxy in the firewall ACLs?

Example:

(ignore the offline timer I have a schedule offtime for our house at 12:30am or we become zombies)
Line two port 3128-3130
Line three is for use with my WPAD make sure you enable WPAD before you make a port 80 rule or you will get locked out.

https://docs.netgate.com/pfsense/en/latest/recipes/http-client-proxy-wpad.html

Best Document for explaining WPAD ->

I am testing this WPAD today

[Jeffrey_223]

@jonathanlee Thanks for all the extra info.
Wow, restart the firewall after new changes didn't think of doing that.
Will do my initial setup and try it again.
All my DNS is running off a Domain Controller.

As I'm using this in a work environment and only need to apply this to users, is there a way to make the proxy run on specific subnets?
I thought I had this right but now not so sure.
e.g. filer on 10.10.10.0/24 and 10.10.40.0/24 but exclude all my other subnets.

[JonathanLee]

@jeffrey_223 I only have 4gb ram some Netgate systems have more memory and can do better on the fly changes. Squid guard let's you do different target categories that can be IP address specific.

[JonathanLee]

@jeffrey_223 make sure you run squid check in the command area

this is what I get I just learned about this today

it will show you configuration errors if you have any in squid

[JonathanLee]

@jeffrey_223 One last note,

for Wpad to work with the blocked sites like this. . .

(Image: Hotjar blocked and splash screen showing)
You have to adapt the admin access certificate to be a intermediate, it must use the ca that you created with Squid, or it will give common name errors. Or use a PfSense CA and make a intermediate just for admin access