Guide to filtering web content (http and https) with pfsense 2.3

chris4916 · Jul 25, 2016, 4:59 AM

But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.

You are not obliged to redirect to LAN IP. Thik about localhost ;) then I don't think there is any loop.

What makes this "dual proxy" option difficult is that, in order to set-up transparent proxy, you will have to allow requests ton internet on port 80, while, when configuring explicit proxy, you may (should) want to deny such access so that you ensure everything goes through your proxy.
And same for HTTPS except that transparent proxy, unless you configure ssl-bump, will not intercept HTTPS flow.

phunni · Jul 30, 2016, 1:25 PM

@aGeekHere:

Bypass the proxy for Roku and Netflix.

This fixes the issue with itv hub, although bypassing for netflix.co.uk - works on the roku, but then breaks netflix on Android - presumably they're using different destination ips.

Will have to re-enable ip addresses in the url in squid.

Also, this doesn't fix the problem with youtube - you just have to allow unsafe mode…

everwake · Aug 3, 2016, 7:29 PM

this guide was very nice, but it does not work for me.

http proxy part works fine (transparent mode)

on android using automatic discovery it uses the wpad file, and looking in the log I can see entries like:

10.111.11.111/android-a41b317d63f30562.local sb.scorecardresearch.com:443 Request(default/blk_BL_tracker/-) - CONNECT REDIRECT

which implies to me that https is working, but regular expression matching for a site only works if it is http, not over https, which means that filtering using regexp for youtube and google does not work.
no filtering on sites using https seems to be happening.

safe search for youtube only works for desktop-pc, not on youtube apps. the desktop pc's say that safe search was forced by the network admin (which is what I want), but our android pads don't seem to care (rendering safe search totally useless)

I found this looking for a way to filter youtube and google, but since both sites use https it does not work, has anyone got it working or should I go to using certificates?

Regards

dotdash · Aug 3, 2016, 9:52 PM

Just a note, you can simplify the directions by touching the file instead of creating with vi.
e.g.
Ssh into the router
type 8
cd /
cd var/unbound
vi forecegoogle.conf
leave blank for now
save (wq)

Could be done with-
Ssh into the router
type 8
touch /var/unbound/forecegoogle.conf
Or you could just paste the touch command into diagnostics, command prompt and not shell in.
Just a suggestion, there are many ways to do these things.

oddworld19 · Aug 4, 2016, 1:09 PM

I have followed the guide, WPAD is working and the proxy is working in NON-transparent mode. I want to block specific subreddits, like Reddit.com/r/nsfw without blocking the remainder of Reddit.com. I do not want to block the entire domain, and I use pfblockerng and DNSBL with unbound to handle DNS and Top level domain blocking. I don't want to block all of Reddit… Just the NSFW material (and I have a list of all websites in squidguard).

At the moment, if the user access HTTP - Reddit.com/r/nsfw, then squidguard blocks at the proxy. However HTTPS requests to the same site are not blocked.

Is the proxy able to block this type of traffic without MITM / certificate installation on each host? At the moment, "transparent mode" and "MITM" are disabled on squid. Any advice is appreciated.

chris4916 · Aug 4, 2016, 1:19 PM

@oddworld19:

At the moment, if the user access HTTP - Reddit.com/r/nsfw, then squidguard blocks at the proxy. However HTTPS requests to the same site are not blocked.

Is the proxy able to block this type of traffic without MITM / certificate installation on each host? At the moment, "transparent mode" and "MITM" are disabled on squid. Any advice is appreciated.

No this will not work without MITM because CONNECT method on which Squidguard relies in order to block HTTPS knows only the left part of your URL, meaning here http://reddit.com
This part is sent in clear text but then everything else is within HTTPS tunnel thus this can't be read therefore blocked.

oddworld19 · Aug 4, 2016, 1:42 PM

Since that is literally the only site I'm trying to block on squid (and everything else is blocked using DNSBL / pfblockerng) how would you suggest I intercept HTTPS?

Do you think I could alter the WPAD rules to PROXY reddit.com and default to pass/DIRECT all other traffic? Then on squid, I enable MITM/SSL. I shouldn't use transparent mode, right, because WPAD would notify users of the proxy?

Would I use port 3128 or 3129 (I'm a little confused about the separate SSL port in the options).

Since this isn't a banking site or google site, any idea if I could configure squid such that no certs are needed on the host computers?

chris4916 · Aug 4, 2016, 2:52 PM

I can't really answer :-[

To me DNSBL doesn't aim at replacing HTTP filtering. These are on 2 different layers. If you implement DNSBL, then you will access less sites, that's it (and this is already not so bad) but HTTP proxy permits to filer at fqdn level (like DNSBL does) but also in URL (for HTTP or HTTPS is MITM) and, on top of that, to check for virus (again HTTP and also HTTPS if MITM)

Then decision to deploy proxy in transparent mode or not is entirely on your side. I'm definitely not prone to deploy transparent proxy but can understand that is some cases it may help.

oddworld19 · Aug 4, 2016, 3:18 PM

Thanks. Yes - I use DNSBL just to block at the DNS level. pfblockerng has a new update where it also includes top level domains. It does a pretty good job of blocking content at the "domain-level" via DNS. I then set-up rules in pfsense to require the network to use pfsense's DNS unbound server. I agree these are two fundamentally different technologies. However, the DNS blacklist does not cover quite everything I intend to block. I want reddit.com to work, but want to block certain sub-reddits. Obviously, DNS is the wrong tool for this job, which brought me to squid.

I do not want to enable a transparent proxy. That seems overkill if I just want to block one site.

Does SSL filtering work in non-transparent mode?

I assume so. Is the port still 3128?

chris4916 · Aug 4, 2016, 3:22 PM

@oddworld19:

I do not want to enable a transparent proxy. That seems overkill if I just want to block one site.
Does SSL filtering work in non-transparent mode?
I assume so. Is the port still 3128?

yes and yes.
And BTW you don"t need to edit proxy.pac in order to go direct for all sites but reddit.
You can still use proxy for all and only block reddit content.
Other HTTP sites will benefit from cache and all sites will potentially benefit from anti-virus if one day you enable it.

oddworld19 · Aug 4, 2016, 3:28 PM

Thanks. Right. My only thought there was:

I understand SSL decryption and inspection gets…. Nasty.

For example, if browsers require HSTS or whatever other protocols check certificates, then it might get annoying to keep installing certs on all on my computers on the LAN. For example... A small FTP server has no need to access Reddit, but I don't know if SSL bumping will cause issues with package updates in the future (without a cert).

Maybe it will... Maybe it won't. I don't know... Do you think it would potentially be an annoyance if I don't have certs installed?

maverik1 · Aug 17, 2016, 5:03 AM

Lots of good info! Thanks!!

Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box? I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard. Cannot for the life of me understand what is blocking notifications. Very annoying.

chris4916 · Aug 17, 2016, 5:31 AM

@maverik1:

Any clue as how to get android phones to be able to receive push notifications when using wifi via router connected to a pfsense box? I've enabled that particular vlans firewall to any any and the wan is any any as well. No filtering going on for that vlan via squid or squidguard. Cannot for the life of me understand what is blocking notifications. Very annoying.

What "notification" are you speaking about?
I'm confused :-[

maverik1 · Aug 18, 2016, 12:36 AM

I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.

maverik1 · Aug 18, 2016, 12:39 AM

@aGeekHere:

Update Youtube safe mode
Click add under Host overrides
Host = www
Domain = youtube.com
IP = 216.239.38.120
Description = youtube
Save
NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.

How can we get this working with mobile devices Android and iOS that use the youtube mobile app or m.youtube.com?

chris4916 · Aug 18, 2016, 7:49 AM

@maverik1:

I was speaking about push notifications on android and iOS based phones. I think I've got it working..hopefully.

Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members ;)

maverik1 · Aug 18, 2016, 5:01 PM

@chris4916:

Sorry but I still don't understand what "notification" means here and if you managed to make this working, you could at least share it (i.e. what did you do?) with other forum members ;)

Push notifications are alerts that are received by cellular phones;mostly smart phones, for different types of apps such as social media, email, games and whatnot. I have no issues receiving these alerts when using data only. However, when I was on my wifi network they were not coming through. I spend a few weeks working with the firewall rules and proxy settings trying everything I could think of to troubleshoot the issue. Turns out the problem was with the phone and not pfsense.

kpa · Aug 25, 2016, 6:39 PM

I'd be very surprised if all those "push" system weren't all emulated by polling a server periodically, that kind of set up needs absolutely nothing else but an outgoing connection from the device to the server. The other option would be a service running on the device and the notifications would be then really pushed on the device by an incoming connection from the server to the device. I don't think you'd want implement it like that however ;)

laurenzzo · Aug 30, 2016, 8:11 AM

@aGeekHere:

[…]

First we are going to setup squidguard
Update
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard).

In Package/Proxy filter SquidGuard: General settings/General settings
click blacklist
enter http://www.shallalist.de/Downloads/shallalist.tar.gz
download
wait to finish

[…]

Hi aGeekHere,

Thanks a lot for your guide !
~~I tried to follow your steps to get my blacklist downloaded but without success.

Could you please have a look on my post for more details ?~~
https://forum.pfsense.org/index.php?topic=117314.msg649961#msg649961

~~I really don't understand why the download doesn't work…~~

~~Thanks in advance for any help !~~

EDIT : SOLVED !
this was an issue with Firefox…
Could you please update your nice guide with a small note ? (e.g. "don't use Firefox to download the blacklist, IE do the job correctly")

Thanks !

Kind regards,
Laurenzzo

jadog · Sep 2, 2016, 1:21 PM

Thanks for taking the time to post this excellent guide! Just a few additions I would recommend adding.

YouTube mobile still allows unrestricted access. Including the below corrected it for me.


Click add under Host overrides
Host = m
Domain = youtube.com
IP =  216.239.38.120
Description = youtube mobile
Save

Also when using vi after you ssh into your router, using wq does not allow saving. You need to use ":wq" (without the quotes). All working perfectly for me!