Web and Application Filtering on VLANs

raverX · Oct 28, 2014, 12:29 AM

Hi All,

I have a site that has a single WAN interface (150mbps bidirectional fibre connection), with 20 VLANs on one Gigabit LAN interface (numbered VLAN220-VLAN239).

We are trying to work out ways to setup some form of Web Filtering on select VLANs. I have tried installing Squid+SquidGuard, however enabling "Transparent Proxing" results in all web traffic ignoring any Traffic Shaping/Limiting I have applied on HTTP traffic for that VLAN.

I tried installing DansGuardian simply to provide a way to setup a rule on the individual VLAN to forward all HTTP traffic to DansGuardian, which then forwarded to Squid, however the users complained profusely of slow web performance.

I tried tweaking the settings in Squid, SquidGuard and DansGuardian without much success. The machine running pfSense is by no means "sluggish".

Running: pfSense 2.1.5-RELEASE (amd64)
CPU Type: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (16 CPUs: 2 package(s) x 4 core(s) x 2 SMT threads)
Memory usage: 14% of 24546 MB
SWAP usage: 0% of 65536 MB
Disk usage: 13% of 389G

I don't really care about proxying. What I want to be able to achieve is to block access to certain types of websites and content - ie: blocking access to online radio, iTunes, Spotify, DropBox, etc. It seems most the big bandwidth using applications now use HTTP/HTTPS as their transport medium, making it rather difficult to curb these bandwidth hogs.

I'm getting to a point where I am wondering if I've reached the extent of pfSense's capabilities and may need to put in something like a Cyberoam UTM with Web and Application filtering, though due to the size of the site, it won't be a "cheap" exercise - which is why we've always used pfSense (and done so very well thank you to the team!)

Does anyone have any recommendations on how this sort of filtering might be achieved with pfSense?

cthomas · Jan 24, 2015, 8:19 PM

You could create your own Aliases for each service you want to block. It's not that hard to research the IP's that are in use by a given company, or simply perform a packet capture and see where they are connecting to. (I know, everyone wants to simply click a button..)

Or, you could use pfSense to peel off the domain names (which catches the sub-domains) using "Domain Override" and black-hole them.

And there are a number of IP List providers that might be of use.

https://www.iblocklist.com
http://zeltser.com/combating-malicious-software/malicious-ip-blocklists.html

Derelict · Jan 24, 2015, 8:25 PM

If Squid/Squidguard do what you need, how about a second pfSense in transparent mode between your main router and WAN that does nothing but that?

Harvy66 · Jan 24, 2015, 8:58 PM

What does this have to do with traffic shaping?

Derelict · Jan 24, 2015, 9:29 PM

I have tried installing Squid+SquidGuard, however enabling "Transparent Proxing" results in all web traffic ignoring any Traffic Shaping/Limiting I have applied on HTTP traffic for that VLAN.

Never tested that. I'm assuming he's correct in his observation.

BBcan177 · Jan 24, 2015, 10:26 PM

@raverX:

I don't really care about proxying. What I want to be able to achieve is to block access to certain types of websites and content - ie: blocking access to online radio, iTunes, Spotify, DropBox, etc. It seems most the big bandwidth using applications now use HTTP/HTTPS as their transport medium, making it rather difficult to curb these bandwidth hogs.

When pfBlockerNG is released as a package, you will be able to pull IPs from sources like HE (Hurricane Electric).

These are web-based links that need to be converted into a txt file so that pfBlockerNG can create firewall rules to block access. These can be polled as required to get any IP changes.

These same lists can also be used to Permit or Match, so that you can log the activity and actively pay a visit to the desks of those involved. >:( ;D

A lot easier than using a Proxy etc… (Just can't do content filtering). people can still try to go around with VPNs or online proxies... But I'm sure there are lists that can be used for those also. Cat and mouse saga will never end.

http://bgp.he.net/search?search%5Bsearch%5D=twitter&commit=Search
http://bgp.he.net/search?search%5Bsearch%5D=facebook&commit=Search
http://bgp.he.net/search?search%5Bsearch%5D=spotify&commit=Search
http://bgp.he.net/search?search%5Bsearch%5D=dropbox&commit=Search

raverX · Apr 18, 2016, 1:03 AM

@Harvy66:

What does this have to do with traffic shaping?

Ideally we want to be able to apply traffic shaping rules ONTO different types of web traffic.

For example, on our Cyberoam UTM's, we can apply a policy on Spotify or YouTube traffic, so that it gets a lower priority that identified business services. This way the type of traffic doesn't need to be blocked, just that if a user is trying to access something business related, the business related activity gets priority over the non-productive activity.

bwf.it35218 · Apr 21, 2016, 5:52 AM

Ideally we want to be able to apply traffic shaping rules ONTO different types of web traffic.

To do this would require a layer7 filter, which has unfortunately been removed from 2.3 (it was removed because it wasn't working correctly in 2.2.x) I have already submitted a feature request for a replacement option - nDPI from ntop - https://redmine.pfsense.org/issues/5813 - maybe add your vote to it