Squid question

chris4916

Decided to give up on squid all together. The web performance is to un predictable. Frankly, the web runs better without it. Don't know why.

If goal is performance (I would say only), then browsing internet is quite obviously faster without any proxy.
Reason is that when done directly (i.e. w/o proxy) there is at least one stage less, thus less latency, no anti-virus, blacklist and similar stuff with slow-down impact.
This is specially true nowadays with dynamic nature of internet. Few years ago, content was more static and cache impact on speed (assuming not HTTPS and multiple users) was obvious.

Decision for proxy deployment should, IMO, rather be based on services it adds (like profiling, anti-virus, black-list) rather than speed, internet access through proxy being always slower but this is the cost for additional services.

Guest

All this I understand but considered it due to 1 main consideration; which I mentioned in the other post I made. Basically, with things being the way they are, and wanting the best possible school experience for my kids, my wife and I decided to home school our kids via K-12 online. lets' face it, with public education declining and enforcement of common core in practice, kids are being taught politically correct nonsense. I already have a sever running home to stream TV and Movies using a DLNA program and I set it up also to run a Active directory service, DNS, DHCP and WSUS to help keep our network clean and trouble free so it seemed logical to me to be able to filter the web content being viewed by my teenagers since they have their own PC's. Plus with Active Directory, I am able to set login times, block computer features and such which saves me a lot of hassle of yelling at them when they are doing things on the computer and web during school hours that they are not supposed to be doing.

I really don't mind the web being a little slow. However the issue I was having running squid was that the web would come to a stop speed quite frequently, web pages would simply not load even without squid guard loaded which is not good considering school work must be done online. I'm planning on whipping the installation again today since last time I did the easy setup which left setting intact on the drive. I will reload squid proxy when I get it up and going again.

I would like to ask if you could explain to me exactly, step by step please, how to make the wpad files. The links provided don't explain it step by step.

KOM

The pfSense WPAD page has it right there under Create wpad.dat:

The contents of the example wpad.dat file are:

function FindProxyForURL(url,host)
{
return "PROXY 192.168.1.1:3128";
}

It's just a text file with the contents above and nothing more. Replace 192.168.1.1 with your pfSense LAN IP address.

Make a copy of the wpad.dat file and name it proxy.pac

Copy both to the root of an HTTP server (not HTTPS, doesn't work with that). You can use pfSense for this if you aren't running WebGUI in HTTPS mode (copy the files to /usr/local/www).

Add an A record in your DNS that points wpad.YourDomain.whatever to the IP address of the HTTP server where the wpad.dat file is.

(Optional) Add a DHCP Option 252 record and set it to http://wpad.YourDomain.whatever

Test by trying to access http://wpad.YourDomain.whatever/wpad.dat and see if it can resolve and serve it.

That's it. You're done.

If you want, you can explore the other examples of wpad.dat that can control various aspects of proxy redirection, but this should get you going.

Guest

once I have the file created, how do I place it in the pfsense directory. I mean, is there a line command feature to enable this? Also, I'm not familiar with what you mean by A record

Guest

I think I figured out how to create the files in pfsense. I navigated to Diagnostics/Edit File, chose a random file and renamed it to wpad.dat and save it which created the file. Then while it was still open, I deleted all entries and replaced them with the specified text for the wpad file. I then saved this as wpad.da and proxy.pac. I pressed forward with the link you posted about creating this file and created the firewall rule for port 80 but it black all web traffic. I think this is what you meant by A record. Obviously, something is causing either my computer from getting the auto config or it is not working right. I should also mention my browsers are set to auto detect. You mentioned something about creating a DHCP option but I am not sure how to do this.

Would you mind clarifying the A record thing and the DHCP option. Plus let me know if I created the wpad files correctly. By the way, I gave this a shot "http://wpad.YourDomain.whatever/wpad.dat" and it worked with the setting I entered.

Guest

Plus with Active Directory, I am able to set login times, block computer features and such which saves
me a lot of hassle of

Then please it would be really easier to set up the Proxy settings by using the GPOs!
If any PC is an member of an active directory it would be really easy to do so for you.

Using a transparent proxy or a non-transparent proxy is also mostly pending on the needs
you have and the goals you want or must reach. For sure for some peoples also a philosophy.

For a transparent proxy you must bridge ports together to set them in a so called "promiscuous mode"
or at these days more common using the bypass mode function of the NICs that are in use.

But this is mostly earlier or later bringing more or less problems then benefits in my eyes.
And yes the entire speed of the Squid, SquidGuard & HAVP proxy would be significant
slow down the entire packet flow or Internet throughput, here it should be really fast
hardware or clever tuned the proxy.

chris4916

@BlueKobold:

Plus with Active Directory, I am able to set login times, block computer features and such which saves
me a lot of hassle of

Then please it would be really easier to set up the Proxy settings by using the GPOs!
If any PC is an member of an active directory it would be really easy to do so for you.

Sure. For Windows clients only (using IE ony ?) :P

Using a transparent proxy or a non-transparent proxy is also mostly pending on the needs
you have and the goals you want or must reach. For sure for some peoples also a philosophy.

For a transparent proxy you must bridge ports together to set them in a so called "promiscuous mode"
or at these days more common using the bypass mode function of the NICs that are in use.

When using iptables, transparent proxy is achieved forwarding packets to squid.
For sure it depends on needs. Some years ago, if goal was to provide HTTP cache only, I mean without content filtering neither access control, transparent proxy was a valid option. You transparently redirect all HTTP flow to Squid cache and it works. No HTTPS (that is not cached BTW). perfect 8)

But when it comes to provide ACL, profiling, anti-virus, transparent proxy doesn't work. And this is not, IMO ;) a matter of philosophy.

And yes the entire speed of the Squid, SquidGuard & HAVP proxy would be significant
slow down the entire packet flow or Internet throughput, here it should be really fast
hardware or clever tuned the proxy.

I fully share. There is an impact due to introduction of Squid and associated services in the middle. Still, if hardware is sized as expected, impact should be negligible, unless you spend (waste ???) time measuring if there is an impact or not ;D

Guest

Would you mind clarifying the A record thing and the DHCP option

chris4916

To be honest, the need for A record is something still not 100% clear in my mind.
Some RFC describe DNS content as to be an A record while some others describe either A record or CNAME.

Frankly, I don't see why it would not work with CNAME.

To rephrase it, if you create an alias in your DNS pointing to your web server as wpad.your_local_domain, it should work.

DHCP option: this is as simple as pushing option 252 with other DHCP informations.
it requires 2 lines in DHCP:

one describing the option itself as "option 252"
another describing its content

see an example here.
If you are using pfSense as your DHCP server, you can just add it to the "option" section of DHCP web interface ;)

Guest

Sure. For Windows clients only (using IE ony ?) :P

My answer was based on the statement shown in the next Quote 8)

I should also point out that I run windows 8.1 64 pro on all my computers and server 2012 R2

KOM

A CNAME is an alias for an existing A record.

http://www.zytrax.com/books/dns/

Guest

Got it all and I think my router is now properly configured with both squid 2 and squid guard. One question though; and I can't stress enough how much of an ear full I'm getting from my wife about this issue, is that the internet seems to now crawl to a stop since running these programs in pfsense. I don't mean it runs super slow, I mean if you reboot the router the web is fast. It will runs fast for a few minutes then it start slowing down until it comes to a stop where now web pages will load yet we are still connected to the internet. I am assuming this has something to do with web caching. I had left it set at the default 100mb and the internet had stopped like above. I changed it to 200 and end it is moving quickly again but not sure if it will keep the internet moving as it should. Any recommendations on this. Additionally, I wasn't sure what most of these setting did so I left them as default. What else should I change. See screen shots attached:

EDIT: I forgot to add here that our TV's run through the same LAN port as our pc's and my wife use the smart apps on the TV to watch Netflix and amazon prime streaming services. Is it possible that this is what is filling up the cache? If so, how do I stop the router from caching them?

EDIT 2: Under the Squid\Cache Mgmt, there is an option labeled " Hard Disk Cache System " with a drop down. I tried setting this the "diskd" to see what happens but I am curious what the affects/impacts are if I set it to null. There is a description for each and null says do not use any storage. Obviously this means it caches nothing but what will be the impact or affects of setting to null?

Untitled.png_thumb

Untitled1.png_thumb

Guest

Is it possible that this is what is filling up the cache?

This could really being.

If so, how do I stop the router from caching them?

Click please on the second piture in your last posting (Untiteld1.jpg)
Please have a look at the bottom line, the last entry Do not cache
Place the right entries of the TV or streaming providers that must match
this and I think you should also reboot the pfSense then.

Guest

I get that, I searched for Netflix IP and such and came up with a massive list:

108.175.32.0/24 Netflix Streaming Services Inc. 256
108.175.33.0/24 Netflix Streaming Services Inc. 256
108.175.34.0/24 Netflix Streaming Services Inc. 256
108.175.35.0/24 Netflix Streaming Services Inc. 256
108.175.38.0/24 Netflix Streaming Services Inc. 256
108.175.39.0/24 Netflix Streaming Services Inc. 256
108.175.40.0/24 Netflix Streaming Services Inc. 256
108.175.41.0/24 Netflix Streaming Services Inc. 256
108.175.42.0/24 Netflix Streaming Services Inc. 256
108.175.43.0/24 Netflix Streaming Services Inc. 256
108.175.44.0/24 Netflix Streaming Services Inc. 256
108.175.46.0/24 Netflix Streaming Services Inc. 256
108.175.47.0/24 Netflix Streaming Services Inc. 256
185.2.220.0/24 Netflix Streaming Services Inc. 256
185.2.221.0/24 Netflix Streaming Services Inc. 256
185.2.222.0/24 Netflix Streaming Services Inc. 256
185.2.223.0/24 Netflix Streaming Services Inc. 256
185.9.188.0/24 Netflix Streaming Services Inc. 256
185.9.189.0/24 Netflix Streaming Services Inc. 256
185.9.190.0/23 Netflix Streaming Services Inc. 512
192.173.112.0/20 Netflix Streaming Services Inc. 4,096
192.173.64.0/20 Netflix Streaming Services Inc. 4,096
192.173.64.0/24 Netflix Streaming Services Inc. 256
192.173.80.0/20 Netflix Streaming Services Inc. 4,096
192.173.96.0/20 Netflix Streaming Services Inc. 4,096
198.38.100.0/24 Netflix Streaming Services Inc. 256
198.38.101.0/24 Netflix Streaming Services Inc. 256
198.38.102.0/23 Netflix Streaming Services Inc. 512
198.38.102.0/24 Netflix Streaming Services Inc. 256
198.38.108.0/24 Netflix Streaming Services Inc. 256
198.38.109.0/24 Netflix Streaming Services Inc. 256
198.38.110.0/24 Netflix Streaming Services Inc. 256
198.38.111.0/24 Netflix Streaming Services Inc. 256
198.38.112.0/24 Netflix Streaming Services Inc. 256
198.38.113.0/24 Netflix Streaming Services Inc. 256
198.38.114.0/24 Netflix Streaming Services Inc. 256
198.38.115.0/24 Netflix Streaming Services Inc. 256
198.38.116.0/24 Netflix Streaming Services Inc. 256
198.38.117.0/24 Netflix Streaming Services Inc. 256
198.38.118.0/24 Netflix Streaming Services Inc. 256
198.38.119.0/24 Netflix Streaming Services Inc. 256
198.38.120.0/24 Netflix Streaming Services Inc. 256
198.38.121.0/24 Netflix Streaming Services Inc. 256
198.38.122.0/24 Netflix Streaming Services Inc. 256
198.38.123.0/24 Netflix Streaming Services Inc. 256
198.38.124.0/24 Netflix Streaming Services Inc. 256
198.38.125.0/24 Netflix Streaming Services Inc. 256
198.38.96.0/24 Netflix Streaming Services Inc. 256
198.38.97.0/24 Netflix Streaming Services Inc. 256
198.38.98.0/24 Netflix Streaming Services Inc. 256
198.38.99.0/24 Netflix Streaming Services Inc. 256
198.45.48.0/24 Netflix Streaming Services Inc. 256
198.45.49.0/24 Netflix Streaming Services Inc. 256
198.45.52.0/24 Netflix Streaming Services Inc. 256
198.45.53.0/24 Netflix Streaming Services Inc. 256
198.45.54.0/24 Netflix Streaming Services Inc. 256
198.45.55.0/24 Netflix Streaming Services Inc. 256
198.45.56.0/24 Netflix Streaming Services Inc. 256
198.45.57.0/24 Netflix Streaming Services Inc. 256
198.45.58.0/24 Netflix Streaming Services Inc. 256
198.45.61.0/24 Netflix Streaming Services Inc. 256
198.45.62.0/24 Netflix Streaming Services Inc. 256
198.45.63.0/24 Netflix Streaming Services Inc. 256
208.75.77.0/24 Netflix Streaming Services Inc. 256
23.246.10.0/24 Netflix Streaming Services Inc. 256
23.246.11.0/24 Netflix Streaming Services Inc. 256
23.246.12.0/24 Netflix Streaming Services Inc. 256
23.246.13.0/24 Netflix Streaming Services Inc. 256
23.246.14.0/24 Netflix Streaming Services Inc. 256
23.246.15.0/24 Netflix Streaming Services Inc. 256
23.246.16.0/24 Netflix Streaming Services Inc. 256
23.246.17.0/24 Netflix Streaming Services Inc. 256
23.246.18.0/24 Netflix Streaming Services Inc. 256
23.246.20.0/24 Netflix Streaming Services Inc. 256
23.246.2.0/24 Netflix Streaming Services Inc. 256
23.246.22.0/24 Netflix Streaming Services Inc. 256
23.246.23.0/24 Netflix Streaming Services Inc. 256
23.246.24.0/24 Netflix Streaming Services Inc. 256
23.246.25.0/24 Netflix Streaming Services Inc. 256
23.246.26.0/24 Netflix Streaming Services Inc. 256
23.246.27.0/24 Netflix Streaming Services Inc. 256
23.246.28.0/22 Netflix Streaming Services Inc. 1,024
23.246.28.0/24 Netflix Streaming Services Inc. 256
23.246.29.0/24 Netflix Streaming Services Inc. 256
23.246.30.0/24 Netflix Streaming Services Inc. 256
23.246.3.0/24 Netflix Streaming Services Inc. 256
23.246.31.0/24 Netflix Streaming Services Inc. 256
23.246.36.0/24 Netflix Streaming Services Inc. 256
23.246.37.0/24 Netflix Streaming Services Inc. 256
23.246.4.0/24 Netflix Streaming Services Inc. 256
23.246.5.0/24 Netflix Streaming Services Inc. 256
23.246.58.0/24 Netflix Streaming Services Inc. 256
23.246.59.0/24 Netflix Streaming Services Inc. 256
23.246.6.0/24 Netflix Streaming Services Inc. 256
23.246.62.0/24 Netflix Streaming Services Inc. 256
23.246.63.0/24 Netflix Streaming Services Inc. 256
23.246.7.0/24 Netflix Streaming Services Inc. 256
23.246.8.0/24 Netflix Streaming Services Inc. 256
23.246.9.0/24 Netflix Streaming Services Inc. 256
37.77.184.0/24 Netflix Streaming Services Inc. 256
37.77.185.0/24 Netflix Streaming Services Inc. 256
37.77.186.0/24 Netflix Streaming Services Inc. 256
37.77.187.0/24 Netflix Streaming Services Inc. 256
37.77.188.0/24 Netflix Streaming Services Inc. 256
37.77.189.0/24 Netflix Streaming Services Inc. 256
37.77.190.0/24 Netflix Streaming Services Inc. 256
37.77.191.0/24 Netflix Streaming Services Inc. 256
64.120.128.0/17 Netflix Streaming Services Inc. 32,768
66.197.128.0/17 Netflix Streaming Services Inc. 32,768
69.53.224.0/24 Netflix Streaming Services Inc. 256
69.53.225.0/24 Netflix Streaming Services Inc. 256
69.53.226.0/24 Netflix Streaming Services Inc. 256
69.53.229.0/24 Netflix Streaming Services Inc. 256
69.53.231.0/24 Netflix Streaming Services Inc. 256
69.53.234.0/24 Netflix Streaming Services Inc. 256
69.53.236.0/24 Netflix Streaming Services Inc. 256
69.53.237.0/24 Netflix Streaming Services Inc. 256
69.53.238.0/24 Netflix Streaming Services Inc. 256
69.53.249.0/24 Netflix Streaming Services Inc. 256
69.53.255.0/24 Netflix Streaming Services Inc. 256

If delete all but the IP address and past them into that box, squid guard errors out when i try to save it. If i past the Netflix in there is accepts that but i have no idea if that works since the site shows the service have different names.

https://ipinfo.io/AS2906

I have also been looking for amazon prime streaming service IP but unable to locate those. Additionally, I need the same for PlayStation network as well.

Guest

You can also be inserting the domain only and then I think all IPs that will be used by this domain name
will also not to be cached! Would be much easier as millions of IP addresses to insert in. Like this;

https://openconnect.netflix.com
https://www.netflix.com/
bgp.he.net/AS2906

Guest

Should I select null for caching? How do I make this allow PSN traffic?

Guest

How do I make this allow PSN traffic?

Perhaps sniffing with WireShark in your network which IP addresses will be used for this
if a game is started and then you could enter those IP addresses or you find out the domain
from the PSN.

Guest

Been witnessing some strange behavior everyday for the last few days. Basically, squid and squid guard service stops for some unknown reason. Messing around with the setting doesn't seem to make it start and stay started. The first day it happened I was scratching my head for over an our, messing with stuff, I decided to click on update blacklist and then both services restarted on there own and will stay started till the next day. What is the cause of this? How do I fix it?

Guest

Another thing I am noticing. I installed lightsquid to mess around with it. I have configured wpad as mentioned earlier but when I open the proxy report for lightsquid, my computers don't show up in the real time report. If I I set the proxy setting in my browser though, my pc pops up right away in the proxy report. Why is this? Is my wpad configured correctly or is this normal behavior?

Guest

I found this which concerns auto updating the blacklists. Can you explain exactly how to do this?

https://forum.pfsense.org/index.php?topic=35479.0

It is from 2011 and I am not sure if it is relevant anymore.