It says right on the main page that the list is repackaged each night.  When you open up the archive, all files and folders have a timestamp.

Nope, that's the place.  Glad to hear you got it working.

@Xeboc:

@doktornotor - I've been using pfSense for a month now, trying out the packages, reading the forums, and testing everything out.  Your information above about those packages is valuable and useful.  But how would a new person coming to this software know any of that?  Most of the useful tidbits of info I've read come from you or a few other dedicated programmers buried in a forum post somewhere.  A few packages are useful and well maintained, but, as you indicated, some are really broken and unmaintained.  Can we get a 'date last updated' added to the package manager?  Or some other indicator of what you have shared?  I like pfSense, but I've been hacking away at it for hours and combing forums for months to learn anything useful….

When I tried to mark Squid 2.7 unsupported in the package list, the PR was closed by pfSense devs telling me that Squid 2.7 is one of the few "officially supported" packages. Never mind noone's maintaining and fixing anything there and that package is buggy like hell and abandoned everywhere. There. Don't get me started with that again. The 2.7 zombie thing is gone from pfSense 2.3, thanks god. Dansguardian is gone as well, the E2G did not get anywhere last time I checked and will need bunch of fixes for 2.3 anyway. Squidguard is still there and is still broken and I still get severe headaches when I look at the code, cannot see myself fixing it anytime soon. Rewriting from scratch would probably be easier.

As for "last updated", you can see that on Github.

on webgui proxy server tab setting and you found name "SSL man in the middle Filtering"  ;)

@rocketdog:

Edit: And how do I get rid of the local hits on "Real Time"? I have added the WAN and LAN IP at "Do not cache", but it still floods 'squid_monitor_data.php'

I was able to stop this by adding a proxy exception for the firewall IP on the windows computer I'm using.
(Internet Properties -> Connections -> LAN settings -> Advanced)

As for no HITS, I found that binding squid to localhost caused it to MISS everything.  Removing the localhost binding caused squid to start functioning correctly again.  No idea why….

I also found that squid didn't like to use the disk cache at all until it was rebuilt.  I probably re-booted the router while the initial creating of directories was happening...  I used:

squid -k shutdown squid -z -S

Thanks torsurfer for pointing this out

I am using apache to serve the config file but I will move this to an IIS server and report back

Thanks for the suggestion KOM.  I used the wpad link you sent and am dropping the transparent proxy.  I guess I am stuck in the past - had that set up using centos but lost the hard drive and was trying to recreate it on pfsense.  This is a better solution.  I will monitor it and make certain that it works as expected.

Thanks for the help doktornotor as well - when I have time I will try the sniffer.

@Blade1:

What am I doing wrong?

Trying to reach a machine from LAN using WAN IP. Sorry but meaningful reverse proxy testing cannot be done from LAN. Unless you have a separate DMZ interface and RP set up on a different LAN interface, this just won't work.

When you get one of these errors, take a look in /var/squid/logs/access.log and see what it says about that URL.

Very strange.  Did you change the defaults in any meaningful way?

Unless you have no other antivirus solution where you are, I would highly recommend getting rid of ClamAV.  It slows everything down.  I don't have an immediate answer to your problem.  ANything of interest in either /var/squid/logs/access.log or cache.log?  If nothing, you could try increasing the debug level by putting this into squid's Integrations section:

debug_options rotate=1 ALL,2 11,5

Ok. Good news. I have had positive results here. Sarg is compiling reports for four days now and real-time has not crashed either. So if you are having any problems here is what I have done to resolve reporting issues. This is on a 64-bit system.

This is for a transparent proxy with a port foward redirect to dansguardian. My Sarg config pulls its' reports from the dansguardian access log. Not the squid log. I do not know of any issues with any different proxy and Sarg config.

1. Navigate to /var/log/dansguardian and delete all access logs in this directory.

2. Create a schedule in Sarg to get all necessary directories and access logs created. Force a report update and save it. Navigate to var/log/dansguardian. Here you need to see access.log.
Now navigate to usr/pbi/sarg-amd64/local. Here you need to see the sarg-reports directory. You may need to restart the firewall as I did. When you have all directories and files proceed to
the next step.

3. Run these commands.
rm -rf /usr/local/sarg-reports
ln -s /usr/pbi/sarg-amd64/local/sarg-reports /usr/local/sarg-reports

Force a report update

Navigate to /usr/local/ you should see the linked sarg-reports directory. If it is a file and not a directory delete the file and run step 3 again. You should now have a directory and Sarg should
now be able to retrieve the reports.

HOLD ON! We're not done yet.

Now delete the schedule you created in Sarg. You will need to have cron installed for the next steps.

Create these jobs in the cron utility. The time can be of your preference. The who and command fields are most important here. Also I would keep the wildcards where they are.

minute hour mday month wday who command

0 */8 * * * root /usr/local/etc/rc.d/squid.sh
0 */8 * * * root /usr/pbi/sarg-amd64/bin/sarg

These two commands restart the proxy and run sarg reporting at the command line level. A cron command is created by the schedule you create in Sarg but from what I have gathered it is broken for some reason after a couple of days of reporting and the logs are not showing any errors related to this which is what confused me.

With what I have done here I have not had any issues with reporting or real-time crashing thus far.

Good Luck! Hope this helps you.

understood
clamav radomly stops working for me. makes it useless.

There shouldn't be any softlink to that anywhere (and for sure isn't created by adding a dummy target category). As said, it just doesn't make sense.

Ok, I dump some of the traffic and think I find the problem:

Client 1.1.1.1
Server 2.2.2.2

Here is the dump for a listing in passive mode:

17:51:56.209388 IP 1.1.1.1.34966 > 2.2.2.2.58523: Flags [s], seq 367546487, win 29200, options [mss 1460,sackOK,TS val 438097740 ecr 0,nop,wscale 7], length 0 17:51:56.228354 IP 2.2.2.2.38538 > 1.1.1.1.34966: Flags [S.], seq 3116512794, ack 367546488, win 5792, options [mss 1460,sackOK,TS val 811246550 ecr 438097740,nop,wscale 3], length 0 17:51:56.228379 IP 1.1.1.1.34966 > 2.2.2.2.38538: Flags [R], seq 367546488, win 0, length 0 17:51:57.207804 IP 1.1.1.1.34966 > 2.2.2.2.58523: Flags [s], seq 367546487, win 29200, options [mss 1460,sackOK,TS val 438097990 ecr 0,nop,wscale 7], length 0 17:51:57.216513 IP 2.2.2.2.38538 > 1.1.1.1.34966: Flags [S.], seq 3116512794, ack 367546488, win 5792, options [mss 1460,sackOK,TS val 811246650 ecr 438097740,nop,wscale 3], length 0 17:51:57.216579 IP 1.1.1.1.34966 > 2.2.2.2.38538: Flags [R], seq 367546488, win 0, length 0 17:51:59.211787 IP 1.1.1.1.34966 > 2.2.2.2.58523: Flags [s], seq 367546487, win 29200, options [mss 1460,sackOK,TS val 438098491 ecr 0,nop,wscale 7], length 0 17:51:59.240789 IP 2.2.2.2.38538 > 1.1.1.1.34966: Flags [S.], seq 3116512794, ack 367546488, win 5792, options [mss 1460,sackOK,TS val 811246850 ecr 438097740,nop,wscale 3], length 0 17:51:59.240815 IP 1.1.1.1.34966 > 2.2.2.2.38538: Flags [R], seq 367546488, win 0, length 0 17:51:59.812596 IP 2.2.2.2.38538 > 1.1.1.1.34966: Flags [S.], seq 3116512794, ack 367546488, win 5792, options [mss 1460,sackOK,TS val 811246910 ecr 438097740,nop,wscale 3], length 0 17:51:59.812651 IP 1.1.1.1.34966 > 2.2.2.2.38538: Flags [R], seq 367546488, win 0, length 0 17:52:03.215791 IP 1.1.1.1.34966 > 2.2.2.2.58523: Flags [s], seq 367546487, win 29200, options [mss 1460,sackOK,TS val 438099492 ecr 0,nop,wscale 7], length 0 17:52:03.224268 IP 2.2.2.2.38538 > 1.1.1.1.34966: Flags [S.], seq 3116512794, ack 367546488, win 5792, options [mss 1460,sackOK,TS val 811247251 ecr 438097740,nop,wscale 3], length 0 17:52:03.224288 IP 1.1.1.1.34966 > 2.2.2.2.38538: Flags [R], seq 367546488, win 0, length 0 17:52:05.812505 IP 2.2.2.2.38538 > 1.1.1.1.34966: Flags [S.], seq 3116512794, ack 367546488, win 5792, options [mss 1460,sackOK,TS val 811247510 ecr 438097740,nop,wscale 3], length 0 The client initiates the connection from port 34966 to 58523\. But the answer come from port 38538 and not port 58523\. So the client send a RST and retransmit.. A bug in the package? Any ideas? We really need a working ftp helper implementation for our customers :-/ Thx! [/s][/s][/s][/s]

Thanks, Doktornotor.

the process of removing the package from the web interface is actually removes all or reference files for future installations.

How can I do to remove the package from the command line?

Any help/thoughts/suggestions??

Not possible with just Squid/squidGuard.  If these people have the right to bypass restrictions when they feel like it, why are you restricting them in the first place?

But I do not have all day every day to sit and white list sites that our teaching staff need access to.

Sometimes this is the only way.  I can't believe it's that many sites, nor should they change that much that it would require you to spend a lot of time maintaining a whitelist.  If you use squidGuard with blacklists, you can check the URL against the blacklist to see which category it falls under (if any) and then allow just those users to access that category of sites.