PfSense 2.0.1-RELEASE with Squid & HAVP = major problems

[pftdm007]

Hi.

I always had major problems with pfsense when I tried to use packages on top of it.  pfSense on its own is great, but at the moment I install the packages I need to use, all go wrong..  I followed dozens of configuration tutorials, guides and instructions, to no avail..  I always end up with a semi-functional machine, and when I'm not so lucky, it simply doesn't work.

So here are the steps I did to install pfSense on this machine.

2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:43:51 EST 2011 

1-Downloaded the USB image (pfSense-memstick-2.0-RELEASE-amd64.img.gz)  CRC check OK
2-Extracted the image
3-dd 'd to my USB stick (dd if=pfSense-memstick-2.0-RELEASE-amd64.img.gz of=/dev/sdX where X is my USB stick)
4-Boot the machine with the stick.

Then I install pfsense using the custom/advanced install mode, I specify 3 partitions as follows (on a 160GB HDD):

1- /  *
2- /var  32000M
3- SWAP 32000M

Then pfSense reboots and start for the 1st time.  No problem so far.  I configure the LAN & WAN interfaces, all is great.  When everything is setup and running properly, I start the installation of the following packages:

vnstat2 / CRON / TFTP / ntop / snort / squid / squidGuard / HAVP

The installation goes well for each packages..  No errors reported by the installer.

I configure the packages without problems.  The configuration is simple, straight forward and easy for all packages, except I am experiencing major dysfunction with the machine.

Some of the problems:

Repetitive error message in the system logs saying:

Dec 21 23:38:55 	havp[20495]: connect() failed: Operation not permitted
Dec 21 23:38:54 	havp[20495]: connect() failed: Operation not permitted

Repetitive error page while browsing the web saying:

HAVP

The following server is down:
Connection failed

Strangely, pfsense.org will 100% trigger these two problems, making the issue even worst because I cant post here to ask for support..  On other sites, its random, it may crash, it may not..

I tried to uninstall & reinstall the packages, but I cant.  At first when I click on the remove button in the package tab, the (un)installer says that some include files are missing and the operation FAILED.  Then, I lose the version column in the package page, and the Repository page is empty with pfsense saying:

"Unable to communicate with www.pfsense.org" or something similar.  At this moment the system logs has:

/pkg_mgr_installed.php: XMLRPC communication error: Operation not permitted

Reinstalling the packages also fails..  Only a reboot will help and if lucky I will be able to reinstall the packages without apparent errors..  Not saying there is NO errors but I am not aware of.  Neither the system logs is..

I am clueless as to which package(s) causes these problems, but I am 99.9% sure this is the packages since I used pfsense without packages for more than a year and never had a problem.  Since I started using the packages, I have been having major problems for months now..  I have been trying to convince myself that I was not properly configuring the system, but no.  I am pretty sure there is bugs of some sort in the packages.  I have googled every single issue I have, and found at least one thread either here on pfsense.org or somewhere else on the web where someone had the same problem, and either the problem went away on its own (!??) or they simply stopped using the package(s).  I haven't found a solution(s) that worked yet.

By the way, the machine is "clean" i.e. RAM tested for 18 hours, no errors, CPU stressed, no errors, and detailed HDD surface test, no bad sectors.

These are the packages I currently have installed:

Cron  0.1.5
File Manager  0.1.1
HAVP antivirus	0.91
ntop  4.0.1_1 v2
snort 2.9.1 pkg v. 2.0.2
squid  2.7.9_4.2
squidGuard  1.3_1 pkg v.1.9.1
vnstat2  1.10_2 

My system is configured as follows:

Squid
Proxy interface: LAN
Allow users on interface: CHECKED
Transparent proxy" CHECKED
Log store directory: /var/log/squid
Log rotate: 7
Proxy port: 3128
What to do with requests that have whitespace characters in the URI: strip
Custom options (automatically added by SG):

never_direct allow all;cache_peer 127.0.0.1 parent 3125 0 name=havp no-query no-digest no-netdb-exchange default;redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf;redirector_bypass on;redirect_children 3

Hard disk cache size: 8000
Hard disk cache system: aufs
Hard disk cache location: /var/squid/cache
Memory cache size: 128
Level 1 subdirectories: 16
Memory replacement policy: Heap LFUDA
Cache replacement policy: Heap LFUDA
Allowed subnets: 192.168.0.100/24

SquidGuard

Enable: CHECKED
Enable GUI log: CHECKED
Enable log: CHECKED
Enable log rotation: CHECKED
Blacklist: CHECKED
Blacklist URL: http://www.shallalist.de/Downloads/shallalist.tar.gz
Target Rules: <bunch of="" stuff="" selected="" there..="">Proxy Denied Error: http://192.168.0.101/netserver-blocked.html
Redirect mode: ext url err page
Redirect info: http://192.168.0.101/netserver-blocked.html
Log: CHECKED

Groups ACL:
Name: blocked
Client (source): 192.168.0.100/24
Target Rules: <bunch of="" stuff="" selected="" there..="">Redirect mode: ext url err page
Redirect: http://192.168.0.101/netserver-blocked.html
Description: blocked
Log: checked

Target categories:
Name: blocked
Domain list: partypoker.com bing.com
Redirect mode: ext url err page
Redirect: http://192.168.0.101/netserver-blocked.html
Description: blocked

Blacklist:
Blacklist update: http://www.shallalist.de/Downloads/shallalist.tar.gz

HAVP

Http proxy:
Enable: CHECKED
Proxy mode: Parent for Squid
Proxy interface(s): LAN
Proxy port: 3125
Enable RAM Disk: CHECKED
Scan max file size: 5000k
Log: CHECKED
Syslog: CHECKED

Settings:
AV base update: every 6 hours
Log: CHECKED
Syslog: CHECKED

Not sure what else to add …  Anybody can guide step by step in troubleshooting my pfsense install and making it better? (or usable)?
I appreciate any help.

Thanks!</bunch></bunch>

[marcelloc]

I suggest you going package by package.

First a clean install and then, only hapv.

After stress test, include squid and go on until you find where it stops.

Did you found any docs about squid + squidguard + hapv together?

[pftdm007]

Fresh install (once again  :'( ) and already I have problems:

Installing HAVP, I get :

Beginning package installation for HAVP antivirus...
Downloading package configuration file... done.
Saving updated package information... done.
Downloading HAVP antivirus and its dependencies... 
Checking for package installation... 
 Downloading http://files.pfsense.org/packages/amd64/8/All/havp-0.91_1.tbz ...  could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/havp-0.91_1.tbz.
of havp-0.91_1 failed!

Installation aborted.Backing up libraries... 
Removing package...
Starting package deletion for havp-0.91_1...done.
Removing HAVP antivirus components...
Tabs items... done.
Menu items... done.
Services... done.
Loading package instructions...
Include file havp.inc could not be found for inclusion.
Deinstall commands... 
Not executing custom deinstall hook because an include is missing.
Removing package instructions...done.
Auxiliary files... done.
Package XML... done.
Configuration... done.
Cleaning up... Failed to install package.

Installation halted.

Also,  under Firmware -> Auto Update, the webinterface says:

Downloading new version information...done
Unable to check for updates.
Could not contact custom update server.

Maybe the servers are down?

[marcelloc]

Check dns name resolution. I always first disable dns forwarder before anything else.

[marcelloc]

Or Maybe its down

http://forum.pfsense.org/index.php/topic,44242.msg229525.html#msg229525

[dindinudin]

i got same problem, after upgrade from pfSense 2.0.0 to 2.0.1. all my installation package is lost (squid ,squid guard, proxy report etc).
cannot be reinstalled, error message like posted by : lpallard
my upgrade file is pfSense-Full-Update-2.0.1-RELEASE-i386.tgz
is there something wrong with my upgrade ?

regards

[marcelloc]

Nothing wrong, just wait files.pfsense.org get back.

[pftdm007]

Still down this morning..  They dont have mirrors?

Anyways, all we can do for now is to wait.

[Supermule]

Its a problem if a site goes down and a lot of inst. fail or dont receive updates asf….

And people cant get on with installing everything. Is it possible to create an offsite line install where pacakages can be DL and installed from another location?

[pftdm007]

Still fail..

Ive just successfully updated to the latest version of pfsense

2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:43:51 EST 2011
FreeBSD 8.1-RELEASE-p6

but trying to install HAVP I still get:

Installation of HAVP antivirus FAILED!

Beginning package installation for HAVP antivirus...
Downloading package configuration file... done.
Saving updated package information... done.
Downloading HAVP antivirus and its dependencies... 
Checking for package installation... 
 Downloading http://files.pfsense.org/packages/amd64/8/All/havp-0.91_1.tbz ...  could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/havp-0.91_1.tbz.
of havp-0.91_1 failed!

Installation aborted.Backing up libraries... 
Removing package...
Starting package deletion for havp-0.91_1...done.
Removing HAVP antivirus components...
Tabs items... done.
Menu items... done.
Services... done.
Loading package instructions...
Include file havp.inc could not be found for inclusion.
Deinstall commands... 
Not executing custom deinstall hook because an include is missing.
Removing package instructions...done.
Auxiliary files... done.
Package XML... done.
Configuration... done.
Cleaning up... Failed to install package.

Installation halted.

What do you need to get this working?

[marcelloc]

Just wait http://files.pfsense.org get back.

[pftdm007]

Yeah that what I figured… its still down. Will wait.

But the pfsense devs should really consider a mirroring system of some sort..  Imagine if sites like kernel.org or the mirrors and repositories for the other major linux distros were doing this..

[sirconna]

Hello guys
I'm sure you're aware of all the problems.
I'm not new to pfsense, so it is my fault for not checking first, but I've been trying since 7am this morning to get my pfS box working. I wanted to upgrade to the new version, thought I'd do it today. Well…. surpirse, I couldn't even reinstall squid or anything. Eventually I thought I messed up somewhere and reinstsalled (stupid), at least i have backup from monday... but the problem is I'm now at home, waiting for the update site to get back online so that I can reinstall & config everything (downloading 2.0.1 right now, hoping) - I have about 200 user already configured for proxy, so I'm stressing, 'cos I can't figure out how to port forward 8080 proxy to 80 on the outside so that they can bypass proxy and I can't wait till next week to get everyone working again. Otherwise I have to change all of them users to get out bypassing proxy and change back later... I only saw some post about 2 hours after I reinstalled that the update/ downlload site is down... now I'm buggered. I learnt a lesson though, but it is freaking me out - gonna go for IPcop just to get everybody to stop complaining soon (11 pm here, work starts at 7 am) and then try to work around the prblem later on.... what happened; you get hacked or some serious power outage or what... hope you get all running again SOON. When abouts can we expect all to 2 b normal again - please guys, I'm not gonna sleep until I know my customer can do online baning and mailing tomorrow morning... I relaised it wasn't my config or your product, just not thinking.... shoulda left all as is, but I'm stufft as we sit here... please tell us how long till we can get going again...
Thanks for a greatest product this side of the world and for your dedication and work, all of you... you don't understand - africa is different and not as hot as you are in the US or EU... I love pfS and I actively try to compete with MS ISA / Foreforont over here, and doing a damn good job of it... :-)

[marcelloc]

You can try to install freebsd 8.1 squid package and configure it by hand until files.pfsense get back.

[marcelloc]

Follow this topic

http://forum.pfsense.org/index.php/topic,44242.msg229815.html#msg229815

[wagonza]

As per cmb, files.pfsense.org is back and it won't happen again.

[pftdm007]

OK its been a few days now that things were not too bad, but now: BOOM again!  pfsense is acting again….  Right now, it refuses to deliver http://forum.xbmc.org/ and I get an error page like:

HAVP - DNS error

HAVP

A DNS error occurred
while opening the page

forum.xbmc.org
Please contact your tech support

Why?  I really dont know.  This morning I accessed this forum without problems.  Snort does not block anything the blocked list is empty.  Squidguard's is deactivated

Im really tired of the randomness…  It works now, 5 minutes later, it no longer works.. Why?  I bet even God doesnt know.

Here are the problems, on top of this thread (and my 10 other threads on this forum):

Problems accessing youtube (buffing for 15-30 sec every minute or so while playing, plus takes about 3 to 5 minutes to start the video playback)
random websites stops going thru pfsense (forum.xbmc.org, www.mls.ca, this forum also stopped working at some point)...
Firefox behaves very strangely (like right now, it permanently says "Transferring data from forum.pfsense.org..." in the corner, and the "wheel contonuously spins in the page tab)

I dont know how many times I mentioned this, but pfsense IS the problem.  I plug my laptop directly to the cable modem, and BANG it works..!
What will it be next?

I am getting to miss my stupid D-Link router..  Anybody cares to step forward and help me before I abandon pfsense?  I really believes in the project, but to be honest, if I had a good run at it I would be more confidant to use it….

[marcelloc]

I think the best way now is to buy few paid support hours or contact havp package maintainer.

I do not have this problems but I do not use havp.

Dansguardian package with antivirus is under devel. Maybe when It's done you will have no need to use havp.

[marcelloc]

Do you have antivirus on your dlink?

If your problem is with havp, why not just disable it and use only squid+squidguard?

It will do more then dlink.

[pftdm007]

What puzzles me is that I seem to be the only one with such problems…  Am I or its that nobody cares (except you of course)?

Hell, I thought some websites actually banned my IP since ive been several weeks without being able to access them... Going to the cable modem directly solved it.  Browsing feels also much much much snappier...