PfSense crashes after installing squid & squidGuard

jmarquez

Hi there.

First thing first, thank you for this great peace of code.

I am expericencing the following problem:
We are running pfsense 2.0 on a box with 4 intel cards (Intel(R) PRO/1000 GT Desktop Aapter). Every looks fine and the system works great using IPSec tunneling, DNS Forwarder, DHCP Server and PPTP Server. So far so good.

Once we are happy with the settings for the above services (~5 days using the system as described), whenever we install Squid & SquidGuard from the Packages menu, pfSense just
hangs. It might be half and hour later or maybe the following morning. The systems becomes unusable (in fact we rather install from scratch)

We have changed Intel NIC's for Realtek NIC's for the sake of testing but it crashes again.

Please find below any info you might find useful. Please let me know if more info is needed.
Any help appreciated.

NIC's (dmesg)
–---
Nov 9 12:04:56 pfSense-naves kernel: em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.3="">port 0xd080-0xd0bf mem 0xfe300000-0xfe31ffff,0xfe2e0000-0xfe2fffff irq 16 at device 0.0 on pci6
Nov 9 12:04:56 pfSense-naves kernel: em0: [FILTER]
Nov 9 12:04:56 pfSense-naves kernel: em1: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.3="">port 0xd040-0xd07f mem 0xfe2a0000-0xfe2bffff,0xfe280000-0xfe29ffff irq 17 at device 1.0 on pci6
Nov 9 12:04:56 pfSense-naves kernel: em1: [FILTER]
Nov 9 12:04:56 pfSense-naves kernel: em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.3="">port 0xd000-0xd03f mem 0xfe240000-0xfe25ffff,0xfe220000-0xfe23ffff irq 18 at device 2.0 on pci6
Nov 9 12:04:56 pfSense-naves kernel: em2: [FILTER]

$ uname -a
FreeBSD pfSense-naves.oficina 8.1-RELEASE-p4 FreeBSD 8.1-RELEASE-p4 #0: Tue Sep 13 16:58:57 EDT 2011 root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 i386

log entries after rebooting pfSense after crash
–------------------------------------------
Nov 9 09:11:47 squid[36353]: Squid Parent: child process 37084 started
Nov 9 09:11:47 check_reload_status: Reloading filter
Nov 9 09:11:47 php: : Not calling package sync code for dependency squidcache of squid because some include files are missing.
Nov 9 09:11:47 php: : Not calling package sync code for dependency squidnac of squid because some include files are missing.
Nov 9 09:11:47 php: : Not calling package sync code for dependency squid of squid because some include files are missing.
Nov 9 09:11:47 php: : Not calling package sync code for dependency squidtraffic of squid because some include files are missing.
Nov 9 09:11:47 php: : Not calling package sync code for dependency squidupstream of squid because some include files are missing.
Nov 9 09:11:47 php: : Not calling package sync code for dependency squidauth of squid because some include files are missing.
Nov 9 09:11:47 php: : Not calling package sync code for dependency squidusers of squid because some include files are missing.
Nov 9 09:11:47 php: : Not calling package sync code for dependency squidcache of squid because some include files are missing.
Nov 9 09:11:49 check_reload_status: Syncing firewall
Nov 9 09:11:49 php: : Reloading Squid for configuration sync
Nov 9 09:11:49 check_reload_status: Reloading filter
Nov 9 09:11:49 squid[37084]: The url_rewriter helpers are crashing too rapidly, need help!
Nov 9 09:11:49 squid[36353]: Squid Parent: child process 37084 exited due to signal 6
Nov 9 09:11:49 kernel: pid 37084 (squid), uid 62: exited on signal 6
Nov 9 09:11:51 php: : IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
Nov 9 09:11:52 login: login on ttyv0 as root
Nov 9 09:11:52 sshlockout[54338]: sshlockout/webConfigurator v3.0 starting up
Nov 9 09:11:56 Squid_Alarm[62725]: Squid has exited. Reconfiguring filter.
Nov 9 09:11:56 Squid_Alarm[62944]: Attempting restart…
Nov 9 09:11:56 squid[63977]: Squid Parent: child process 666 started
Nov 9 09:11:59 Squid_Alarm[2051]: Reconfiguring filter…
Nov 9 09:12:40 apinger: Error while feeding rrdtool: Broken pipe
Nov 9 09:12:40 apinger: /usr/local/bin/rrdtool respawning too fast, waiting 300s.
Nov 9 09:13:17 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:14:21 check_reload_status: Reloading filter
Nov 9 09:15:53 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:18:00 check_reload_status: Syncing firewall
Nov 9 09:18:00 dnsmasq[42593]: read /etc/hosts - 25 addresses
Nov 9 09:18:07 Squid_Alarm[12784]: Squid has resumed. Reconfiguring filter.
Nov 9 09:18:19 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:19:19 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:19:38 check_reload_status: Linkup starting re1
Nov 9 09:19:38 kernel: re1: link state changed to UP
Nov 9 09:19:43 php: : Hotplug event detected for lan but ignoring since interface is configured with static IP (172.24.24.252)
Nov 9 09:19:43 check_reload_status: rc.newwanip starting re1
Nov 9 09:19:48 php: : rc.newwanip: Informational is starting re1.
Nov 9 09:19:48 php: : rc.newwanip: on (IP address: 172.24.24.252) (interface: lan) (real interface: re1).
Nov 9 09:19:48 apinger: Exiting on signal 15.
Nov 9 09:19:49 check_reload_status: Reloading filter
Nov 9 09:19:49 apinger: Starting Alarm Pinger, apinger(16100)
Nov 9 09:19:54 php: : Forcefully reloading IPsec racoon daemon
Nov 9 09:19:54 dnsmasq[42593]: reading /etc/resolv.conf
Nov 9 09:19:54 dnsmasq[42593]: using nameserver 80.58.61.254#53
Nov 9 09:19:54 dnsmasq[42593]: using nameserver 80.58.61.250#53
Nov 9 09:19:54 dnsmasq[42593]: ignoring nameserver 127.0.0.1 - local interface
Nov 9 09:19:54 dnsmasq[42593]: ignoring nameserver 127.0.0.1 - local interface
Nov 9 09:19:56 php: /index.php: Successful webConfigurator login for user 'admin' from 172.24.3.51
Nov 9 09:19:56 php: /index.php: Successful webConfigurator login for user 'admin' from 172.24.3.51

Uninstalling SquidGuard from package menu
–---------------------------------------
Backing up libraries...
Removing package...
Starting package deletion for squidGuard-1.4_3...done.
Skipping package deletion for openldap-client-2.4 because it is a dependency.
Skipping package deletion for perl-5.12 because it is a dependency.
Starting package deletion for squid-2.7...done.
Starting package deletion for cyrus-sasl-2.1...done.
Starting package deletion for openldap-client-2.4...done.
Starting package deletion for perl-5.12...done.
Starting package deletion for db42-4.2...done.
Starting package deletion for db41-4.1.25_4...done.
Starting package deletion for db3-3.3.11_3,1...done.
Starting package deletion for cyrus-sasl-2.1.23_3...done.
Removing squidGuard components...
Tabs items... done.
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.
Removing package instructions...done.
Auxiliary files... done.
Package XML... done.
Configuration... done.
Cleaning up...
Package deleted.

Uninstalling Squid from package menu

Backing up libraries...
Removing package...
Starting package deletion for squid-2.7.9_1...done.
Starting package deletion for squid_radius_auth-1.10...done.
Starting package deletion for libwww-5.4.0_4...done.
Removing squid components...
Tabs items... done.
Menu items... done.
Services... done.
Loading package instructions...
Deinstall commands... done.
Removing package instructions...done.
Auxiliary files... done.
Package XML... done.
Configuration... done.
Cleaning up...
Package deleted.

log entries after uninstalling squid & squidGuard

-- After deleteting Squid & squidGuard
Nov 9 09:20:26 php: : SQUID is installed but not started. Not installing "pfearly" rules.
Nov 9 09:20:30 Squid_Alarm[23841]: Squid has exited. Reconfiguring filter.
Nov 9 09:20:30 Squid_Alarm[24261]: Attempting restart…
Nov 9 09:20:30 squid[25158]: Squid Parent: child process 25610 started
Nov 9 09:20:33 Squid_Alarm[29978]: Reconfiguring filter…
Nov 9 09:20:43 Squid_Alarm[51620]: Squid has resumed. Reconfiguring filter.
Nov 9 09:20:49 apinger: Error while feeding rrdtool: Broken pipe
Nov 9 09:20:49 apinger: /usr/local/bin/rrdtool respawning too fast, waiting 300s.
Nov 9 09:21:00 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:21:13 check_reload_status: Syncing firewall
Nov 9 09:21:29 squid[25610]: The url_rewriter helpers are crashing too rapidly, need help!
Nov 9 09:21:29 squid[25158]: Squid Parent: child process 25610 exited due to signal 6
Nov 9 09:21:29 kernel: pid 25610 (squid), uid 62: exited on signal 6
Nov 9 09:21:29 check_reload_status: Syncing firewall
Nov 9 09:21:29 check_reload_status: Syncing firewall
Nov 9 09:21:29 php: /pkg_mgr_install.php: Reloading Squid for configuration sync
Nov 9 09:21:29 php: /pkg_mgr_install.php: The command '/usr/local/sbin/squid -k reconfigure' returned exit code '127', the output was '/usr/local/sbin/squid: not found'
Nov 9 09:21:29 check_reload_status: Reloading filter
Nov 9 09:21:30 check_reload_status: Reloading filter
Nov 9 09:21:32 squid[25158]: Squid Parent: child process 43989 started
Nov 9 09:21:32 squid[43989]: execvp failed: (2) No such file or directory
Nov 9 09:21:32 squid[25158]: Squid Parent: child process 43989 exited with status 1
Nov 9 09:21:35 squid[25158]: Squid Parent: child process 51902 started
Nov 9 09:21:35 squid[51902]: execvp failed: (2) No such file or directory
Nov 9 09:21:35 squid[25158]: Squid Parent: child process 51902 exited with status 1
Nov 9 09:21:38 squid[25158]: Squid Parent: child process 52322 started
Nov 9 09:21:38 squid[52322]: execvp failed: (2) No such file or directory
Nov 9 09:21:38 squid[25158]: Squid Parent: child process 52322 exited with status 1
Nov 9 09:21:41 squid[25158]: Squid Parent: child process 63077 started
Nov 9 09:21:41 squid[63077]: execvp failed: (2) No such file or directory
Nov 9 09:21:41 squid[25158]: Squid Parent: child process 63077 exited with status 1
Nov 9 09:21:44 squid[25158]: Squid Parent: child process 3829 started
Nov 9 09:21:44 squid[3829]: execvp failed: (2) No such file or directory
Nov 9 09:21:44 squid[25158]: Squid Parent: child process 3829 exited with status 1
Nov 9 09:21:44 squid[25158]: Exiting due to repeated, frequent failures
Nov 9 09:21:45 php: : SQUID is installed but not started. Not installing "filter" rules.
Nov 9 09:21:46 check_reload_status: Syncing firewall
Nov 9 09:21:48 check_reload_status: Syncing firewall
Nov 9 09:21:48 php: /pkg_mgr_install.php: SQUID is installed but not started. Not installing "nat" rules.
Nov 9 09:21:48 php: /pkg_mgr_install.php: SQUID is installed but not started. Not installing "pfearly" rules.
Nov 9 09:21:53 php: /pkg_mgr_install.php: SQUID is installed but not started. Not installing "filter" rules.
Nov 9 09:21:54 check_reload_status: Reloading filter
Nov 9 09:22:00 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:23:00 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:24:00 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:25:00 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:25:08 check_reload_status: Syncing firewall
Nov 9 09:26:00 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:27:00 filterdns: host_dns: failed looking up "em-arabia.dyndns.tv": hostname nor servname provided, or not known
Nov 9 09:27:23 check_reload_status: Syncing firewall</intel(r)></intel(r)></intel(r)>

jmarquez

Hi there.

I finally figured it out what could be the problem.
The only change I have made to the current configuration was a BIOS update.
After having a look at the mainboard manufacturer webpage, they recommended an update for
a better system stability. Also, pfSense book recommends BIOS updates when available.
After applying the BIOS update the error dissapeared (fingers crossed).

We are still keeping an eye on the system, just in case.

Hope this helps.

Regards,
Jesus

jmarquez

Hi there.

Even though I said in my previous post that the systems was back to work, I was wrong.
We have had lots o problems since then and I hope somebody can help me.

This is our current hardware/software configuration:

Mainboard: Asus P8P67LE (1 Realtek nic onboard).
Intel i7 processor.
4 Gb. RAM.
3 Intel PRO/1000 Desktop PCI Network cards (we have tested 3 Realtek NIC's too).
Raid 1 with 2 10K RPM SATA HD's (also tested with no RAID, just 1 SATA 500 Mb HD 7200 RPM)
pfSende 2.0.1 (also tested pfSense 2.0.0)

I can attach the config.xml file if it helps. The config.xml file is also used in a pfSense session running on a virtualbox machine.
It's been working so far with no problems at all.
We can swap config.xml file between both, the virtual pfSense and the real one, but the only one that has been stable so far is the one running on a virtualbox machine.

The one running on hardware has only been working for 5 days maximun before watchdog messages appear on the terminal screen and the system hangs.
Please find attached a screenshot of the last messages before it hangs.

Any help would be appreciated.
Regards,
Jesus

P1020042.JPG_thumb

marcelloc

Did you tried both 32 and 64 bits version?

jmarquez

Hi,

Do you mean 32/64 bits version of pfSense?

As far as I know there is only a 32 bits version of pfSense. Am I right?

Having a look at the "which version to use" of pfSense, it states that it needs a compatible processor. I don't know whether this means that it has to be an AMD processor or any 64bits processor (AMD/Intel).

Thank you very much.
Jesus

marcelloc

@jmarquez:

As far as I know there is only a 32 bits version of pfSense. Am I right?

There is both since pfSense 2.0

@jmarquez:

Having a look at the "which version to use" of pfSense, it states that it needs a compatible processor. I don't know whether this means that it has to be an AMD processor or any 64bits processor (AMD/Intel).

If your hardware is not new, then probably it will accept only 32 bit version.

jmarquez

Thank you very much.

I have just installed the 64 bits version.
Let's see if this version fixes the problem. I'll come back to the forum with news about this issue.
It usually takes 4-5 days to hang.

Regards,
Jesus

jmarquez

Hi everyone.

I had no luck on this. The system crashed again in less than one day using the 64 bit version of pfsense.
So far I don't have a clue about what's going on.
I have changed every single peace of hardware (except the mainboard & processor), have run memory test,
changed HD, change Intel NIC's for Realtek ones and reinstalled pfSense 32/64 bits from scratch.
The fact is that a virtualbox machine is working great using the same config file.

Reading a few posts in the Hardware area of this forum, it is suggested that some hardware might be trying
to set the system on a sleep mode so it might crash pfSense as this is a wrong state for the system.

Is there a way to check this?

Any idea appreciated.
Thank you very much.
Jesus

marcelloc

Disable any sleep time for all option in bios power manager.

jmarquez

I found an entry in the BIOS named "EPU Power Saving Mode". The mainboard manual states that it is disabled by default, but I found it Enabled.
I have disabled it. Let's see what happens.

Thank you very much.

fr3drik

i thinks thats a bug .
something goes wrong on pfsense

jmarquez

Hi there.

I'd like to paste the following entries from dmesg:

ACPI Error (psargs-0464): [RAMB] Namespace lookup failure, AE_NOT_FOUND
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20100331/nsinit-442)

Is this something to look at?

Besides, we are running pfSene 2.0 x32 on a virtualbox machine. This machine shows a line that doesn't show in the
one failing. The line is as follow:

acpi0: Sleep Button (fixed)

Any help appreciated.
Regards,
Jesus.

packeteer

Rather than spending your time on trying to find the cause. You're better off building a dedicated standalone squid box with Dansguardian.

I stay away from all in one solution.