Pfsense install spiraled out of control after upgrade to 2.0.3
-
I
think I mayhave found the problem.. Squid (surprise!! surprise!)Uninstalled it, and the HDD light went down right away. the router is also very responsive.
I installed it back, same thing. I just remembered that a while back, I had the same problem. Obviously there is a big problem with squid and 2.0.3
EDIT: I no longer doubt squid is the problem. I will try to find the posts I saw on this forum to see what was the resolution. If this fails, I am reverting to 2.0.1 cause so far I am extremely dissatisfied with 2.0.3.
-
Which Squid version were you running?
You seemed to have far too many instances of Squidguard running in the shot above.
Steve
-
Squid 2.7.9 pkg 4.3.3
Yeah when I saw the output of the command I noticed the number of instances and assuming it was the problem I removd it.
-
~~On a side note, I think I know what could have been wrong:
You see, I installed 2.0.3, then restored the config with the XML file I had exported when running 2.0.1 and the previous squid package (dont remember which version exactly but I know for sure it wasnt 2.7.9 pkg 4.3.3)
I noticed restoring the config reinstall the packages. Could it be that restoring my config reinstalled the former squid package on top (or replaced completely) the new 2.7.9 pkg 4.3.3?? If so, perhaps the previous squid package is incompatible with 2.0.3 hence causing this issue..
If its true, then perhaps this thread should be moved to the packages forum… Obviously the maintainers need to be made aware of this serious issue.~~
I no longer think this is true, see next post.
-
I just did a little bit of troubleshooting.
I reinstalled squid. On its own, nothing happens, no issues whatsoever. Package installs fast, router performance doesnt degrade, everything is normal.
Then I installed squidguard. Immediately after package installation, everything slowed to a crawl, the router almost wasnt responding at all.
I tried the top command again (that was suggested above by wallabybob) and I saw the exact same output, that is, 7 or 8 SquidGuard processes running simultaneously.
I uninstalled squidguard, things were much better (at least I could use the webinterface).
Then I uninstalled squid. Everything came back to normal.
For the sake of completeness, the package version for SG was:
Beta 1.4_4 pkg v.1.9.4 platform: 1.1 -
No one has an idea of what happens with squid?
why is squidGuard running several instances at the same time??
-
Hmmm ok its working now! What I've done differently this time is reboot the pfsense box after squid's installation and then installed SG.
The machine is lightning fast now.
Who knows what may have happened..?! :-\
-
Unfortunately Im back with some interesting info. Im still having this issue, and now I can relate the issue with a specific event: utility power loss. I also found the "culprit": squidGuard.
Basically, my pfsense box is connected to a reliable commercial UPS while the cable modem is NOT (located elsewhere in the house).
During the last few days, we had a lot of thunderstorms and frequent (up to 2X per day) power utility loss or auto recloses (thats when the utility drops and returns very quickly) which more or less resets everything in the house including the router.
I am not sure why, perhaps the fact that the modem has been reset, but right after the event, the pfsense box becomes extraordinary slow as described earlier in this thread. If I manage (yes its almost impossible) to see the output of top, I will see dozens of squidGuard processes running at the same time. No wonder why the machine crawls to a halt..
The pfsense box is protected by the UPS so it shouldnt see the event at all… (theoretically speaking of course).
The only way to recover is to kill squidguard and then disable it in the webGUI. After that, the machine is back to normal.
I have tried searching the system logs for strange messages or anything out of the ordinary, nothing.
What are you guys thinking??? Somehow squid lost track of the modem because of the power fluctuation and goes wild? Am I the only one having this kind of problems??
**Why are so many squidGuard processes being launched????? **
-
Just did a bit more testing: after I killed squidGuard and disabled it in the webGUI, I logged in with SSH and cleared squid's cache with:
cd /var/squid/cache
rm -rf *then enabled squidguard again and rebooted.
Did not help, after reboot the machine crawled again to a complete stop and top showed about 25 squidGard processes running..
-
The issue is more present than ever. Every time the machine boots or lose WAN connectivity, it takes 30+minutes for it to respond to anything (SSH, webinterface).
I managed to confirm that this is caused by squidguard.
About 30-40 squidguard processes are running. I deactivated squidguard, and all was fine.
I uninstalled this package. I think it has severe issues.
Nobody has replied to this thread yet. I take that either nobody understands what is going on, or nobody had this issue before or care about it.
Has any dev looked into this???
-
Again tonight, I modified SG's config (added a domain to the target categories) like I do probably 10 times a week, and the router stopped working. Severe HDD activity, router not responding, websites stopping loading, etc >:( >:( >:(
Now my VOIP depends on pfsense. That cant continue.
$ ps -A PID TT STAT TIME COMMAND 0 ?? DLs 177:38.54 [kernel] 1 ?? SLs 0:00.05 /sbin/init -- 2 ?? DL 1:50.16 [g_event] 3 ?? RL 4:25.76 [g_up] 4 ?? DL 2:53.40 [g_down] 5 ?? DL 0:00.00 [crypto] 6 ?? DL 0:00.00 [crypto returns] 7 ?? DL 0:00.00 [sctp_iterator] 8 ?? DL 1:03.50 [pfpurge] 9 ?? DL 0:00.00 [xpt_thrd] 10 ?? DL 0:00.00 [audit] 11 ?? RL 23533:39.71 [idle] 12 ?? WL 483:41.74 [intr] 13 ?? DL 0:00.00 [ng_queue] 14 ?? DL 7:57.60 [yarrow] 15 ?? DL 0:42.49 [usb] 16 ?? DL 1:39.58 [acpi_thermal] 17 ?? DL 0:16.16 [pagedaemon] 18 ?? DL 0:00.36 [vmdaemon] 19 ?? DL 0:00.04 [pagezero] 20 ?? DL 0:03.54 [idlepoll] 21 ?? DL 0:17.68 [bufdaemon] 22 ?? DL 15:17.22 [syncer] 23 ?? DL 0:14.00 [vnlru] 24 ?? DL 0:21.51 [softdepflush] 40 ?? DL 0:19.84 [md0] 245 ?? INs 3:21.70 /usr/local/sbin/check_reload_status 247 ?? IWN 0:00.00 check_reload_status: Monitoring daemon of check_reloa 257 ?? Is 0:00.02 /sbin/devd 2396 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 2715 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 2738 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 2845 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 4907 ?? D 0:09.28 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 5011 ?? D 0:08.78 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 5319 ?? D 0:09.04 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 5396 ?? D 0:09.29 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 5529 ?? Is 0:00.13 /usr/local/sbin/sshlockout_pf 15 5736 ?? D 0:08.72 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 6035 ?? Is 0:00.00 /usr/sbin/sshd 6365 ?? D 0:22.03 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 6468 ?? D 0:23.23 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 6515 ?? D 0:21.55 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 6801 ?? Is 0:00.07 dhclient: re0 [priv] (dhclient) 6848 ?? D 0:21.44 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 7114 ?? D 0:21.84 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 8100 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 8230 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 8480 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 8808 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 9023 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 9289 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 9496 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 9753 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 10724 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 10778 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 10913 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 12344 ?? Ss 0:20.32 dhclient: re0 (dhclient) 13208 ?? Ss 0:15.62 /usr/sbin/cron -s 16871 ?? Ss 4:33.25 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log 17328 ?? D 0:17.47 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 17662 ?? D 0:17.63 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 17685 ?? D 0:17.99 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 17777 ?? D 0:17.64 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 17814 ?? D 0:17.42 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 17934 ?? D 0:33.44 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 18190 ?? D 0:33.49 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 18243 ?? D 0:34.27 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 18529 ?? D 0:32.95 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 18705 ?? D 0:33.19 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 20216 ?? S 0:00.67 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 20557 ?? S 0:00.47 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 20578 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 20768 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 20884 ?? Is 0:00.04 /usr/local/sbin/squid -D 20949 ?? S 0:00.17 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 21239 ?? S 0:00.27 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 21403 ?? S 0:00.02 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 21675 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 21798 ?? I 0:00.30 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 21881 ?? S 0:00.09 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 22095 ?? Ds 307:46.96 /usr/local/bin/ntop -i re0,re1 -u root -d -4 -M -x 81 22142 ?? Is 2:19.13 /usr/local/sbin/filterdns -p /tmp/filterdns.pid -i 30 22209 ?? I 0:00.02 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 22304 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 23045 ?? Ds 71:53.62 /usr/local/sbin/clamd -c /usr/local/etc/clamd.conf 23741 ?? DL 0:06.21 [md10] 24125 ?? Ss 7:37.85 /usr/local/sbin/apinger -c /var/etc/apinger.conf 25631 ?? SN 0:00.00 sleep 60 25762 ?? R 0:00.01 ps -A 28072 ?? S 0:59.81 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfig 28602 ?? IWs 0:00.00 /usr/local/bin/php 30096 ?? IWs 0:00.00 /usr/local/bin/php 30530 ?? Ss 0:25.95 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroo 32419 ?? S 0:06.04 /usr/local/bin/php 32731 ?? D 0:50.29 /usr/local/bin/php 38698 ?? S 0:01.39 (squid) -D (squid) 38859 ?? I 0:00.00 (unlinkd) (unlinkd) 39204 ?? I 0:00.09 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 39339 ?? I 0:00.07 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 39437 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 39503 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 39559 ?? Ss 2:04.82 /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf 39682 ?? S 0:00.07 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 39825 ?? S 0:00.09 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 39965 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 40116 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 40538 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 40849 ?? I 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 40980 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 41205 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 42829 ?? I 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 42997 ?? D 0:09.75 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 43100 ?? IWs 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid / 43129 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 43158 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 43186 ?? D 0:09.26 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 43229 ?? R 0:11.26 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 43281 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 43530 ?? I 0:01.81 minicron: helper /usr/local/bin/ping_hosts.sh (minic 43541 ?? D 0:09.40 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 43674 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 43677 ?? I 0:09.61 /usr/local/bin/rrdtool - 43730 ?? D 0:31.64 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 43739 ?? D 0:09.46 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 43767 ?? IWs 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts 43771 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 43823 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 44076 ?? R 0:30.82 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 44086 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 44098 ?? I 0:00.10 minicron: helper /etc/rc.expireaccounts (minicron) 44167 ?? IWs 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_u 44226 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 44348 ?? D 0:30.33 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 44389 ?? S 3:08.20 /usr/local/sbin/dnsmasq --local-ttl 1 --all-servers - 44473 ?? D 0:31.46 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 44562 ?? I 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 44594 ?? I 0:00.01 minicron: helper /etc/rc.update_alias_url_data (mini 44657 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 44676 ?? INs 0:00.02 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd. 44736 ?? R 0:32.10 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard. 44811 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 44910 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 45068 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 45118 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 45263 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 45599 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 45796 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 46069 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 46356 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 46702 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 46944 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 47136 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 47311 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 47382 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 47469 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 47705 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 47919 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 48205 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 48545 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 48681 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 48716 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 48874 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 49163 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 49502 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 49515 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 49847 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 50167 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 50227 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 50540 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 50757 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51098 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51166 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51192 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51209 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51454 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51585 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51676 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51734 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 51769 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 52037 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 53482 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 53518 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 54795 ?? Ss 17:23.59 /usr/sbin/powerd -b adp -a adp 56210 ?? I 0:00.00 sleep 55 59021 ?? Ss 0:00.09 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 59708 ?? S 0:00.95 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 59959 ?? S 0:00.60 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 59965 ?? S 0:00.82 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 60073 ?? S 0:00.01 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 60360 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 60528 ?? S 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 61680 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 61798 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 61995 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 62170 ?? I 0:00.00 /usr/local/sbin/havp -c /usr/local/etc/havp/havp.conf 16277 v0- S 1:16.15 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog 16308 v0- S 1:51.49 logger -t pf -p local0.info 32161 v0- I 0:49.28 /bin/sh /usr/local/pkg/sqpmon.sh 52753 v0- SN 4:51.06 /bin/sh /var/db/rrd/updaterrd.sh 52813 v0 Is+ 0:00.01 /usr/libexec/getty Pc ttyv0 53228 v1 Is+ 0:00.01 /usr/libexec/getty Pc ttyv1I have posted the output of ps -A before, and it showed way too many SG processes… now there are havp processes as well... Im wondering which package needs to be overhauled?
If someone can give me instruction on what to do to help the devs solve this ongoing situation, so be it. Im willing to do tests and troubleshoot but unfortunately Im not advanced enough in coding to solve the issue alone.
