Distro updating issues behind pfsense
-
Hello,
for the last few weeks I have been constantly having problems to run package updates on various boxes sitting behind my pfsense router. The issues I am encountering are different from time to time but generally are regarding problems fetching files or other index files, retrieving repo info, etc…
I have 8 machines running behind pfsense, 5 Centos machines, and 2 Ubuntu machines, and a Windows XP machine. No problem to run windows upate on the wxp machine but I have issues on all other linux based machines.
pfSense is currently configured with Snort, Squid, Squidguard, HAVP and pfBlocker.
I have tried these to proceed by elimination and find the root cause, to no avail:
-Disable snort completely: No hosts were blocked by snort but nevertheless I did not take any chance and disabled it.
-Uninstall Snort completely and reboot router.
-Disable squidguard
-Look in the firewall logs for hosts being blovked, none found so far.
-Disable pfblocker temporarily
-Uninstall pfblocker completely then rebooting routerSome sample outputs of failed updates:
A Centos machine
[root@centos-prod ~]# yum update
Loaded plugins: fastestmirror, priorities, security
Setting up Update Process
Determining fastest mirrors
epel/metalink | 14 kB 00:01- base: mirror.esecuredata.com
- centosplus: centos.mirror.netelligent.ca
- contrib: mirror.esecuredata.com
[…]
elrepo/primary_db | 703 kB 00:00
epel | 4.4 kB 00:00
http://mirror.steadfast.net/epel/6/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.
epel | 4.4 kB 00:00
epel/primary_db | 6.3 MB 00:01
extras | 3.3 kB 00:00
extras/primary_db | 19 kB 00:00
remi | 2.9 kB 00:00
remi/primary_db | 1.0 MB 00:00
rpmforge | 1.9 kB 00:00
rpmforge/primary_db | 2.7 MB 00:00
rpmfusion-free-updates | 2.7 kB 00:00
http://mirror.pw/rpmfusion/free/el/updates/6/x86_64/repodata/245c77772e38b05ca0cd82e4106e41aa2e4fe1644d9cda511b04d7de4329bf1b-primary.sqlite.bz2: [Errno 12] Timeout on http://mirror.pw/rpmfusion/free/el/updates/6/x86_64/repodata/245c77772e38b05ca0cd82e4106e41aa2e4fe1644d9cda511b04d7de4329bf1b-primary.sqlite.bz2: (28, 'Operation too slow. Less than 1 bytes/sec transfered the last 30 seconds')
Trying other mirror.
rpmfusion-free-updates/primary_db | 267 kB 00:00
[…]
Another centos machine
[root@workstation]# yum update
Loaded plugins: fastestmirror, priorities, refresh-packagekit, security
Setting up Update Process
Determining fastest mirrors
epel/metalink | 14 kB 00:00- base: centos.mirror.netelligent.ca
- centosplus: centos.mirror.netelligent.ca
- contrib: mirror.science.uottawa.ca
[…]
adobe-linux-x86_64 | 951 B 00:00
adobe-linux-x86_64/primary | 1.2 kB 00:00
adobe-linux-x86_64 2/2
base | 3.7 kB 00:00
base/primary_db | 4.6 MB 00:00
centosplus | 3.4 kB 00:00
centosplus/primary_db | 313 kB 00:00
contrib | 2.9 kB 00:00
contrib/primary_db | 1.2 kB 00:00
elrepo | 2.9 kB 00:00
elrepo/primary_db | 705 kB 00:00
epel | 4.4 kB 00:00
http://mirror.pnl.gov/epel/6/x86_64/repodata/7172843deb89bdc76f7779173c517a7b0e6d580928c147a7f43c4fca72fbdebe-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://archive.linux.duke.edu/pub/epel/6/x86_64/repodata/7172843deb89bdc76f7779173c517a7b0e6d580928c147a7f43c4fca72fbdebe-primary.sqlite.bz2: [Errno 12] Timeout on http://archive.linux.duke.edu/pub/epel/6/x86_64/repodata/7172843deb89bdc76f7779173c517a7b0e6d580928c147a7f43c4fca72fbdebe-primary.sqlite.bz2: (28, 'Operation too slow. Less than 1 bytes/sec transfered the last 30 seconds')
Trying other mirror.
epel/primary_db | 6.3 MB 00:00
extras | 3.3 kB 00:00
extras/primary_db | 19 kB 00:00
nux-dextop | 2.9 kB 00:00
http://mirror.li.nux.ro/li.nux.ro/nux/dextop/el6/x86_64/repodata/f2f76652b3a22b925d64e5384628cc122017374bbdda8452d62c9d7e6f492adb-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://li.nux.ro/download/nux/dextop/el6/x86_64/repodata/f2f76652b3a22b925d64e5384628cc122017374bbdda8452d62c9d7e6f492adb-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://mirror.li.nux.ro/li.nux.ro/nux/dextop/el6/x86_64/repodata/f2f76652b3a22b925d64e5384628cc122017374bbdda8452d62c9d7e6f492adb-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
http://li.nux.ro/download/nux/dextop/el6/x86_64/repodata/f2f76652b3a22b925d64e5384628cc122017374bbdda8452d62c9d7e6f492adb-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"
Trying other mirror.
Error: failure: repodata/f2f76652b3a22b925d64e5384628cc122017374bbdda8452d62c9d7e6f492adb-primary.sqlite.bz2 from nux-dextop: [Errno 256] No more mirrors to try.
A Ubuntu machine
vm@musicbrainzvm:~$ sudo apt-get update
Hit http://security.ubuntu.com precise-security Release.gpg
Hit http://security.ubuntu.com precise-security Release
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://ppa.launchpad.net precise Release
[…]
Hit ftp://ubuntu.mirror.iweb.ca precise/main Translation-en
Hit ftp://ubuntu.mirror.iweb.ca precise/restricted Translation-en
Hit ftp://ubuntu.mirror.iweb.ca precise-updates/main Translation-en
Hit ftp://ubuntu.mirror.iweb.ca precise-updates/restricted Translation-en
Fetched 3,231 kB in 1min 28s (36.3 kB/s)
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/precise-security/main/binary-amd64/Packages Hash Sum mismatchW: Failed to fetch http://security.ubuntu.com/ubuntu/dists/precise-security/main/binary-i386/Packages Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
I have tried flushing the package manager's cache (on centos yum clean all, manually deleting the cache files on ubuntu machines, following all I could find on the web) but nothing helps. Everything points to either ISP transparent proxy cache corruption, or router issues.
The repos failing to properly update change from time to time, and sometimes the update succeeds but most of the time it fails with similar errors as reported above..
Any idea how I could troubleshoot this from a router POV??
Thanks!
-
Try without squid or any proxy involved.
I update them all the time behind the firewall without a proxy involved and it's fine there. Must be something in the proxy/av settings.
-
I'm currently trying to investigate some trouble I have with squid/squidGuard. Simply disabling squidGuard does not immediate solve the issue; it seems that squid needs a restart to get on its feet. Switching squid from transparent to normal helps immediately. I currently have very little data, as the issue pops up at some time, then everything's fine again for a few days, the logs show nothing…a typical Heisenbug.
https://forum.pfsense.org/index.php?topic=82510.msg453034#msg453034
This is not neccessarily the same issue as you are facing, but with the currently less than total clear situation I am experiencing I cannot be sure of anything...
-
I am happy to report that after destroying squid's cache and recreating it, all works as intended!!!!
Basically I used the following commands:
squid -k shutdown rm -fr /var/squid/cache/* squid -z /usr/local/sbin/squid -DThat however makes me wonder if this problem will not come back in a few months…
-
As I expected 10 minutes after cleaning up squid the problems resurfaced…..
I uninstalled it completely.
Now the problem is gone, but for whatever reason I am having extreme speed problems with an average speed to the outside world of 30Kb/s when I should be getting 3Mb/s (not MBPS but Megabytes per sec)
I tried rebooting the router and it didnt help.
frustrating...
-
As a follow up to this thread, and for future reference, I ended up reinstalling pfsense completely without squid and its associated packages (SG, etc). Now everything works fine and firewall performance is as expected. Its been a month since I restarted fresh and so far so good.
But before I did so, I did a test and reinstalled pfsense with squid, then uninstalled squid. I ended up with the same performance issues..
I strongly assume that somehow, installing squid and squidguard alters pfsense in such a way that when uninstalled, pfsense remains altered and that causes the performance degradation I had.
-
OOps reviving this thread!
Same issue again. This time, I need squid to perform web filtering and caching, but of course the issue with package managers on LAN clients resurfaced and I am getting the very same issues as before..
I posted a bug report on pfsense bug tracker but the ticket was rejected saying "this is almost certainly a problem within squid itself, or a problem on the servers in question"
AFAIK all of my servers are configured for standard connection to the web and yum (or synaptics, apt-get, windows update, etc) are all configured standard out of the box (no special proxy settings).. As a matter of fact, squid is configured to be a transparent proxy on my pfsense tbox so LAN clients shouldnt "see it". Also, why all platforms would have issues? I mean even a standard windows XP box has issues with Windows Update..
Nevertheless, it doesnt work and I am really disappointed that so far I havent been able to find a solution to this other than manually clearing squid's cache. If I do this (with the commands of post #4 here) all is well until the cache is filled again a few days/weeks later and needs a flush again..
Who maintains squid? Perhaps talking to this (these) guys would be a good start..
-
Replying to myself for the sake of documenting the issue..
Replacing Squid2 by Squid3 solved the issue.
Apparently the problem is a combination of package manager assuming HTTP/1,1 protocol mechanisms (revalidation particularly) while Squid-2.7 is only HTTP/1.0 compliant. Anyways this is what I understood from the reply I got on Squid-cache's bugzilla.
Apparently, the APT problem has been confirmed fixed years ago in Squid-3.1
IMO Squid2 should be marked deprecated in the package repo or strong warning!!!
