Pfsense 2.1 i386 - squid2 transparent - http (80) performance really bad
-
Hello everybody,
I am using pfsense and squid since some years and in general it is working without major problems. But now I have a problem I could not solve and I would appreciate any help :-)
I have a pfsense 2.1 i386 running as a VM on Proxmox VE 3.1 (KVM). 4GB RAM + 4 core Intel Xeon E1220v2 3.1GHz. I have installed all the VirtIO drivers for HDD, NIC and so on. All this works.
On the WAN interface I have one separate NIC and on LAN interface another NIC with 8 VLANs.
I installed squid2 on this pfsense and it is running in transparent mode so squid only filters http (port 80) traffic on all 8 VLANs.Now I have problems that I cannot open any http websites. I can see that the browser title changes corresponding to the destination website but it does not load. I set firewall rules which allow "any to any" but it does not work. Opening https websites works - of course - it does not pass squid.
If I set the client to the "bypass proxy" list all websites are working. So it seems to be really a squid problem but I don't know why and what. On another pfsense machine it is running. I reinstalled squid but no difference.
Any suggestions?
-
What you get on cache.log? Most squid errors are there.
-
What you get on cache.log? Most squid errors are there.
Unfortunately nothing strange in there:
2013/11/09 20:07:47| Reconfiguring Squid Cache (version 2.7.STABLE9)... 2013/11/09 20:07:47| FD 17 Closing HTTP connection 2013/11/09 20:07:47| FD 18 Closing HTTP connection 2013/11/09 20:07:47| FD 19 Closing HTTP connection 2013/11/09 20:07:47| FD 22 Closing HTTP connection 2013/11/09 20:07:47| FD 23 Closing HTTP connection 2013/11/09 20:07:47| FD 24 Closing HTTP connection 2013/11/09 20:07:47| FD 25 Closing HTTP connection 2013/11/09 20:07:47| FD 26 Closing HTTP connection 2013/11/09 20:07:47| FD 27 Closing HTTP connection 2013/11/09 20:07:47| FD 28 Closing HTCP socket 2013/11/09 20:07:47| FD 29 Closing SNMP socket 2013/11/09 20:07:47| aioSync: flushing pending I/O operations 2013/11/09 20:07:47| aioSync: done 2013/11/09 20:07:47| logfileClose: closing log /dev/null 2013/11/09 20:07:47| Including Configuration File: /usr/pbi/squid-i386/etc/squid/squid.conf (depth 0) 2013/11/09 20:07:47| Cache dir '/var/squid/cache' size remains unchanged at 20480000 KB 2013/11/09 20:07:47| WARNING: use of 'override-expire' in 'refresh_pattern' violates HTTP 2013/11/09 20:07:47| WARNING: use of 'override-lastmod' in 'refresh_pattern' violates HTTP 2013/11/09 20:07:47| Initialising SSL. 2013/11/09 20:07:47| logfileOpen: opening log /dev/null 2013/11/09 20:07:47| Store logging disabled 2013/11/09 20:07:47| Referer logging is disabled. 2013/11/09 20:07:47| DNS Socket created at 0.0.0.0, port 47761, FD 12 2013/11/09 20:07:47| Adding nameserver 80.237.196.2 from squid.conf 2013/11/09 20:07:47| Adding nameserver 8.8.4.4 from squid.conf 2013/11/09 20:07:47| Adding nameserver 8.8.8.8 from squid.conf 2013/11/09 20:07:47| Adding nameserver 78.138.97.33 from squid.conf 2013/11/09 20:07:47| Adding nameserver 85.214.20.141 from squid.conf 2013/11/09 20:07:47| Adding nameserver 194.150.168.168 from squid.conf 2013/11/09 20:07:47| Adding nameserver 78.138.98.82 from squid.conf 2013/11/09 20:07:47| Adding nameserver 208.67.222.222 from squid.conf 2013/11/09 20:07:47| Adding nameserver 208.67.220.220 from squid.conf 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.0.1, port 3128, FD 17. 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.60.1, port 3128, FD 18. 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.64.1, port 3128, FD 19. 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.120.1, port 3128, FD 22. 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.180.1, port 3128, FD 23. 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.240.1, port 3128, FD 24. 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.252.1, port 3128, FD 25. 2013/11/09 20:07:47| Accepting proxy HTTP connections at 172.18.244.1, port 3128, FD 26. 2013/11/09 20:07:47| Accepting transparently proxied HTTP connections at 127.0.0.1, port 3128, FD 27. 2013/11/09 20:07:47| Accepting HTCP messages on port 4827, FD 28. 2013/11/09 20:07:47| Accepting SNMP messages on port 3401, FD 29. 2013/11/09 20:07:47| WCCP Disabled. 2013/11/09 20:07:47| Loaded Icons. 2013/11/09 20:07:47| Ready to serve requests.
I checked the firewall logs and it looks like there ist outgoing connection to port 80 on the two upstream pfsense boxes so traffic seems to pass.
At the moment I am using triple NAT because this pfsense box is in a testing environment. This test pfsense does NAT, the pfsense in front of this does NAT and my ADSL modem/router does NAT.
But as I said - https is working. So this should not be a problem. Further all squid config I did is the same as on my other pfsense with squid2 running.
Here is a part of my squid access.log:
1384010835.791 48 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_gutscheinauto.jpg - DIREC T/www.exxonmobil.com - 1384010835.791 39 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_safety.jpg - DIRECT/www.e xxonmobil.com - 1384010835.791 47 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_smartphone.jpg - DIRECT/w ww.exxonmobil.com - 1384010835.792 50 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_DClogo.jpg - DIRECT/www.e xxonmobil.com - 1384010835.792 49 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_flinc.jpg - DIRECT/www.ex xonmobil.com - 1384010835.817 76 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/feature_442x195_kristallglaeser_schott _zwiesel.jpg - DIRECT/www.exxonmobil.com - 1384010836.045 100553 172.18.240.200 TCP_HIT/200 76342 GET http://www.ard.de/pool/fonts/TheSans_LT_TT5_.woff - NONE/- application/font-woff 1384010836.132 311 172.18.240.200 TCP_MISS/200 724 GET http://www.exxonmobil.com/Imports/xom_web_large.css - DIRECT/158.28.238.7 text/css 1384010836.135 313 172.18.240.200 TCP_MISS/200 724 GET http://www.exxonmobil.com/Imports/xom_web_small.css - DIRECT/158.28.238.7 text/css 1384010836.469 652 172.18.240.200 TCP_MISS/404 5083 GET http://www.exxonmobil.com/favicon.ico - DIRECT/158.28.238.7 text/html 1384010836.472 0 172.18.240.200 TCP_NEGATIVE_HIT/404 5089 GET http://www.exxonmobil.com/favicon.ico - NONE/- text/html 1384010836.766 948 172.18.240.200 TCP_MISS/200 21403 GET http://www.exxonmobil.com/Germany-German/PA/products_fm.aspx - DIRECT/158.28.238.7 te xt/html 1384010837.541 1230 172.18.240.200 TCP_MISS/200 38283 GET http://www.exxonmobil.com/Imports/js/lightview.js - DIRECT/158.28.238.7 application/x -javascript 1384010854.172 17863 172.18.240.200 TCP_MISS/200 74670 GET http://www.exxonmobil.com/Imports/js/prototype.js - DIRECT/158.28.238.7 application/x -javascript 1384010854.174 4 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/feature_442x195_kristallglaeser_schott _zwiesel.jpg - DIRECT/158.28.238.7 - 1384010854.174 17875 172.18.240.200 TCP_MISS/200 83690 GET http://www.exxonmobil.com/Imports/xom.css - DIRECT/158.28.238.7 text/css 1384010854.245 69 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_gutscheinauto.jpg - DIREC T/158.28.238.7 - 1384010854.245 69 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_waschanlage.gif - DIRECT/ www.exxonmobil.com - 1384010854.245 69 172.18.240.200 TCP_MISS/000 0 GET http://www.exxonmobil.com/Germany-German/PA/Images/teaser_60x60_DClogo.jpg - DIRECT/158.2 8.238.7 -
So squid is getting the requests and searches the cache for HITs or MISSes.
I cleared the browsers cache, I did squid -z on pfsense. I confiured the webbrowser (Firefox) with the proxy port 3128. I used IE and no success.The webpage in my brwosers is loading a little bit but does not finish. Probably it just reloads what is still in the browsers webcache.
The client I am using for this does not have any anti-virus software installed but it is a Windows Server 2008R2. Not sure if I need to change something on the Windows Server machine but probably the common internet traffic should pass.
That's so strange …
-
Hi again,
I did some further tests, tried with different sysctl values, cleaned the squid cache and I must correct my statement of the first post aa little bit:
squid works on port 80 (hhtp) but it performs relally really bad. A webpage takes ~5minutes to completely load.If I un-select squid an an interface or add the computer to bypass squid proxy everything works as it should. So I thought it could be the HDD but when I set HDD cache to 0MB it still takes ~5minutes to load a webpage.
The CPU utilization on pfsense is really low, it is a quad core CPU and most time its 100% idle. Proxmox VE tells me the same CPU usage, too. And the IO delay on the VM is low, too. It is between 0% and short peaks of 1-2%.
-
Hi again,
I still didn't find the problem. :(
I changed the MTU on all mit interfaces and VLANs to 1450 and rebooted without luck.
I disabled flowcontrol in loader.conf and rebootet pfsense without luck
I disabled " Disable hardware checksum offload" and rebootet pfsense without luck
I enabled " Enable device polling" and rebooted pfsense without luck
I changed squid HDD cache from ufs, to aufs to null without luck
I enabled IPv6 support on pfsense even if I do not use it without luck
I changed the pfsense webGUI port to 47011 to make sure it is not listening on http 80 even if it does not make any senseI modified my squid.conf this way:
# Do not edit manually ! http_port 172.18.0.1:3128 http_port 172.18.60.1:3128 http_port 172.18.64.1:3128 http_port 172.18.120.1:3128 http_port 172.18.180.1:3128 http_port 172.18.240.1:3128 http_port 172.18.252.1:3128 http_port 172.18.244.1:3128 http_port 127.0.0.1:3128 transparent icp_port 0 pid_filename /var/run/squid.pid cache_effective_user proxy cache_effective_group proxy error_directory /usr/pbi/squid-i386/etc/squid/errors/English icon_directory /usr/pbi/squid-i386/etc/squid/icons visible_hostname unknown cache_mgr unknown@unknown.com access_log /var/squid/logs/access.log cache_log /var/squid/logs/cache.log cache_store_log none logfile_rotate 1 shutdown_lifetime 3 seconds # Allow local network(s) on interface(s) acl localnet src 172.18.0.0/255.255.252.0 172.18.60.0/255.255.252.0 172.18.64.0/255.255.252.0 172.18.120.0/255.255.252.0 172.18.180.0/255.255.252.0 172.18.240.0/255.255.252.0 172.18 .252.0/255.255.252.0 172.18.244.0/255.255.252.0 httpd_suppress_version_string on uri_whitespace strip dns_nameservers 80.237.196.2 8.8.4.4 8.8.8.8 78.138.97.33 85.214.20.141 194.150.168.168 78.138.98.82 208.67.222.222 208.67.220.220 cache_mem 1024 MB maximum_object_size_in_memory 2048 KB memory_replacement_policy lru cache_replacement_policy heap LFUDA cache_dir ufs /var/squid/cache 20000 16 256 minimum_object_size 64 KB maximum_object_size 378000 KB offline_mode off cache_swap_low 80 cache_swap_high 85 acl donotcache dstdomain "/var/squid/acl/donotcache.acl" cache deny donotcache # No redirector configured # Setup some default acls acl all src 0.0.0.0/0.0.0.0 acl localhost src 127.0.0.1/255.255.255.255 acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 47011 3128 1025-65535 acl sslports port 443 563 47011 acl manager proto cache_object acl purge method PURGE acl connect method CONNECT acl dynamic urlpath_regex cgi-bin \? cache deny dynamic http_access allow manager localhost # Allow external cache managers acl ext_manager_1 src 172.18.0.1 http_access allow manager ext_manager_1 http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !safeports http_access deny CONNECT !sslports # Always allow localhost connections http_access allow localhost quick_abort_min 8192 KB quick_abort_max 16384 KB quick_abort_pct 75 request_body_max_size 0 KB reply_body_max_size 0 deny all delay_pools 1 delay_class 1 2 delay_parameters 1 -1/-1 -1/-1 delay_initial_bucket_level 100 delay_access 1 allow all # Custom options ##### BEGIN ##### ##### Erstellen einer ACL die den Browser Intenet Explorer identifiziert # acl block_internet_explorer browser MSIE 8\.0 # acl block_internet_firefox browser Firefox/ # acl block_internet_chrome browser Chrome/ ##### ##### Erstellen einer ACL von Quell-IPs deren Browser wir nicht blocken wollen (Gaestehaeuser und LS106) # acl browsers_allowed_src src 172.17.252.0/22 172.17.120.0/22 172.17.64.0/22 172.17.0.21/32 172.17.0.22/32 172.17.0.23/32 ##### ##### Zuerst die ACL platzieren, die den Internet Explorer erlaubt aber nur fuer das spezielle Subnetz # http_access deny !browsers_allowed_src block_internet_explorer # http_access deny !browsers_allowed_src block_internet_firefox # http_access deny !browsers_allowed_src block_internet_chrome ##### ##### Anschliessend die ACL platzieren, die den ausgewaehlten Browser fuer alle weiteren Subnetze blockiert # http_access deny block_internet_explorer # http_access deny block_internet_firefox # http_access deny block_internet_chrome ##### ##### ##### Betriebssystem Updates ##### Haltbarkeit 180 Tage refresh_pattern -i .*apple\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz|pkg|dmg) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignor e-no-cache ignore-private refresh_pattern -i .*microsoft\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz|msp) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignor e-no-cache ignore-private refresh_pattern -i .*windowsupdate\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz|msp) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload i gnore-no-cache ignore-private ##### refresh_pattern -i .*ubuntu\.com/.*\.(tar|bz|tbz|bz2|gpg|gz|zip|deb) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-pri vate ##### refresh_pattern -i .*debian\.org/.*\.(tar|bz|tbz|bz2|gpg|gz|zip|deb) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-pri vate ##### ##### Adobe und Pinnacle Updates ##### Haltbarkeit 180 Tage refresh_pattern -i .*adobe\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cac he ignore-private refresh_pattern -i .*pinnaclesys\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|gz) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ign ore-private ##### ##### Sonstige Software ##### Haltbarkeit 180 Tage refresh_pattern -i .*smarttech\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no -cache ignore-private refresh_pattern -i .*divx\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cach e ignore-private refresh_pattern -i .*fujitsu\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-c ache ignore-private refresh_pattern -i .*real\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cach e ignore-private refresh_pattern -i .*vmware\.com/.*\.(exe|zip|rar|cab|tar|bz|tbz|bz2|msi|mst|msu|mzz|gz) 259200 90% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-ca che ignore-private ##### ##### Videos, Audio ##### Haltbarkeit 7 Tage refresh_pattern -i /.*\.(iso|img|wmv|wma|mov|rm|avi|mp3|mp4|mpeg|mpg|divx|xvid|swf|flv|x-flv) 10080 90% 10080 override-expire override-lastmod reload-into-ims ignore-reload ignore-no -cache ignore-private ##### ##### Erzwingt das Herunterladen der kompletten Datei (-1) oder gar nicht (0) oder laedt alles wenn der Teil vor dem Downloadbeginn kleiner als 10 MB (range_offset_limit 10 MB) ist range_offset_limit 50 MB ##### ##### SQUID vorgegebene Eintraege refresh_pattern -i ^ftp: 1440 20% 10080 refresh_pattern -i ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern -i . 0 20% 4320 ##### ##### Senkt den Timer fuer die DNS Aufloesung negative_ttl 1 seconds negative_dns_ttl 60 seconds positive_dns_ttl 6 hours dns_timeout 30 seconds ##### Number of IPs cached for a domain ##### DEFAULT ipcache_size 1024 ipcache_size 32768 ipcache_low 92 ipcache_high 95 ##### DEFAULT fqdncache_size 1024 fqdncache_size 16384 ##### DEFAULT /etc/hosts hosts_file /etc/hosts ##### ##### Im cachemgr - Store Directory Stats - Filemap bits in use sollte unter 50 Prozent liegen ##### Im cachemgr - General Runtime Information - Mean Object Size oder kleiner ergibt unten stehenden Wert store_avg_object_size 512 KB ##### ##### Beschleunigt die Performance von "pipelined requests" wie in Umgebungen ohne Proxy ##### Default ist und muss "off" sein, wenn man Bandbreitenmanagement nutzt und Logging "access.log" aktiviert hat pipeline_prefetch off ##### reply_header_max_size 64 KB ##### request_header_max_size 64 KB ##### half_closed_clients off ##### ##### For transparent proxies httpd_accel_no_pmtu_disc on ##### ##### Wie haeufig SQUID neue requests akzeptiert. Default 30, je niedriger dest haeufiger incoming_rate 10 ##### ##### VORAUSSETZUNG: Zugriff auf cachemgr.cgi im GUI ist erlaubt ##### "disable" deaktiviert den Offline-Mode, Reconfig/Neustart und Herunterfahren cachemgr_passwd disable offline_toggle reconfigure shutdown ##### "none" deaktiviert eine Passwortabfrage bzw. man ersetzt "none" durch ein Passwort ##### "Manager Name" im GUI ist "admin" cachemgr_passwd notforpublicuse all ##### END redirect_program /usr/pbi/squidguard-i386/bin/squidGuard -c /usr/pbi/squidguard-i386/etc/squidGuard/squidGuard.conf redirector_bypass off url_rewrite_children 75 # Setup allowed acls # Allow local network(s) on interface(s) http_access allow localnet # Default block all to be sure http_access deny all
This is my loader.conf:
autoboot_delay="3" vm.kmem_size="512M" vm.kmem_size_max="532M" kern.ipc.nmbclusters="131072" hw.usb.no_pf="1"
This is my loader.conf.local:
virtio_load="YES" virtio_pci_load="YES" if_vtnet_load="YES" virtio_balloon_load="YES" virtio_blk_load="YES" kern.ipc.somaxconn=2048 vm.pmap.shpgperproc=2048 net.inet.tcp.sendbuf_max=1048576 net.inet.tcp.recvbuf_max=1048576 kern.maxfilesperproc=32768 kern.maxfiles=262144 legal.intel_wpi.license_ack="1" legal.intel_ipw.license_ack="1" kern.ipc.nmbclusters="131072" hw.em.fc_setting=0
I am running a fresh install of pfsense 2.1 i386:
Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz 4 CPUs: 1 package(s) x 4 core(s)
8GB RAM in a Proxmox VE 3.1 VM as KVM (VirtIO drivers installed as you can see in loader.conf.local)
I have two Intel NICs in my server. One is bridged in Proxmox as the WAN interface in pfsense. The second one is bridged and I created VLANs on pfsense.Firewall allows traffic from any to any - WAN and on VLANs.
I could not explain and find a solution why squid in transparent mode is so slow when browsing the web. I read ton's of threads here in the forum when searching for "squid slow" and googled but nothing improved performance.
So I would really appreciate any help or suggestions.
-
So for everyone who is interested:
I had two NICs, both running with VirtIO drivers. No I changed the NIC with the different VLANs to E1000 and now everything is working.
CPU usage increased a little bit but it's not a big problem as there ist not so much traffic.
-
It was a network card driver issue?
-
It was a network card driver issue?
I don't know what exactly was the issue, the driver, KVM (Proxmox), FreeBSD or something else but it was somehow network car related.
For me this makes no sense because with VirtIO https traffic was fast, without squid ist was fast and windows file transfer was fast, too. Only http through squid was slow.
-
In my actual configuration there are 3 Vlans in one netcard, wan+2lan. Works under VmWare ESXi 4. Before I made firewall blocks for 137-139 and 445 port, it was ruined if somebody starts to download files from different vlan. Every server have own adress in each vlan. I designed router for satellite Internet with bandwidth 3 megabits, but not 100+100 megabits :)
Now everething works as designed, because everething has cached. 36-39% hits. No blocks for antiviruses updates, microsoft etc… -
In my actual configuration there are 3 Vlans in one netcard, wan+2lan. Works under VmWare ESXi 4. Before I made firewall blocks for 137-139 and 445 port, it was ruined if somebody starts to download files from different vlan. Every server have own adress in each vlan. I designed router for satellite Internet with bandwidth 3 megabits, but not 100+100 megabits :)
Now everething works as designed, because everething has cached. 36-39% hits. No blocks for antiviruses updates, microsoft etc…Hi,
I don't understand how this is related to this thread!?
-
My resume: watch "states" and make sure there are no unwanted trafic.