Squid3 - New GUI with sync, normal and reverse proxy

marcelloc

version 2.0.5 is out with:

new binaries again to fix transparent proxy
new option to patch captive portal to work together with non transparent use

As I'm including new features to this package, the status is back to beta until improvements and tests are done.

att,
Marcello Coutinho

al_reidy

version 2.0.5 is out with:

new binaries again to fix transparent proxy

new option to patch captive portal to work together with non transparent use

As I'm including new features to this package, the status is back to beta until improvements and tests are done.

att,
Marcello Coutinho

dude, legend it works now. i uninstalled the package then installed.
it didn't work right away though i had to stop the package, altered the config to 32 directories to store the cache.ran squid -z, then chmod -R 777 the cache directory then rebooted.
perhaps on a fresh install that won't be needed.
thanks for your time on this. :-D

Donny

Hello all,

I have tested between Squid3 with LDAP (Windows Server 2008). I can use domain users to authentication login to web browser and successes.

pfSense configuration detail

System > General setup > DNS Servers :
172.31.21.10 (Internal DNS, DHCP Windows Sever 2008 )
208.67.222.222 (OpenDNS)
208.67.220.220 (OpenDNS)

Sevices > DNS forwarders : Enable DNS forwarders has checked.

On Windows Server 2008

At DNS forwarder tab I forward to
172.31.21.1 pfSense
208.67.222.222 OpenDNS
208.67.220.220 OpenDNS
also I have made pfsense record name on DNS server.

After domain users successes login with web browser (Firefox, IE,Opera and Chrome). At system log I got DNS-rebind attack as the detail below.

Apr 22 13:13:31 	dnsmasq[30943]: possible DNS-rebind attack detected: ForestDnsZones.xxxx.dsns
Apr 22 13:13:31 	dnsmasq[30943]: possible DNS-rebind attack detected: ForestDnsZones.xxxx.dsns
Apr 22 13:13:31 	dnsmasq[30943]: possible DNS-rebind attack detected: DomainDnsZones.xxxx.dsns
Apr 22 13:13:31 	dnsmasq[30943]: possible DNS-rebind attack detected: DomainDnsZones.xxxx.dsns
Apr 22 13:13:31 	dnsmasq[30943]: possible DNS-rebind attack detected: xxxx.dsns
Apr 22 13:13:31 	dnsmasq[30943]: possible DNS-rebind attack detected: xxxx.dsns

I tried to find another solution by google search and some pfsense forum but can not solve this problem. Also I tried to "disable DNS Rebinding Checks" or "Alternate Hostnames" or
"Browser HTTP_REFERER enforcement" at System > Advanced and domain overrides but when I do this I can not login with domain users to web browser. finally reboot pfSense and it does not help.

Any suggestion !

Donny

Hello Marcelloc,

I just would like to inform you that Squid3 authentication with LDAP Windows Server 2008 does not work very well with OpenDNS. When I only use OpenDNS 208.67.222.222 and 208.67.220.220 at System > General Setup > DNS Servers, and I try to login via web browser with domain users name, the web browser still hang up only "loading" and take too long before the web page is coming up.

The way I solved this problem is :

1. Use DNS Server from ISP : 67.xx.xxx.xx and 203.xx.xxx.xx or Use DNS Server from google : 8.8.8.8 and 8.8.4.4
2. At System > General Setup > DNS Servers. I take off IP address from internal DNS Server Windows 2008 because it will cause "DNS-rebind attack detected" If I still use internal dns ip address.

So, at System > General Setup > DNS Servers, I only use DNS Server from my ISP (67.xx.xxx.xx and 203.xx.xxx.xx) or use Google DNS Server 8.8.8.8 and 8.8.4.4. that's it.
Now I can use domain users to authenticate login via web browser and I don't get any DNS-rebind attack detected anymore. Every users from the domain that I tested, it's succeses.

SARG report at "View Report and Realtime tab", I have success to use a real user name from domain users (Windows Server 2008).

See screenshot.

Thank u very much Marcelloc

Donny

Hello Marcello,

When I reboot pfSense. At the console I saw some warning: Invalid argument supplied for foreach() in /usr/local/pkg/squid.inc on line 946.
This is squid.inc code and **this is a line 946>**foreach ($config['installedpackages']['squidremote']['config'] as $settings)

function squid_resync_upstream() {
   global $config;
   $conf = "\n#Remote proxies\n";
foreach ($config['installedpackages']['squidremote']['config'] as $settings){
      if ($settings['enable'] == 'on') {
         $conf .= "cache_peer {$settings['proxyaddr']} {$settings['hierarchy']} {$settings['proxyport']} ";
         if ($settings['icpport'] == '7')
              $conf .= "{$settings['icpport']} {$settings['icpoptions']} {$settings['peermethod']} {$settings['allowmiss']} ";
          else
               $conf .= "{$settings['icpport']} ";
            #auth settings
         if (!empty($settings['username']) && !empty($settings['password'])){
            $conf .= " login={$settings['username']}:{$settings['password']}";
            }
         else{
            $conf .= "{$settings['authoption']} ";
         }
         #other options settings
         if (!empty($settings['weight']))
            $conf .= "weight={$settings['weight']} ";
         if (!empty($settings['basetime']))
            $conf .= "basetime={$settings['basetime']} ";
         if (!empty($settings['ttl']))
            $conf .= "ttl={$settings['ttl']} ";
         if (!empty($settings['nodelay']))
            $conf .= "no-delay";
      }
      $conf .= "\n";
    }
   return $conf;
}

marcelloc

Donny,

I've pushed a fix for these array right now, wait 15 minutes, reinstall the package, and check if it stops the bootup error.

Donny

@marcelloc:

Donny,

I've pushed a fix for these array right now, wait 15 minutes, reinstall the package, and check if it stops the bootup error.

Hello Marcelloc,

After reinstall Squid3 and reboot system, the bootup error problem has solved.

Thank u

tester_02

I am getting the following error after installing squid 3. I've looked at the folder and there is no mime.conf file.

I had squid 2 + squidguard. I installed squid 3, then uninstalled squid 2 and this started happening (had originally thought 3 would overwrite 2, but both were shown in the packages). I've even tried installing 3 again, but the same error happens. I would have stayed with 2, but I've always had trouble with ncix.com and some youtube videos (preview window plays video and then it runs another preview in the preview)

Apr 24 22:00:06 squid: MIME Config Table /usr/local/etc/squid/mime.conf: (2) No such file or directory
Apr 24 21:59:32 php: : SQUID is installed but not started. Not installing "filter" rules.
Apr 24 21:59:32 php: : SQUID is installed but not started. Not installing "pfearly" rules.
Apr 24 21:59:32 php: : SQUID is installed but not started. Not installing "nat" rules.
Apr 24 21:59:26 check_reload_status: Reloading filter
Apr 24 21:59:18 php: : SQUID is installed but not started. Not installing "filter" rules.
Apr 24 21:59:17 php: : SQUID is installed but not started. Not installing "pfearly" rules.
Apr 24 21:59:17 php: : SQUID is installed but not started. Not installing "nat" rules.
Apr 24 21:59:16 php: /pkg_edit.php: The command '/usr/local/sbin/squid' returned exit code '1', the output was '2012/04/24 21:59:16| ERROR: MIME Config Table /usr/local/etc/squid/mime.conf: (2) No such file or directory FATAL: MIME Config Table /usr/local/etc/squid/mime.conf: (2) No such file or directory Squid Cache (Version 3.1.19): Terminated abnormally. CPU Usage: 0.007 seconds = 0.007 user + 0.000 sys Maximum Resident Size: 5744 KB Page faults with physical i/o: 0'
Apr 24 21:59:16 squid: MIME Config Table /usr/local/etc/squid/mime.conf: (2) No such file or directory

So I manually created a blank mime.conf file. That error went away and I then I in turn got a missing "icons" folder in the same location. I created that, and no squid works, but squidguard fails to work..

squid[58395]: Squid Parent: child process 58727 exited due to signal 6 with status 0

Closer, but not quite working at this stage for me…
hints anyone?

tester_02

Update:
Saved all screens in squid and squidguard for luck, and now it's up. I was scared to reinstall squidguard as I had read that squidguard would reinstall squid 2 again.

So finally squid 3 + squidguard working good. ncix.com even works! now to just watch some youtube videos and see if the problem comes up again.

installer still does need a fix for the missing file and missing folder.

Also getting error 22 invalid argument if I try to edit the message above this.

phil.davis

When you are able to make PBIs for installing Squid3 on 2.1-DEVELOPMENT I am happy to test it. No rush - I see that you already have plenty of work just now!

marcelloc

@phil.davis:

When you are able to make PBIs for installing Squid3 on 2.1-DEVELOPMENT I am happy to test it. No rush - I see that you already have plenty of work just now!

On 2.1, install package gui and then go to console to pkg_add -r binaries until I find time to build and test pbi

IGIdeus

Hi,

I looked at throttle_exts.acl generated with "Throttle multimedia files" option checked. IMHO it lacks of extensions: wma, wav, mka, mkv, ogg, oga, ogm, ogv, rmvb.

Best regards
IGIdeus

pizetta

Hi,
At "Proxy server: Traffic management" we can manage a single delay pool with the options: Per-host throttling / Overall bandwidth throttling / Maximum upload size.
I need to manage many groups of delay pools and set to different networks/Ips. This is very usefull, are you going to implement this?

Thanks in advance.

marcelloc

@pizetta:

I need to manage many groups of delay pools and set to different networks/Ips. This is very usefull, are you going to implement this?

No plans for this feature yet. But if you need it, you can post a bountry or make a donation for that ;)

Do you have any config sample for this?

pizetta

Something like this…

delay_pools 3 ######3 delay pools
delay_class 1 2
delay_parameters 1 -1/-1 12800/12800
delay_access 1 allow client_100k ######limited clients 100kbps

delay_class 2 2
delay_parameters 2 -1/-1 25600/25600
delay_access 2 allow clientes_200k ######limited clients 200kbps

delay_class 3 2
delay_parameters 3 -1/-1 38400/38400
delay_access 3 allow clientes_300k ######limited clients 300kbps

I wish to manage as many pools as I can, grouping users to limit the use of internet. I'll take a look at bounties.
Best regards!

yosu

Hi,

I am using package squid3 version 3.1.19 pkg 2.0.5_2.

In /usr/local/pkg/squid.inc I think this line:

http_access deny CONNECT !sslports

should be:

http_access deny connect !sslports

Also in reverse proxy web gui I need to set port 443 in order to https work. If I left blank reverse HTTPS port, it doesn't open port 443.

Also the reverse proxy HTTPS always redirect to the reverse HTTPS default site. Don't mind what mappings you use.

HTTP reverse proxy works fine.

Best regards.

nutt318

When using the Reverse Proxy in theory will it redirect the traffic based upon the URL?

I've got 2 internal webservers with one public IP, should I be able to route the traffic based upon server1.mydomain.com to 192.168.1.50 and say server2.mydomain.com to 192.168.1.51 ?

It seems I've configured the reverse proxy properly and added a rule to allow http traffic to each private address but i'm not having any luck.

Anyone have any ideas or a detailed instructions?

marcelloc

@nutt318:

When using the Reverse Proxy in theory will it redirect the traffic based upon the URL?

I've got 2 internal webservers with one public IP, should I be able to route the traffic based upon server1.mydomain.com to 192.168.1.50 and say server2.mydomain.com to 192.168.1.51 ?

yes,

@nutt318:

It seems I've configured the reverse proxy properly and added a rule to allow http traffic to each private address but i'm not having any luck.

what you got on squid access log?

nutt318

Here is what I see in the access.log file /var/squid/logs - I only see internal traffic. Shouldn't I see attempts from public IP's that are trying to access the web servers?

1338991913.168 179241 MY.DESKTOP.I.P TCP_MISS/504 1290 GET http://domain1.mydomain.com/ - DIRECT/MY.PUBLIC.I.P text/html

I'll try and add some screenshots of my setup

EDIT: added screenshots

marcelloc

The setup looks fine, I'll try to simulate it.

yosu

@nutt318:

Make URI textbox blank in order to get:

acl test1 url_regex -i http://test1.mydomain.com.*$

Now you get:

acl test1 url_regex -i test1.mydomain.com/http://test1.mydomain.com.*$

You can look at /usr/local/etc/squid/squid.conf in a ssh shell.

Best regards.

nutt318

That made it work from internal on the LAN, but I still cant get to it from the outside.

Any other ideas?

Thanks!

cjbujold

Unable to get reverse squid 3 to work. Here is my configuration, if somebody can help. The example i'm trying to get to work is 2 web servers; one on port 80 and another on port 8081. The request comes to port 80 should be pickedup by squid and depending on the URL squid should send the request either to port 80 of the web server or to port 8081. The test Im using is www goes to port 80 and helpdesk goes to port 8081. When I try it, everythings goes to port 80. Port 8081 is never sent aqnything and the helpdesk goes to port 80.

The squid.conf file reverse proxy section looks like this: (XXX is equal to mydomainname)

Reverse Proxy settings

http_port 192.168.XXX.XXX:80 accel defaultsite=XXXX.ca vhost
http_port 156.34.XXX.XXX:80 accel defaultsite=XXXX.ca vhost
#XXXX HelpDesk
cache_peer 192.168.XXX.15 parent 8081 0 proxy-only no-query originserver login=PASS name=XXXXHelpDesk

acl XXXXHelpDesk url_regex -i http://helpdesk.XXXX.ca/.$
acl XXXXHelpDesk url_regex -i http://helpdesk.XXXX.com/.$
cache_peer_access XXXXHelpDesk allow XXXXXHelpDesk
cache_peer_access XXXXHelpDesk allow XXXXHelpDesk
cache_peer_access XXXXHelpDesk deny allsrc
cache_peer_access XXXXHelpDesk deny allsrc
never_direct allow XXXXHelpDesk
never_direct allow XXXXHelpDesk
http_access allow XXXXHelpDesk
http_access allow XXXXHelpDesk

deny_info TCP_RESET allsrc

Custom options

![MappingDetail PM.png](/public/imported_attachments/1/MappingDetail PM.png)
![MappingDetail PM.png_thumb](/public/imported_attachments/1/MappingDetail PM.png_thumb)

nutt318

Sounds like the same issue I'm having, however it looks like one of your mappings isnt ON. Maybe that will fix it, if so I need to look over my config again.

nutt318

I looked at my squid.config file at its basically the same as cjbujold's.

Is there anything else to try, or does anyone have any idea why this isnt working?

Thanks for the help.

IGIdeus

Hi,

There is no possible to restart/start squid service from dashboard and services GUI pages.

Best regards
IGIdeus

IGIdeus

Hi,

IMHO squid as a package for firewall should be hardened a little bit more.
From my perspective ACL safe_ports should include only 21, 80, 443 and 1025-65535 ports, ACL SSL should include only 443 port. All other ports should be added manually.
There could be information about other ports in description of the options.

The brilliant function could be possibility to manage the ACLs like in Webmin or like firewall rules in pfSense.

Best regards
IGIdeus

marcelloc

@IGIdeus:

There is no possible to restart/start squid service from dashboard and services GUI pages.

Apply this patch on your 2.0.1 install to fix restart service option
https://github.com/bsdperimeter/pfsense/commit/6ae78f0808747893f30b867c51b744dfe39e2190

@IGIdeus:

From my perspective ACL safe_ports should include only 21, 80, 443 and 1025-65535 ports, ACL SSL should include only 443 port. All other ports should be added manually.

the current list (21 70 80 210 280 443 488 563 591 631 777 901 1025-65535) is not that big. I think(and in some cases I remove) that 1025-65535 is the most "unsecure" port range on this array. You can chage it editing squid.inc file.

@IGIdeus:

The brilliant function could be possibility to manage the ACLs like in Webmin or like firewall rules in pfSense.

It's on the todo list, but I need some free time to finish.

nutt318

@marcelloc:

The setup looks fine, I'll try to simulate it.

Did you ever have a chance to simulate the reverse proxy traffic?

blasterreal

Pfsense 2.0.1 32 BIT

Squid services not started :(

-php: /pkg_edit.php: The command '/usr/local/sbin/squid -k shutdown' returned exit code '1', the output was 'FATAL: Bungled squid.conf line 4: http_port 127.0.0.1:3128 intercept Squid Cache (Version 2.7.STABLE9): Terminated abnormally.'

-squid[54825]: Bungled squid.conf line 4: http_port 127.0.0.1:3128 intercept

marcelloc

@Blasterreal:

Squid Cache (Version 2.7.STABLE9)

Did you installed squidguard after squid? force a squid3 reinstall, check config, apply settings and test again.

blasterreal

Thanks its working ;)

m4st3rc1p0

Hi,

Is there a way that we can enable LDAP and NT authentication properly on this module, I was not able to run this using LDAP or NT.

** PLease advise

TIA

Cino

any news on pbi package? I did a new install of 2.1 and can't install the package.. I may follow these step to manually install; http://forum.pfsense.org/index.php/topic,50572.0.html

Nachtfalke

http://lists.pfsense.org/pipermail/dev/2012-June/000178.html

Cino

@Nachtfalke:

http://lists.pfsense.org/pipermail/dev/2012-June/000178.html

thanks. Guess I should had read the whole thing… I missed the bottom part

EDIT: Squid 3 has been built it looks, http://files.pfsense.com/packages/8/All/squid-3.1.19-i386.pbi

EDIT2: Since the package showed up, I installed it... Looks like it needs some options added to it when the pbi is being built:


2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:17 unrecognized: 'sslcrtd_children'
2012/06/18 13:19:24| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/18 13:19:24| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/18 13:19:24| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/18 13:19:24| WARNING: (B) '127.0.0.1' is a subnetwork of (A) '127.0.0.1'
2012/06/18 13:19:24| WARNING: because of this '127.0.0.1' is ignored to keep splay tree searching predictable
2012/06/18 13:19:24| WARNING: You should probably remove '127.0.0.1' from the ACL named 'ext_manager'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:73 unrecognized: 'delay_pools'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:74 unrecognized: 'delay_class'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:75 unrecognized: 'delay_parameters'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:76 unrecognized: 'delay_initial_bucket_level'
2012/06/18 13:19:24| cache_cf.cc(381) parseOneConfigFile: squid.conf:77 unrecognized: 'delay_access'


 2012/06/18 13:24:54| cache_cf.cc(381) parseOneConfigFile: squid-reverse.conf:11 unrecognized: 'netdb_filename'
2012/06/18 13:24:54| cache_cf.cc(381) parseOneConfigFile: squid-reverse.conf:16 unrecognized: 'sslcrtd_children'

It wont start, I manually was able to start squid by taking the unrecognized commands out.. hand edit the squid.inc file so they aren't added

EDIT3: Still testing but looks like option -f will be needed to keep the config files in the same location:

-f file Use given config-file instead of
/usr/pbi/squid-i386/etc/squid/squid.conf

jimp

What build_options were used when making the custom package? I can add whatever is needed to get it building. I tried adding ECAP and that just blew up the build.

If it isn't known, just get /var/db/ports/squid/options from the box that built the current .tbz and post it and I can translate it into the syntax we need.

And yes all packages with config files should be using whatever parameter is there like -f to manually specify where you want the config (should really be /var/etc/something, not /usr/local/etc/something)

Cino

thanks Jim, I'll let Marcelloc charm in on the dev stuff ;-)

marcelloc

Hi jimp,

these are the options on /var/db/ports/squid31/options

# This file is auto-generated by 'make config'.
# No user-servicable parts inside!
# Options for squid-3.1.19
_OPTIONS_READ=squid-3.1.19
WITH_SQUID_KERB_AUTH=true
WITH_SQUID_LDAP_AUTH=true
WITH_SQUID_NIS_AUTH=true
WITH_SQUID_SASL_AUTH=true
WITH_SQUID_IPV6=true
WITH_SQUID_DELAY_POOLS=true
WITH_SQUID_SNMP=true
WITH_SQUID_SSL=true
WITH_SQUID_SSL_CRTD=true
WITH_SQUID_PINGER=true
WITHOUT_SQUID_DNS_HELPER=true
WITH_SQUID_HTCP=true
WITH_SQUID_VIA_DB=true
WITH_SQUID_CACHE_DIGESTS=true
WITHOUT_SQUID_WCCP=true
WITH_SQUID_WCCPV2=true
WITHOUT_SQUID_STRICT_HTTP=true
WITH_SQUID_IDENT=true
WITH_SQUID_REFERER_LOG=true
WITH_SQUID_USERAGENT_LOG=true
WITH_SQUID_ARP_ACL=true
WITH_SQUID_IPFW=true
WITH_SQUID_PF=true
WITHOUT_SQUID_IPFILTER=true
WITH_SQUID_FOLLOW_XFF=true
WITHOUT_SQUID_ECAP=true
WITHOUT_SQUID_ICAP=true
WITHOUT_SQUID_ESI=true
WITH_SQUID_AUFS=true
WITHOUT_SQUID_COSS=true
WITHOUT_SQUID_KQUEUE=true
WITH_SQUID_LARGEFILE=true
WITHOUT_SQUID_STACKTRACES=true
WITHOUT_SQUID_DEBUG=true

jimp

sure your ports tree is up-to-date? (portsnap fetch extract, then go to that port and do 'make config' again) They changed the format of that file recently.

We need the format you posted this time, but I just wanted to make sure you had all of the possible config variables set.

EDIT: Looks like they were all set. I updated the pkg xml, as soon as the builders are done with their current jobs I'll try new builds.