Guide to filtering web content (http and https) with pfsense 2.3

[techbee]

aGeekHere,

My wpad now works.  I can download it from chrome browser.
I can ping and nslookup wpad now.
All squidguard group ACL and its target rules are applied to block facebook and youtube for testing purposes for now.
I HAVE NOT INSTALLED THE CA CERT ON CLIENT DEVICES.

I used http and https filtering with splice all option so that I dont need to install the cert to client devices.

I also cleared browser cache and history.

The result is I can browse http but cannot browse https.  Https sites like yahoo and google mail works though.

I have screenshot of the error for browsing https site.

[aGeekhere]

Is squidguard blocking it by mistake? is it set to deny all?

Try allowing all in squidguard rules to test.

Post screen shots of squidguard.

Try clearing states in pfsense.

Reboot.

[techbee]

Squidguard is not blocking by mistake.

I tried clearing firewall states and Rebooted, still the same situation.

[techbee]

I also have this wireshark done where I can see that it made a CONNECT method to facebook, then it has established connection, send client hello but later have RST and there is no SERVER HELLO thus no application data received.

What could I miss here.

[aGeekhere]

lets try without the wpad and use just the transparent, set proxy direct does it work then?
If you still cannot access https sites then post you squid config.

[techbee]

@aGeekHere:

lets try without the wpad and use just the transparent, set proxy direct does it work then?
If you still cannot access https sites then post you squid config.

If I use only the http proxy without the ssl mitm and set browser to auto, then the http proxy filter works.

If you are saying to setup http transparent proxy and ssl mitm filtering and install cert to client device and set browser proxy, then I have not done this yet.

[aGeekhere]

and install cert to client device and set browser proxy, then I have not done this yet.

Just use splice all for mitm (and use the cert created in pfsense under SSL Man In the Middle Filtering\CA). 
Set browser to direct.
This will use transparent proxy for both http and https.

If this works then there is an issue with your WPAD.
If this does not work then something is broken.

[techbee]

@aGeekHere:

and install cert to client device and set browser proxy, then I have not done this yet.

Just use splice all for mitm (and use the cert created in pfsense under SSL Man In the Middle Filtering\CA). 
Set browser to direct.
This will use transparent proxy for both http and https.

If this works then there is an issue with your WPAD.
If this does not work then something is broken.

What do you mean for "Set browser to direct", is this the automatic detect settings ?

On the other hand, my wpad only have the following inside:

function FindProxyForURL(url,host)
{
return "PROXY 192.168.1.1:3128";
}

I am not sure what you example below means so I use the example from pfsense wiki.

function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";

return "PROXY 192.168.1.1:3128";
}

[aGeekhere]

yeah turn off automatic detect setting so the traffic goes direct and gets intercepted by the transparent proxy.

[techbee]

@aGeekHere:

yeah turn off automatic detect setting so the traffic goes direct and gets intercepted by the transparent proxy.

By turn off, you mean uncheck the automatic detect settings and others options as well ?

[aGeekhere]

yes

[techbee]

May I ask though, what is the exact version of pfsense that you are using just for comparison.  If this will not work, then I will need to reinstall my pfsense box, maybe its broken.

I am using:

2.3.4-RELEASE (amd64)
built on Wed May 03 15:13:29 CDT 2017
FreeBSD 10.3-RELEASE-p19

On the other hand, below is the screen shot of my testing NOT USING WPAD but setup the browser proxy.

Also, I tried direct approach that you said, and the result the same error as before.

What exact pfsense version are you using ?

[aGeekhere]

2.3.4-RELEASE (amd64)
Something is broken.

[techbee]

@aGeekHere:

2.3.4-RELEASE (amd64)
Something is broken.

what is in your wpad file?

did you install your pfsense or upgraded to 2.3.4 release?

[aGeekhere]

wpad is in the guide.

I have been upgrading since i joined.

[4Qman]

Hi,

I am new to pfsense, can someone have a look at the attached image and tell me if it looks correct please.


[4Qman]

I have followed this guide.

When I run proxy test it shows that I am under proxy. Google works fine, and so do most websites. But such websites as Amazon won't load, gives error.

Any advice would be greatly appreciated.

[aGeekhere]

Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

[4Qman]

@aGeekHere:

Post your squid config. Are you using just WPAD or Transparent proxy with mitm (cert of all devices) or Transparent proxy mitm set to splice all?

The best setup i found to use is WPAD and transparent proxy with mitm splice all, set devices to auto config.

Hi thanks for your reply.

I followed the guide, page 1.  I copied and pasted logs from PackageSquidGuardLogs

10.07.2017 17:06:47 [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 17:05:35 [squid_reconfigure] Add new redirector options to Squid config.
10.07.2017 17:05:35 [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 17:05:35 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
10.07.2017 17:05:35 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
10.07.2017 17:05:35 [sg_create_config] Add Default
10.07.2017 17:05:35 [sg_create_config] Add rewrites: safesearch;
10.07.2017 17:05:35 [sg_create_config] Add destinations: whitelist; blacklist;
10.07.2017 17:05:35 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
10.07.2017 17:05:35 [sg_create_config] Add blacklist entries
10.07.2017 17:05:35 [squidguard_rebuild_db] Start rebuild DB.
10.07.2017 17:05:24 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
10.07.2017 17:05:24 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
10.07.2017 17:05:24 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
10.07.2017 17:05:24 [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
10.07.2017 17:05:24 [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
10.07.2017 17:05:24 [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
10.07.2017 17:05:24 [sg_reconfigure_user_db] Add user entries
10.07.2017 17:05:24 [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
10.07.2017 16:36:15 [squid_reconfigure] Add new redirector options to Squid config.
10.07.2017 16:36:15 [squid_reconfigure] Remove old redirector options from Squid config.
10.07.2017 16:36:15 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
10.07.2017 16:36:15 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
10.07.2017 16:36:15 [sg_create_config] Add Default
10.07.2017 16:36:15 [sg_create_config] Add rewrites: safesearch;
10.07.2017 16:36:15 [sg_create_config] Add destinations: whitelist; blacklist;
10.07.2017 16:36:15 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
10.07.2017 16:36:15 [sg_create_config] Add blacklist entries
10.07.2017 16:36:15 [squidguard_rebuild_db] Start rebuild DB.
10.07.2017 16:36:05 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
10.07.2017 16:36:05 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
10.07.2017 16:36:05 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.
10.07.2017 16:36:05 [sg_create_simple_config] Warning Ignored empty item 'whitelist' = '/var/db/squidGuard/whitelist'.
10.07.2017 16:36:05 [sg_create_simple_config] Begin with dbhome='/var/db/squidGuard'.
10.07.2017 16:36:05 [squidguard_rebuild_db] Begin with path '/var/db/squidGuard'.
10.07.2017 16:36:05 [sg_reconfigure_user_db] Add user entries
10.07.2017 16:36:05 [sg_reconfigure_user_db] Begin with '/var/db/squidGuard'
09.07.2017 18:01:06 [squid_reconfigure] Add new redirector options to Squid config.
09.07.2017 18:01:06 [squid_reconfigure] Remove old redirector options from Squid config.
09.07.2017 18:01:06 [sg_reconfigure] Save squidGuard config to '/usr/local/etc/squidGuard/squidGuard.conf'.
09.07.2017 18:01:06 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
09.07.2017 18:01:06 [sg_create_config] Add Default
09.07.2017 18:01:06 [sg_create_config] Add rewrites: safesearch;
09.07.2017 18:01:06 [sg_create_config] Add destinations: whitelist; blacklist;
09.07.2017 18:01:06 [sg_create_config] Added: blk_BL_adv; blk_BL_aggressive; blk_BL_alcohol; blk_BL_anonvpn; blk_BL_automobile_bikes; blk_BL_automobile_boats; blk_BL_automobile_cars; blk_BL_automobile_planes; blk_BL_chat; blk_BL_costtraps; blk_BL_dating; blk_BL_downloads; blk_BL_drugs; blk_BL_dynamic; blk_BL_education_schools; blk_BL_finance_banking; blk_BL_finance_insurance; blk_BL_finance_moneylending; blk_BL_finance_other; blk_BL_finance_realestate; blk_BL_finance_trading; blk_BL_fortunetelling; blk_BL_forum; blk_BL_gamble; blk_BL_government; blk_BL_hacking; blk_BL_hobby_cooking; blk_BL_hobby_games-misc; blk_BL_hobby_games-online; blk_BL_hobby_gardening; blk_BL_hobby_pets; blk_BL_homestyle; blk_BL_hospitals; blk_BL_imagehosting; blk_BL_isp; blk_BL_jobsearch; blk_BL_library; blk_BL_military; blk_BL_models; blk_BL_movies; blk_BL_music; blk_BL_news; blk_BL_podcasts; blk_BL_politics; blk_BL_porn; blk_BL_radiotv; blk_BL_recreation_humor; blk_BL_recreation_martialarts; blk_BL_recreation_restaurants; blk_BL_recreation_sports; blk_BL_recreation_travel; blk_BL_recreation_wellness; blk_BL_redirector; blk_BL_religion; blk_BL_remotecontrol; blk_BL_ringtones; blk_BL_science_astronomy; blk_BL_science_chemistry; blk_BL_searchengines; blk_BL_sex_education; blk_BL_sex_lingerie; blk_BL_shopping; blk_BL_socialnet; blk_BL_spyware; blk_BL_tracker; blk_BL_updatesites; blk_BL_urlshortener; blk_BL_violence; blk_BL_warez; blk_BL_weapons; blk_BL_webmail; blk_BL_webphone; blk_BL_webradio; blk_BL_webtv; .
09.07.2017 18:01:06 [sg_create_config] Add blacklist entries
09.07.2017 18:01:06 [squidguard_rebuild_db] Start rebuild DB.
09.07.2017 18:00:56 [squidguard_rebuild_db] Create rebuild config '/usr/local/etc/squidGuard/squidGuard__usrdbrebuild.conf'.
09.07.2017 18:00:56 [sg_redirector_base_url] Select redirector base url (http://192.168.1.1:80/sgerror.php?url=403%20404&a=%a&n=%n&i=%i&s=%s&t=%t&;u=%u)
09.07.2017 18:00:56 [sg_create_simple_config] Warning Ignored empty item 'blacklist' = '/var/db/squidGuard/blacklist'.

Proxy Config # This file is automatically generated by pfSense

Do not edit manually !

http_port 192.168.1.1:3128
http_port 127.0.0.1:3128
icp_port 0
digest_generation off
dns_v4_first off
pid_filename /var/run/squid/squid.pid
cache_effective_user squid
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr drobinson@brightsure.co.uk
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
netdb_filename /var/squid/logs/netdb.state
pinger_enable off
pinger_program /usr/local/libexec/squid/pinger

logfile_rotate 2
debug_options rotate=2
shutdown_lifetime 3 seconds

Allow local network(s) on interface(s)

acl localnet src  192.168.1.0/24 127.0.0.0/8
forwarded_for on
httpd_suppress_version_string on
uri_whitespace strip

cache_mem 1000 MB
maximum_object_size_in_memory 256 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
minimum_object_size 0 KB
maximum_object_size 1000 MB
cache_dir ufs /var/squid/cache 3000 16 256
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all

Add any of your own refresh_pattern entries above these.

refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|?) 0  0%  0
refresh_pattern .    0  20%  4320

#Remote proxies

Setup some default acls

ACLs all, manager, localhost, and to_localhost are predefined.

acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 3129 1025-65535
acl sslports port 443 563

acl purge method PURGE
acl connect method CONNECT

Define protocols used for redirects

acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

Always allow localhost connections

http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

Reverse Proxy settings

Custom options before auth

Setup allowed ACLs

Allow local network(s) on interface(s)

http_access allow localnet

Default block all to be sure

http_access deny allsrc

[aGeekhere]

if you set squidguard to allow all do still get the issue?