WPAD Block Port 80 Rule is blocking all of my traffic

pfBasic

MitM should only be required if running a transparent proxy. I'm running an explicit proxy so it should filter http & http/s without any additional configuration. And it was working that way successfully for awhile but stopped working, I just don't know why.

KOM

I would disable pfBlockerNG, Snort or any other packages that might affect you on this.

pfBasic

Unfortunately I have already tried that and nothing changes.

I changed up my firewall rules and disabled the allow LAN and anti lockout rules, added a floating block everything to everything rule then added a pass rule on LAN for ports I need open and simply didn't include 80 or 443 and did include 3218 in the rule. Now the internet works great without ports 80 or 443 (ran comprehensive nmap scan and they are indeed closed) but squid still doesn't show up on lagado, clamAV doesn't work and squid guard doesn't filter anything.

I really don't get it? Is squid just not working? The system shows that it is up and running.

KOM

Now the internet works great without ports 80 or 443 (ran comprehensive nmap scan and they are indeed closed)

Don't confuse WAN and LAN. Of course 80 and 443 will be blocked on WAN – that's normal. You want to block 80 and 443 on LAN to prevent people from not using the proxy.

Is squid just not working?

Probably. Playing with your firewall rules won't fix the actual base problem with squid. SSH in and run:

squid -k check

and see if your config file has any issues. Next, set a client to use the proxy and then run:

tail -f /var/squid/logs/access.log

to view the realtime log while some web activity is happening.

pfBasic

OK will do soon and report back.

For the nmap WAN v LAN, I ran nmap from a computer on my LAN and pointed it towards my pfsense box so shouldn't it be showing me the open ports that my LAN can see? It did report only ports that I had specifically opened on my LAN.

pfBasic

OK, squid -k check returns nothing. I have no idea what this is doing or what I'm expecting to see here? I tried it with both manual configuring proxy and WPAD.

I attached two .txt files showing the tail -f results. One file is with squid automatically detecting WPAD and lagado reporting it as not working (not much in there), the other is with squid manually configured and squid showing working (quite a bit in there). I went to multiple websites both http & http/s in both instances and went to eicar and downloaded sample files. Browser still shows the same behavior with squid working, http eicar is blocked with a warning but http/s eicar is downloaded.

[squid log WPAD configure squid shows NOT WORKING.txt](/public/imported_attachments/1/squid log WPAD configure squid shows NOT WORKING.txt)
[squid log manual configure squid shows WORKING.txt](/public/imported_attachments/1/squid log manual configure squid shows WORKING.txt)

KOM

According to your transparent logs, someone is trying to access http://192.168.1.1:22. This won't work with transparent mode. Transparent mode only supports intercepting ports 80 and 443. I think you can change that but it involves manually adding your own outbound NAT rules for every port you want to handle. Not fun.

You can see from the last 3-4 lines that it's working fine when you actually use a valid port.

pfBasic

The :22 lines are me SSHing in, it's actually not on port 22 but I replaced my actual SSH port with 22 so I wouldn't be posting my SSH port online.

And I'm not running squid in transparent mode, it's setup in explicit mode with WPAD to auto configure.

KOM

I replaced my actual SSH port with 22 so I wouldn't be posting my SSH port online.

Don't bother. If it's accessible via WAN then it's probably being port-scanned a 100 times a day anyway. Trust your security. If you're really paranoid then use 2-factor with a loin and certificate.

OK, sorry for the confusion with your logs.

squid -k check returns nothing. I have no idea what this is doing or what I'm expecting to see here?

Geez, my brain is off today. Try:

squid -k parse

You should get a long list of output. Look for warnings or errors.

Are you running WebGUI in HTTP or HTTPS mode? Is pfSense the web server serving your WPAD files?

pfBasic

OK thanks, the results for that are attached.

I switched the WebGUI to http for WPAD, and pfSense is the server for the WPAD files.

[squid -k parse results.txt](/public/imported_attachments/1/squid -k parse results.txt)

KOM

There appear to be no problems with your squid configuration, according to your output.

pfBasic

Could it be something wrong with the way pfSense is serving up the WPAD files? Or the way I have it setup?

The proxy.pac is located in "/usr/local/www/proxy.pac" and is linked to a wpad.dat & wpad.da file in the same directory.

They all contain the basic configuration:
function FindProxyForURL(url,host)
{
return "PROXY 192.168.1.1:3128";
}

It seems weird to me that traffic doesn't show up on squid when a computer is setup to autoconfigure, but it does when I point the computer to the pfSense box manually?

It also seems weird that clamAV doesn't work on http/s but does on http with an explicit proxy.

KOM

Looks like you're doing everything right. I don't know why it doesn't work for you. I don't run AV on the firewall – too slow and I suspect the defs aren't as up to date as a commercial provider. Get ClamAV off the firewall and use a decent client AV package, if required.

pfBasic

I'll do that, it does allow things down. Right now I'm using it as an easy way to see if I'm under squids umbrella.

Do you know if there's any way for me to check pfsense's file server?

KOM

Do you know if there's any way for me to check pfsense's file server?

Eh, what?

Maybe this might be the issue?

http://kb.k12usa.com/Knowledgebase/Proxy-Auto-Detect-WPAD-Issues-With-IE-Windows-7

pfBasic

I'll look into that but I'm getting the same results across several OS's.

I was just wondering if there was a way for me to see what's happening when a client requests a WPAD or proxy file from the pfsense box.

KOM

I was just wondering if there was a way for me to see what's happening when a client requests a WPAD or proxy file from the pfsense box.

Go to console and look at /var/log/nginx.log and nginx-error.log.

pfBasic

nginx log is large but a notepadd++ search for "wpad" "proxy" ".pac", etc. returns nothing.

nginx error log only has 10 lines:

2016/08/23 15:18:46 [error] 29178#100114: send() failed (54: Connection reset by peer)
2016/08/23 15:26:53 [alert] 29178#100114: close() socket failed (9: Bad file descriptor)
2016/08/25 00:02:41 [error] 32286#100113: send() failed (54: Connection reset by peer)
2016/08/25 00:05:56 [error] 32286#100113: send() failed (54: Connection reset by peer)
2016/08/25 00:05:56 [error] 32286#100113: send() failed (54: Connection reset by peer)
2016/08/25 00:06:33 [error] 32286#100113: send() failed (54: Connection reset by peer)
2016/08/25 00:08:24 [error] 32286#100113: send() failed (54: Connection reset by peer)
2016/08/25 00:08:24 [error] 32286#100113: send() failed (54: Connection reset by peer)
2016/08/25 00:09:36 [error] 32286#100113: send() failed (54: Connection reset by peer)
2016/08/25 00:11:06 [error] 32541#100082: send() failed (54: Connection reset by peer)

I did find some information that might be useful in chrome though, By going to chrome://net-internals/#proxy and chrome://net-internals/#events I was able to see what's going on in chrome when it requests the files and these are the entries I'm seeing on autoconfigure (when the proxy fails).

On the #proxy page I see this:

Effective proxy settings

Use DIRECT connections.
Source: SYSTEM
Original proxy settings

Auto-detect
Source: SYSTEM

On the #events page there are thousands of entries but here are the ones that seem relevant:

230858: URL_REQUEST
http://192.168.1.1/proxy.pac
Start Time: 2016-08-25 00:42:56.954

t=6927008 [st=  0] +REQUEST_ALIVE  [dt=400]
t=6927008 [st=  0]    URL_REQUEST_DELEGATE  [dt=0]
t=6927008 [st=  0]   +URL_REQUEST_START_JOB  [dt=400]
                      --> load_flags = 176 (BYPASS_PROXY | DISABLE_CACHE | DISABLE_CERT_REVOCATION_CHECKING)
                      --> method = "GET"
                      --> priority = "LOWEST"
                      --> url = "http://192.168.1.1/proxy.pac"
t=6927008 [st=  0]      URL_REQUEST_DELEGATE  [dt=0]
t=6927008 [st=  0]      URL_REQUEST_DELEGATE  [dt=0]
t=6927008 [st=  0]     +HTTP_STREAM_REQUEST  [dt=400]
t=6927008 [st=  0]        HTTP_STREAM_REQUEST_STARTED_JOB
                          --> source_dependency = 230859 (HTTP_STREAM_JOB)
t=6927408 [st=400]        CANCELLED
t=6927408 [st=400]     -HTTP_STREAM_REQUEST
t=6927408 [st=400] -REQUEST_ALIVE

230864: URL_REQUEST
http://wpad/wpad.dat
Start Time: 2016-08-25 00:42:57.354

t=6927408 [st=    0] +REQUEST_ALIVE  [dt=21039]
t=6927409 [st=    1]    URL_REQUEST_DELEGATE  [dt=0]
t=6927409 [st=    1]   +URL_REQUEST_START_JOB  [dt=21038]
                        --> load_flags = 176 (BYPASS_PROXY | DISABLE_CACHE | DISABLE_CERT_REVOCATION_CHECKING)
                        --> method = "GET"
                        --> priority = "LOWEST"
                        --> url = "http://wpad/wpad.dat"
t=6927409 [st=    1]      URL_REQUEST_DELEGATE  [dt=0]
t=6927409 [st=    1]      URL_REQUEST_DELEGATE  [dt=0]
t=6927409 [st=    1]     +HTTP_STREAM_REQUEST  [dt=21038]
t=6927409 [st=    1]        HTTP_STREAM_REQUEST_STARTED_JOB
                            --> source_dependency = 230865 (HTTP_STREAM_JOB)
t=6948447 [st=21039]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                            --> source_dependency = 230865 (HTTP_STREAM_JOB)
t=6948447 [st=21039]     -HTTP_STREAM_REQUEST
t=6948447 [st=21039]   -URL_REQUEST_START_JOB
                        --> net_error = -118 (ERR_CONNECTION_TIMED_OUT)
t=6948447 [st=21039]    URL_REQUEST_DELEGATE  [dt=0]
t=6948447 [st=21039] -REQUEST_ALIVE
                      --> net_error = -118 (ERR_CONNECTION_TIMED_OUT)

230865: HTTP_STREAM_JOB
http://wpad/
Start Time: 2016-08-25 00:42:57.355

t=6927409 [st=    0] +HTTP_STREAM_JOB  [dt=21038]
                      --> alternative_service = "Uninitialized :0"
                      --> original_url = "http://wpad/"
                      --> priority = "LOWEST"
                      --> source_dependency = 230864 (URL_REQUEST)
                      --> url = "http://wpad/"
t=6927409 [st=    0]    TCP_CLIENT_SOCKET_POOL_REQUESTED_SOCKET
                        --> host_and_port = "wpad:80"
t=6927409 [st=    0]   +SOCKET_POOL  [dt=21038]
t=6948447 [st=21038]      SOCKET_POOL_BOUND_TO_CONNECT_JOB
                          --> source_dependency = 230866 (CONNECT_JOB)
t=6948447 [st=21038]   -SOCKET_POOL
                        --> net_error = -118 (ERR_CONNECTION_TIMED_OUT)
t=6948447 [st=21038]    HTTP_STREAM_JOB_BOUND_TO_REQUEST
                        --> source_dependency = 230864 (URL_REQUEST)
t=6948447 [st=21038] -HTTP_STREAM_JOB

230866: CONNECT_JOB
wpad:80
Start Time: 2016-08-25 00:42:57.355

t=6927409 [st=    0] +SOCKET_POOL_CONNECT_JOB  [dt=21038]
                      --> group_name = "wpad:80"
t=6927409 [st=    0]   +SOCKET_POOL_CONNECT_JOB_CONNECT  [dt=21038]
t=6927409 [st=    0]     +HOST_RESOLVER_IMPL_REQUEST  [dt=1]
                          --> address_family = 0
                          --> allow_cached_response = false
                          --> host = "wpad:80"
                          --> is_speculative = false
t=6927409 [st=    0]        HOST_RESOLVER_IMPL_IPV6_REACHABILITY_CHECK
                            --> cached = true
                            --> ipv6_available = true
t=6927409 [st=    0]        HOST_RESOLVER_IMPL_CREATE_JOB
t=6927409 [st=    0]        HOST_RESOLVER_IMPL_JOB_ATTACH
                            --> source_dependency = 230867 (HOST_RESOLVER_IMPL_JOB)
t=6927410 [st=    1]     -HOST_RESOLVER_IMPL_REQUEST
t=6948447 [st=21038]   -SOCKET_POOL_CONNECT_JOB_CONNECT
                        --> net_error = -118 (ERR_CONNECTION_TIMED_OUT)
t=6948447 [st=21038] -SOCKET_POOL_CONNECT_JOB

230867: HOST_RESOLVER_IMPL_JOB
wpad
Start Time: 2016-08-25 00:42:57.355

t=6927409 [st=0] +HOST_RESOLVER_IMPL_JOB  [dt=1]
                  --> host = "wpad"
                  --> source_dependency = 230866 (CONNECT_JOB)
t=6927409 [st=0]    HOST_RESOLVER_IMPL_JOB_STARTED
t=6927409 [st=0]   +HOST_RESOLVER_IMPL_PROC_TASK  [dt=1]
t=6927409 [st=0]      HOST_RESOLVER_IMPL_ATTEMPT_STARTED
                      --> attempt_number = 1
t=6927409 [st=0]      HOST_RESOLVER_IMPL_JOB_REQUEST_ATTACH
                      --> priority = "LOWEST"
                      --> source_dependency = 230866 (CONNECT_JOB)
t=6927410 [st=1]      HOST_RESOLVER_IMPL_ATTEMPT_FINISHED
                      --> attempt_number = 1
t=6927410 [st=1]   -HOST_RESOLVER_IMPL_PROC_TASK
                    --> address_list = ["192.168.1.1:0"]
t=6927410 [st=1] -HOST_RESOLVER_IMPL_JOB

230868: SOCKET
wpad:80
Start Time: 2016-08-25 00:42:57.356

t=6927410 [st=    0] +SOCKET_ALIVE  [dt=21037]
                      --> source_dependency = 230866 (CONNECT_JOB)
t=6927410 [st=    0]   +TCP_CONNECT  [dt=21037]
                        --> address_list = ["192.168.1.1:80"]
t=6927410 [st=    0]     +TCP_CONNECT_ATTEMPT  [dt=21037]
                          --> address = "192.168.1.1:80"
t=6948447 [st=21037]     -TCP_CONNECT_ATTEMPT
                          --> os_error = 10060
t=6948447 [st=21037]      SOCKET_CLOSED
t=6948447 [st=21037]   -TCP_CONNECT
                        --> net_error = -118 (ERR_CONNECTION_TIMED_OUT)
t=6948447 [st=21037] -SOCKET_ALIVE

230869: CONNECT_JOB
wpad:80
Start Time: 2016-08-25 00:42:57.606

t=6927660 [st=    0] +SOCKET_POOL_CONNECT_JOB  [dt=21000]
                      --> group_name = "wpad:80"
t=6927660 [st=    0]    BACKUP_CONNECT_JOB_CREATED
t=6927660 [st=    0]   +SOCKET_POOL_CONNECT_JOB_CONNECT  [dt=21000]
t=6927660 [st=    0]     +HOST_RESOLVER_IMPL_REQUEST  [dt=0]
                          --> address_family = 0
                          --> allow_cached_response = false
                          --> host = "wpad:80"
                          --> is_speculative = false
t=6927660 [st=    0]        HOST_RESOLVER_IMPL_IPV6_REACHABILITY_CHECK
                            --> cached = true
                            --> ipv6_available = true
t=6927660 [st=    0]        HOST_RESOLVER_IMPL_CREATE_JOB
t=6927660 [st=    0]        HOST_RESOLVER_IMPL_JOB_ATTACH
                            --> source_dependency = 230870 (HOST_RESOLVER_IMPL_JOB)
t=6927660 [st=    0]     -HOST_RESOLVER_IMPL_REQUEST
t=6948660 [st=21000]   -SOCKET_POOL_CONNECT_JOB_CONNECT
                        --> net_error = -118 (ERR_CONNECTION_TIMED_OUT)
t=6948660 [st=21000] -SOCKET_POOL_CONNECT_JOB

230870: HOST_RESOLVER_IMPL_JOB
wpad
Start Time: 2016-08-25 00:42:57.606

t=6927660 [st=0] +HOST_RESOLVER_IMPL_JOB  [dt=0]
                  --> host = "wpad"
                  --> source_dependency = 230869 (CONNECT_JOB)
t=6927660 [st=0]    HOST_RESOLVER_IMPL_JOB_STARTED
t=6927660 [st=0]   +HOST_RESOLVER_IMPL_PROC_TASK  [dt=0]
t=6927660 [st=0]      HOST_RESOLVER_IMPL_ATTEMPT_STARTED
                      --> attempt_number = 1
t=6927660 [st=0]      HOST_RESOLVER_IMPL_JOB_REQUEST_ATTACH
                      --> priority = "LOWEST"
                      --> source_dependency = 230869 (CONNECT_JOB)
t=6927660 [st=0]      HOST_RESOLVER_IMPL_ATTEMPT_FINISHED
                      --> attempt_number = 1
t=6927660 [st=0]   -HOST_RESOLVER_IMPL_PROC_TASK
                    --> address_list = ["192.168.1.1:0"]
t=6927660 [st=0] -HOST_RESOLVER_IMPL_JOB

230871: SOCKET
wpad:80
Start Time: 2016-08-25 00:42:57.606

t=6927660 [st=    0] +SOCKET_ALIVE  [dt=21000]
                      --> source_dependency = 230869 (CONNECT_JOB)
t=6927660 [st=    0]   +TCP_CONNECT  [dt=21000]
                        --> address_list = ["192.168.1.1:80"]
t=6927660 [st=    0]     +TCP_CONNECT_ATTEMPT  [dt=21000]
                          --> address = "192.168.1.1:80"
t=6948660 [st=21000]     -TCP_CONNECT_ATTEMPT
                          --> os_error = 10060
t=6948660 [st=21000]      SOCKET_CLOSED
t=6948660 [st=21000]   -TCP_CONNECT
                        --> net_error = -118 (ERR_CONNECTION_TIMED_OUT)
t=6948660 [st=21000] -SOCKET_ALIVE

I'm not really sure what I'm reading here but it looks like the connection is timing out and it retries a few different ways but it never works.

Any new guidance based on this?

KOM

No idea but perhaps you're having the issue of nginx not being able to serve the files due a MIME type issue?

https://forum.pfsense.org/index.php?topic=109190.0

pfBasic

Thank you, I did this and it's the closest thing to working squid that I've seen yet. With these changes both lagado and chrome detect the proxy settings.
Unfortunately, squid (or at least squidguard) doesn't work.

If port 80 is open, then proxy files are transmitted, autoconfigure is completed and lagado and chrome report using the proxy settings. However, everything appears to be bypassing the proxy somehow? Apparently nginx is opening its listening port (nmap reports port 80 on my pfSense box opened by nginx) on my LAN, because with that configuration enabled, port 80 is opened, if I change the listen port, then that port is open (my rules block ports 80 and 443 except in a few specific circumstances). I still don't understand how this is allowing SSL (443 is closed, nginx didn't open it, and nmap doesn't report it opened) but not applying squidguard rules to SSL?

If I close port 80 after the proxy file has been downloaded, then it simply destroys the internet connection,

I tried forwarding all port 80/443 traffic to 127.0.0.1 on 3128 to force http&/s traffic to squid, but that didn't work either.

Any suggestions?

At this point I'd also be interested in a way to use shallalist on pfBlockerNG…. pfBNG does everything that I want from squid except shallalist, and it just works with no issues.