Unwanted blocking outbound WAN traffic to one website
-
I have had PFSense setup and running for years with little to no issues. Recently I've had odd issues coming up where users are unable to reach certain websites. No configuration changes had been made to the server for at least a month beforehand. Last week it was google calendar. Now it is facebook. The common theme is that both were connecting using SSL though i don't know if that's and issue. I'm at wits end with this and hoping someone can help.
Here is the current behavior with facebook.
Able to Ping facebook.com from inside the LAN and come back with replies from IP 31.13.72.36 - seems normal.
Browse to https://www.facebook.com and connection times out (any computer on the network)I checked the firewall logs and found this log entry showing the packet being blocked coming from the public IP of my pfsense box destined for facebook on port 443.
Looking at the rule action on the left side I get this information in the window.
If I'm doing it correctly, I'm tracing the blocking rule back on my WAN rules and finding rule 65 which is:
I'm assuming this is the same rule 65 by what is in the querystring "firewall_rules_edit.php?id=65"
This rule has been in my ruleset forever to only allow access to my web interface for pfsense config from set IP addresses on the WAN interface. The "Management Ports" alias is ports 22 and 443. Just for fun I tried disabling the rule and it makes no difference. The firewall logs still show that the traffic is blocked by rule 65. It doesn't even make sense to me that these rules should be in play because it should only be running the rules on inbound, not outbound traffic on the interface, and also the traffic should have a state created when it goes out, so the firewall shouldn't be seeing it as a brand new connection attempt.
One thing to note with this firewall is that I have tried testing out using snort on it in the past. It was installed on the firewall previously, and when doing a system update my firewall ran out of disk space and some of the packages got half installed with no way to uninstall them. In trying to troubleshoot this issue, I reinstalled pfsense on the box, and reloaded the configuration file that did not have the package information in it. My hope was that if there was some dangling snort stuff in there it would get rid of it. It didn't change at all the issue we're experiencing with Facebook today though.
I have also tried:
Turning off Hardware Checksums
Telling NAT to not drop packets with invalid DF bit set under Advanced
Disabling Firewall ScrubI would appreciate some pointers in what direction to go next. This server has multiple lans, vpns, etc connected to it and I don't want to have to reconfigure from the ground up.
-
Do you have squid proxy installed? When you're having problems, is it just the one HTTPS site at a time or all of them?
-
@KOM:
Do you have squid proxy installed? When you're having problems, is it just the one HTTPS site at a time or all of them?
Today the problem is only with https://www.facebook.com as far as I'm aware. All other sites, including https sites are working normally.
I do not have squid proxy installed (or it shouldn't be as far as I can see)
-
Yeah that is not the rule that is doing the blocking.
The rule in the logs is one of the anti-spoofing rules
That one in particular will block traffic coming into any interface except fxp1 with a source address in X.X.183.152/29
So the real question appears to be why is traffic arriving coming into another interface with a source address contained in your outside /29 on fxp1 ??
what is the fxp1 interface?
-
Thank you for having some understanding in this area to help me :)
fxp1 is my WAN interface. That NIC has the default network which is my WAN connected to the gateway, and also has a vlan on it that allows connection to manage some devices with private IP addresses in between the pfsense box and the ISP modem. fxp1 = CHARTERWAN in the picture of the firewall log entry below.
So if I understand you correctly, a packet is coming in on my WAN interface from my WAN subnet and the anti spoofing rules are blocking the packet because it thinks its on another interface than my WAN even though it is on my WAN?
This is why I'm scratching my head.
An update to this is that crashplan is also being blocked today along with facebook. I have packet captures I could share, but I'm not sure how to obfuscate sensitive info in them at the moment.
-
Doing some more checking this morning and I find that also when I ping the address that is blocked in the logs by the anti-spoofing rule is also blocked. The IP address I'm pinging and is blocked is 157.240.0.35 When I ping facebook.com from my laptop it replies with 31.13.72.36 and does not get blocked. Pinging www.facebook.com returns and is blocked.
Jusf for fun I ping from my laptop 157.240.0.34 and 157.240.0.36 and the traffic is not blocked.
-
Hmm. Feel free to send the output of these two commands to me in a PM
Diagnostics > Command Prompt
Execute these and copy/paste the results
cat /tmp/rules.debug
pfctl -sr
-
I just wanted to post an update and resolution to my issue. I worked with Derelict who kindly reviewed my logs, debug info and packet captures and helped me come to a conclusion.
What I found in the packet captures was that the packets going to certain public IP addresses were being rerouted by my cable modem back to the interface on my firewall server and thus being blocked by the anti spoofing rules. I'm posting this in case it helps anyone else. What I saw was this…
SourceIP -> DestinationIP ------ Source MAC -> Destination MAC
PFSense -> FacebookIP ------ PFSense MAC - > CableModemMAC
PFSense -> FacebookIP ------ CableModemMAC - > PFSense MAC
PFSense -> FacebookIP ------ PFSense MAC - > CableModemMAC
PFSense -> FacebookIP ------ CableModemMAC - > PFSense MAC
PFSense -> FacebookIP ------ PFSense MAC - > CableModemMAC
PFSense -> FacebookIP ------ CableModemMAC - > PFSense MACThis is a simplification of the packet capture, but you can see the source and destination IPs stay the same but the mac addresses keep alternating. The best I can tell my cable modem thought that the facebook ip address needed to be routed to my PFSense box.
The modem is at a remote site and I didn't have access to reboot it previously. I called my cable company and they rebooted it and the issue has been resolved as of now.