Some traffic from Windows Server 2008 being blocked
I recently put pfSense 2.0.3 into production but had to disable it because of an odd problem. I run a small hosting company and some legitimate traffic from some servers to some destinations was being blocked by pfSense.
There are two things that make this really odd:
It only appears to effect sites hosted on Windows Server 2008 or Windows Server 2008 R2 (I don't have anything newer). Sites on Windows Server 2003 R2 and earlier don't appear to be effected. I came to this conclusion by copying one of the effected web pages from a 2008 R2 server to a test server running 2003 R2 and could not reproduce the problem while the same exact webpage on 2008 R2 servers didn't work. Copying the page to another 2008 R2 server confirmed it.
It doesn't effect all traffic to these sites. Some clients are able to access these sites without problems. But others either experience random timeouts or are unable to access the sites at all.
It doesn't effect all pages on the sites that are effected.
I'm not sure but the problem might be related to HTTPS traffic. What's really frustrating is that nothing is being logged (pfSense is configured to log packets blocked by the default rule).
My LAN interface simply has a pass any protocol from any source to any destination rule. On the WAN interface I tried creating a pass any protocol from any source to IPs on one of the effected servers with logging. This didn't resolve the problem and didn't log any of the traffic that was being blocked. The only way I could resolve the problem was to "Disable all packet filtering" under System/Advanced/Firewall and Nat.
Any ideas? Is there another level of logging that can be enabled to try and diagnose the problem?
To try and figure out what's causing my problem I ran a test with 2 instances of WireShark capturing the traffic, one on each side of the firewall.
My test involved an ASP.net application that made an outbound HTTPS request from 184.108.40.206 to 220.127.116.11. It appears that pfSense dropped the SYN ACK packet that comes back in on the WAN connection. Below is a summary of the SYN and SYN, ACK from Wireshark on the WAN side of pfSense. On the LAN side I only see the SYN followed by retransmissions of the SYN.
612 31:20.3 18.104.22.168 22.214.171.124 TCP 66 61487 > https [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
613 31:20.3 126.96.36.199 188.8.131.52 TCP 60 https > 61487 [SYN, ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460
I'd really appreciate any help in understanding what's going on and how I can get this traffic through pfSense.