VPN connections behind pfSense in bridge mode are denied return traffic
-
Edit 1: Originally I thought this was a Cisco AnyConnect problem, but it actually applies to all VPN connections originating from the inside network. UDP return traffic is actively denied. This is also happening with a T-Mobile CellSpot which also works only via VPN.
Edit 2: It is not sufficient to create a blanket UDP/ESP permit (i.e. Permit UDP Source Any, Port 4500, Dest (inside), Port 4500 or Permit ESP Source Any, Dest (inside)). I must define a source IP or Alias (possibly a network, haven't tried). The destination IP can be an IP, Network, or Alias. I've tried "any/any" src/dst but it doesn't work. I have to be specific.
Edit 3: After a LOT of searching I came across this interesting post. Could pfSense be treating all UDP traffic as an unsolicited broadcast? I can understand if the logic is "if traffic is UDP and if traffic qualifies as broadcast" = block. But the traffic in question is neither broadcast, nor unsolicited.
Physical/Virtual Setup
Modem <-> Router <-> ESXi phys port 1 <-> ESXi vSw1 <-> pfSense (VM) <-> ESXi vSW2 <-> ESXi phys port 2 <-> Phys Sw <-> Devices
Logical Setup
pfSense is running in Bridged/Transparent mode. For simplicity's sake, I'm running one flat vlan. IPv4 & IPv6 is working fine. The router hosts DHCPv4/v6-PD, all of that is working fine. All devices within the network are operating normally, there's nothing unusual happening here.
Testing
The AnyConnect VPN session runs only over IPv4; the head-end does not support IPv6.
The inside network contains an outbound permit of source (inside) to destination (any), protocol (any).
If I connect my laptop directly to the modem (Comcast), I can use Cisco AnyConnect VPN.
If I connect my laptop directly to the router (Ubiquiti EdgeRouter X 1.9.7-hf4), I can use Cisco AnyConnect VPN. I did a wireshark packet capture to confirm and provide a baseline for comparison.
If I connect my laptop behind the transparent firewall (pfSense 2.4.2p1), I cannot use Cisco AnyConnect VPN. Wireshark shows UDP traffic going out (4500/443) but no return UDP traffic.
Outbound Firewall Logs
I can see the source (inside) traffic to destination (outside) being permitted "let out anything from firewall host itself (1000016315)" from the laptop to destination port 4500 UDP and 500 UDP.
Inbound Firewall Logs
I can see the source (outside) traffic to destination (inside) being blocked "default deny rule ipv4 (1000000103)." Port 4500 UDP is denied inbound, as well as UDP traffic with no port (the ESP session).
The time difference between these firewall permit/block entries is in the milliseconds. Ultimately I can't figure out why the return traffic is being blocked when it was originally initiated from the inside.
Other Troubleshooting
-
Tried setting firewall optimization to "conservative" (system -> advanced -> firewall) - no effect
-
Tried enabling 'clear invalid DF bits' (system -> advanced -> firewall) - no effect
-
Tried enabling the 'disable pf scrubbing' option (system -> advanced -> firewall) - no effect
My assumption would be that return traffic, if initiated from the inside, would still be in pfSense's state table and permitted. This behavior seems to break that logic.
Workarounds
The only way I've been able to make this work is set up a firewall rule:
Protocol: IPv4 UDP
Source: (vpn server), all ports
Destination: inside network, all ports…which seems pretty ugly to me. I shouldn't have to create a rule like this.
Thoughts, oversimplification?
TCP and UDP flows outbound were created in the state table. Why was only the TCP flow allowed back in?
-
-
Created bug #8247 on the bug tracker:
https://redmine.pfsense.org/issues/8247
-
Following up on this old topic -
I posted the solution here:
https://forum.pfsense.org/index.php?topic=145895.msg793271