IPsec VTI causing asymmetric traffic?
-
I'm attempting to setup a branch office with IPsec VTI on pfSense for the first time. We had been using Cisco Meraki which is simple to setup. However I figured pfSense could handle the task and then we could save a good chunk of money each year by dropping the Meraki licenses (not to mention how much easier it is to setup complex firewall rules with pfSense)
Setup was pretty straight forward but I'm having an odd issues that I can't seem to resolve myself. Below is a simplified diagram (we have a lot more internal networks than what the diagram shows) of what I'm trying to setup. I would like ALL traffic from Site B to pass-through Site A. Currently everything works as expected except for traffic between Site B and the internet. When I checked the firewall logs at Site A, i'm seeing the traffic being block due to TCP:A (see screenshot below). I have fixed several asymmetric routing issues before with other networks but I just can't seem to wrap my head around whats causing the issue here. Any input would be greatly appreciated.
-
I would packet capture on Site A's IPsec filtering on host 10.85.199.16 and try again and see if you can determine why those ACKs do not match the state of the SYNs. You should see the SYNs there. If you do not there is another path to Site A they are using.
I would also reconsider setting the default gateway to the VTI gateway. That will cover all traffic generated on the firewall, including traffic sourced from Localhost. Since NAT does not currently work on a VTI, anything sourced from the firewall at Site B localhost will route out the VTI but a response will be impossible.
I would, instead, leave the default route as the ISP gateway and policy route all traffic from the 10.85.199.0/24 network to the IPsec VTI. This will mean traffic sourced from the firewall itself (such as DNS resolver recursive queries) will use the local WAN.
If that is not going to cover your use case, I am not sure pfSense is going to be an acceptable solution for you as it exists today.
-
Thanks for the information Derelict! I reconfigured to split tunnel as you suggested to avoid NATing over the VTI and it does work just fine. You said "NAT does not currently work on a VTI", just curious if that functionality is on the road map? I have a few networks that split tunneling would not be an option. Could I just setup IPsec the "non-VTI" way and would that allow NAT to work over the IPsec tunnel?
-
Just a quick update... I was able to get all traffic over the tunnel and NATing properly by setting up IPsec the "non-VTI" way ie. P2 set to 0.0.0.0/0. Everything looks good as far as I can tell, is there any caveats of running a setup this way?
-
You could put a NAT on a policy-based Phase 2 entry. But you really wouldn't need one. The thing that would help you there is you could have a policy to 0.0.0.0/0 from the 10.85.199.0/24 network and it would not match traffic sourced from the firewall (unless explicitly set to source from that interface address) but the net effect would be the same.
-
I wasn't sure if i needed to start a new post but its kind of a continuation of the conversation above. As I had mentioned I did get everything working the way I wanted by setting up the P2s with 0.0.0.0/0. Or so i thought... Here is a bit more detailed diagram that includes additional local networks. Traffic flows as expected between the two sites except for the local networks at Site B. For example a PC at 10.85.201.5 can talk to 8.8.8.8 (through Site A) and 10.10.199.2. But 10.85.201.5 cannot talk to 10.85.202.2, 10.85.199.3 or any other devices on subnets local to Site B. I'm guessing the problem is that traffic is being sent over the IPsec tunnel instead of routing locally first. If that's the case how do i get around that?
-
That's the problem with 0.0.0.0/0 IPsec policies. It's sucking up the local traffic too.
That will probably require some creativity to overcome, like policy routing rules on the LAN interfaces that force the traffic to those subnets to something on that interface - or something along those lines. This will be getting into "hacky, ugly workaround" territory if you can make it work at all. I have never tried it.
The single best way to solve it is to probably put a VPN gateway upstream of the local client firewall/router so traffic between subnets is not subject to that IPsec policy. Only traffic sent to the VPN gateway would be sent across the tunnel.
OpenVPN will do this great, but it is not as performant.
-
Yeah I'm kicking myself for not testing that in the first place. It makes sense, 0.0.0.0/0 does mean ALL traffic... I'm not a fan of workarounds, I've been bitten by them in the past.
Since I have some extra hardware laying around the upstream VPN gateway seems like a good solution. Thanks for your input! -
I just happened to be on the pfsense forum today so I thought I should follow up on this post. I did in fact configure an addition device at each of our locations as a VPN gateway and put it upstream of the local firewall/router and it's an excellent solution for our scenario. Although there was some added cost for the additional hardware it really wasn't that much (we are using PCEngine APU4s for the VPN gateways). We were also replacing some existing Meraki equipment so dropping the licensing for that will more that cover the added hardware cost for the APU4s. Thanks again Derelict for your great advice!
