IPsec VTI tunnel dropping PBR packets on OUT queue

keyser

Hi All.

I have been testing converting an increasingly complicated IPsec S2S tunnelmode tunnel to VTI to "simplify" my routing between two sites (lots of VLANs and subnets on both sites).

I have stumbled on a strange problem - I'm running 25.07.1:

The VTI tunnel works as expected and all subnets on both sides can talk without issue depending on my firewall rules on Enc0. I'm not using the advanced IPsec filtering mode with interface rules as I have a need for Mobile IPsec VPN tunnelmode on both sites.

I have ONE client on site B that I would like to use Internet from Site A, so I created a higher priority firewall rule granting it Internet access with a Policy based Route action using the auto created Site A VTI interface as gateway.
This does not work - the packets are all dropped on the Site B firewall (Errors on OUT Queue for the S2S interface).
I have used packet capture on both boxes, and the SiteB firewall thinks it's sending the policy routed packets correctly (I get them in my capture). But they are not sent - Site A does not receive any packets from the policy route action, and all packets impacted are added to the ERRORS counter on the Site B sending firewalls S2S interface Out Queue. All other packetflows between subnets on the sites works as expected over the very same tunnel.

Any ideas? I have tried creating the floating rule with relaxed interface binding for OUT traffic on the IPsec interface with no success.

keyser

I'm sorry to "summon" you like this @stephenw10 :-)

But you are the oracle on the intricacies of pfSense and its IPsec "behaviour".

Do you have any idea what might be causing what I'm experiencing? Is it a known bug as there seems issues with IPsec VTI in many setups now?

Averlon

Facing the same issue. The configuration was working just fine with CE 2.7.2. With CE 2.8.1, I can only get this work by switching IPsec filter mode to VIT/Transport on assigned interfaces.

keyser

@Averlon Yeah, I noticed around the interweb that I was not alone in facing this issue.

Given the deafening silence on resolutions to this issue, I concluded that It must be a cornercase usage situation (leaving the filtermode in default).

Since giving up the EXCELLENT Mobile Warrior IPSec VPN with OS builtin VPN clients is not an option, that left me we two choices:
Go back to IPSec Tunnelmode Site-to-Site or use another VPN solution for my Site to Site needs.

I choose the latter, and went with Wireguard after doing some serious testing of the “penalty” of no hardware encryption assist (As I’m using a lot of small SG-2100’s). The penalty is not really measurable - well within statistical error - and the performance is great. The gains of having such a simple routed setup, that supports PBR for cross site Internet access, is absolutely fantastic.
Also: Wireguard is much a “cleaner” setup and implementation with an actual interface where everything works as a normal NIC (DNS listener, Filter rules, proper MTU discovery and so on), so I should have just done that from the start. It’s even better that running IPSec in “comparable” advanced filtering mode

My OCD is not happy about using two VPN engines when one would do, but I have to admit the advantages here are overwhelming

keyser

@Averlon One thing I noticed:

I initially made a mistake in my Wireguard setup by only allowing the respective sites subnets in the PEER setups.
So when I tried PBR routing internet traffic through the other site for my selected HOSTS, I had the EXACT same error as when attemting to use IPsec VTI:

Packet captures showed the PBR routing Site B firewall thinking it did send the packets down the Tunnel Interface, but actually the all ended as OUT ERRORS on the Wireguard interface, and Site A never received anything.

I initially panicked and though it was a general PBR error in pfSense, but then I remembered I had to allow 0.0.0.0/0 on the Site A PEER to pass the PBR routed packets.

That does suggest to me that the problem with IPsec VTI is definitively with the last stages of assigning and encrypting the packets in the IPsec Kernel module.

keyser

@Averlon We should probably see if we could get someone like @stephenw10 to verify and validate this bug, as he can describe and create a MUCH more accurate and suggestive redmine ticket :-)

Averlon

@keyser I could also change the connection between the affected sites to Wireguard. The downside is I end up with two VPN Technologies for Site-to-Site connection too, cause not all my devices are Wireguard capable. I also have to evaluate how Wireguard interact with dynamic routing running FRR and especially BGP. It might be worth looking more closely into this and switch to Wireguard where possible. The lack of IP fragmentation support with VTI IPsec is also annoying.

I suspect a sort of regression causing this issue. If we're lucky it's due to changes of default configuration and this may get fixed on the fly. But so far I haven't spotted any, when comparing IPsec related settings between 2.7.2 and 2.8.1.