TCP traffic over IPSec stalls with some packets not appearing on enc0

Pentium100

Sorry for double post, but I found something else interesting.
I am not too familiar with BSD, I am more familiar with Linux, but this looks weird to me:

If I start the iperf3 test, then while it is running (stalled transfer) I run
pfctl -s state | grep 192.168.0.x | grep 192.168.10.x | grep 5201
I get this:

vtnet2 tcp 192.168.0.x:5201 <- 192.168.10.x:42202       ESTABLISHED:ESTABLISHED
ipsec30 tcp 192.168.10.x:42202 -> 192.168.0.x:5201       SYN_SENT:CLOSED
all tcp 192.168.10.x:42202 -> 192.168.0.x:5201       ESTABLISHED:ESTABLISHED

So, on "all" and vtnet2 it says ESTABLISHED, but on ipsec30 it says CLOSED.
Are the states supposed to look like this?

I checked this on another pfsense router (older version 2.6.0) and all states there are on "all" and not separate interfaces. What is the difference here?

tinfoilmatt

What does the firewall ruleset look like on both IPsec and IPsec30 tabs?

I checked this on another pfsense router (older version 2.6.0) and all states there are on "all" and not separate interfaces.

Obvious question, but is that system also using Routed IPsec or no?

Pentium100

@tinfoilmatt said in TCP traffic over IPSec stalls with some packets not appearing on enc0:

Obvious question, but is that system also using Routed IPsec or no?

Yes. It's the only other pfsense with routed IPsec..

@tinfoilmatt said in TCP traffic over IPSec stalls with some packets not appearing on enc0:

What does the firewall ruleset look like on both IPsec and IPsec30 tabs?

IPSec tab has only one rule

ipsec30 interface does not have its own firewall tab. That's probably because I have this setting on:

This is because there are policy-based IPsec tunnels too, so I cannot change it.
This setting is the same on both routers, as both have a combination of routed IPsec and policy-based.

tinfoilmatt

ipsec30 interface does not have its own firewall tab. That's probably because I have this setting on:

So then you're aware that that's literally the setting preventing you from using "per-interface rules, NAT, or reply-to" reliably.

Maybe there's a way to make this behavior more reliable/consistent using static routing?

Pentium100

@tinfoilmatt Changing the setting will break the policy-based VPNs.

I do not really need per-interface rules, as I just have allowed everything in the IPsec tab and it works correctly on the other router with an older pfsense version.

As I only have two such routers I cannot say if the other one works because of the version of something else.

tinfoilmatt

The issue is not the firewall ruleset.

tinfoilmatt

Why does this P2 require "Routed (VTI)" mode?

tinfoilmatt

And can we assume you've read this section of the documentation, IPsec Interface Assignment?

Pentium100

@tinfoilmatt
Because using policy based it would be something like 30 phase2 entries and we had a problem where some of them would stop working at random.

Anyway, I found a solution to this.
Firewall State Policy - set to "Floating States" (default was "Interface Bound States" and apparently this default was different on the older pfsense version)
Now it seems to work

tinfoilmatt

If it works and it's messy, it still works.

Pentium100

@tinfoilmatt Yeah. I have to use routed ipsec for this and also have to use policy-based tunnels. If there is no other way to make this work, then using floating states (like it is in the older version) seems like a solution.

tinfoilmatt

If there is no other way to make this work

I think there might be. I'd need to confirm for myself to be sure.