IPSec stops routing from working properly?

[silviub]

Hello,

I have the following scenario:
Two PFSense VMs in HA. There's two VLANs (more but for the purpose of this talk, two are important), VLAN 199 on vtnet1 and VLAN 300 on vtnet2.300.
The subnet on VLAN 199 is 10.41.199.0/24 with PFSense 1 having 10.41.199.2/24, PFSense 2 is having 10.41.199.3/24 and the CARP IP is 10.41.199.1
VLAN 300 has 172.16.254.0/29 with PFSense 1: 172.16.254.2/29 and PFSense 2: 172.16.254.3/29, CARP IP being 172.16.254.1/29.

I can ping / curl / whatever from 172.16.254.5/29 to 10.41.199.198/24 with no problems (rules were added, etc).

The second I configure an IPSec with P2 local address (not network, address) 10.41.199.198 and remote network 10.97.2.0/24, routing of 172.16.254.5 to 10.41.199.198 stops. I can't ping it anymore, I can't curl, nothing. The tunnel is not even up, and this happens.

I ran a tcpdump on the active PFSense box and I was able to see the packets on vtnet2.300 but not on vtnet1. I am also not able to see any packets on enc0 and on pflog0 I only see the same packet as on vtnet2.300 but nothing else.

This is a simplified version of what I've got.

VM1 can reach VM2 as long as the P2 is not set. The second I set it, traffic stops flowing.

I have also tried to add an IPSec Bypass Rule under IPSec -> Advanced Settings like:

Source address: 172.16.254.5/32
Destination address: 10.41.199.198/32

But it didn't make any difference.

Any help would be greatly appreciated.

[silviub]

Ok, problem solved. Stupid config, stupid implementation.

So, I have multiple P2s defined:

SRC: 10.41.199.198/32
DST: 10.97.2.0/24

SRC: 10.41.199.199/32
DST: 10.97.2.0/24

SRC: 10.41.199.199/32
DST: 172.16.240.0/20

I did not have Split connections enabled, so my P2s were smushed together, forming something like:

SRC: 10.41.199.198; 10.41.199.199
DST: 10.97.2.0/24; 172.16.240.0/24

WHY?! This is obviously wrong, as I don't have a P2 for 10.41.199.198 -> 172.16.240.0/24. Why would it smush it like that? I would understand to create a single child_SA for

SRC: 10.41.199.199
DST: 10.97.2.0/24; 172.16.240.0/24

and another SA for

SRC: 10.41.199.198
DST: 10.97.2.0/24

[jimp]

That's exactly how it's supposed to work and why the Split Connections option exists.

Combining the P2 entries maximizes stability by reducing the overhead of rekeys/renegotiations, but it has the obvious drawback you encountered so there is the option to split them if necessary. Also there are third party IPsec implementations that prefer one style or the other, so having both maximizes compatibility.

[silviub]

@jimp well, it doesn't really make sense to me. It would have made sense if, as I said, I'd have two different SAs, one for 10.41.199.199 and another for 10.41.199.198. But adding both of them to a single SA, while 10.41.199.198 should not be there, from my point of view, is a big issue, causing routing issues like the one I've just encountered.
I am basically configuring something but IPSec is doing something else.

[jimp]

When they are combined ("un-split") they are put into one single child SA/SPD entry:

<all local P2 addresses/networks> to <all remote P2 addresses/networks>

Any and every combination of local entries to remote entries is valid in that case. Not just the exact sources and destinations you defined. If you want that, they have to be split.

It's not a "big issue", it's just one check box, and most people benefit from the entries being combined far more than need to be concerned about keeping the entries separate. But those who are concerned and just check that one box and get the behavior they want.

[silviub]

@jimp I don't know, I guess we're seeing things differently then, because from my point of view, if I have

Local subnet: 192.168.1.0/24
Remote subnet: 10.10.10.0/24

and

Local subnet: 192.168.2.0/24
Remote subnet: 10.20.10.0/24

I see no benefit in having a single CA that will install an SPD from 192.168.1.0/24 to 10.20.10.0/24, since I don't need that. Can you share a case where that would actually make sense, because I guess I'm missing something..?

[jimp]

I already mentioned why in my replies above, and it's covered in the docs.

To rephrase a bit: Less Child SA/SPD entries make the tunnel more stable and less likely to break on rekey events and when renegotiating. Multiple child SAs behave differently when it comes to P2 auth, rekeying, and other ways. Simpler configuration, more efficient, less to manage.

Both approaches have merit, and the current default is the most optimal for the majority of users.

Again, if you want them separate, just check the box and it does what you think is right.