Routed IPsec using if_ipsec VTI interfaces

[jimp]

Granted I haven't tried it on an HA pair but nothing has changed in a couple days that would affect that. When was the last snapshot you had working there?

Can you show the conXXXX entry from /var/etc/ipsec/ipsec.conf for that tunnel?

[obrienmd]

Non-HA side:

conn con2
        fragmentation = yes
        keyexchange = ikev2
        reauth = yes
        forceencaps = no
        mobike = no

        rekey = yes
        installpolicy = no

        dpdaction = restart
        dpddelay = 10s
        dpdtimeout = 60s
        auto = start
        left = {non_ha_side_wan_ip}
        right = {ha_side_wan_ip}
        leftid = {non_ha_side_wan_ip}
        ikelifetime = 28800s
        lifetime = 3600s
        ike = aes256-sha1-modp1024!
        esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
        leftauth = psk
        rightauth = psk
        rightid = {ha_side_wan_ip}
        rightsubnet = 10.90.91.1
        leftsubnet = 10.90.91.2/30

HA side:

conn con1
        fragmentation = yes
        keyexchange = ikev2
        reauth = yes
        forceencaps = no
        mobike = no

        rekey = yes
        installpolicy = no

        dpdaction = restart
        dpddelay = 10s
        dpdtimeout = 60s
        auto = start
        left = {ha_side_wan_ip}
        right = {non_ha_side_wan_ip}
        leftid = {ha_side_wan_ip}
        ikelifetime = 28800s
        lifetime = 3600s
        ike = aes256-sha1-modp1024!
        esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
        leftauth = psk
        rightauth = psk
        rightid = {non_ha_side_wan_ip}
        rightsubnet = 10.90.91.2
        leftsubnet = 10.90.91.1/30

[jimp]

Those don't look quite right, there is no reqid in those blocks like there should be. Are the ipsecX interfaces actually present?

Might be due to the tunnel being IKEv2, I think all three of my test systems here have been IKEv1. I'll try to spin up a v2 set.

[obrienmd]

Yup, ipsec1000/2000 (depending on box) ints are there, and show proper /30s.

[jimp]

I did see one problem come up that I just pushed a fix for, but I didn't see that specific error you had unless I had an IKEv1/IKEv2 mismatch between the peers.

The fix I made only touches two lines, you can easily apply it manually to test: https://github.com/pfsense/pfsense/commit/d4b43c48ed1636d3fcd6e47d73ba721bd63d883a

With that I just switched both sides from IKEv1 to IKEv2 and it came right back up.

[obrienmd]

Yep, nailed it. Looking good with that change.

Because of your warning on frr, I'm testing with static routing right now. After everything was fixed and I disabled / re-enabled the interfaces to get traffic flowing, static routes were showing in the route table but set to hn1 rather than the ipsec interface. Editing and re-saving the static route resolved the issue.

With dynamic routing I bet I won't see that in the future, but if there's some resiliency code somewhere to reset interfaces on static routes when gateways disappear/appear, go up/down, go pending, etc... Perhaps something needs to get tweaked there.

[jimp]

I need to work a bit on static routes yet. I had it solved and working on reboot but somewhere in my changes this week that appears to have broken again as I am not seeing my routes in the table after it boots up. I need to investigate more and open another issue up for that.

FRR should be better next week, see my updates on https://redmine.pfsense.org/issues/8449#note-2

[obrienmd]

Great, thanks Jim.

[obrienmd]

Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

[jimp]

@obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

Not without loading it up and seeing what's in /etc/version.lastcommit. Servers are using CDT.

[jimp]

Static routes should be OK now. I'm not quite sure how it worked before, given the changes I had to make, but it's working now.

https://github.com/pfsense/pfsense/commit/0aa52fb21a21f58035f2e2fe3b9328a9c175ffb5

I think that might be most if not all of the functional issues. There are still some anti-foot-shooting measures I need to take like preventing removing an IPsec tunnel or P2 used as a VTI interface.

[obrienmd]

On latest devel for factory and CE, everything functionally is looking great. Had to restart *pinger (I forget which one is used these days) for gateways to get out of pending after initial interface bring-up, but packets are all flowing, no weird state issues, very solid :)

[jimp]

@obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

On latest devel for factory and CE, everything functionally is looking great. Had to restart *pinger (I forget which one is used these days) for gateways to get out of pending after initial interface bring-up, but packets are all flowing, no weird state issues, very solid :)

Great! I'll have to check back on the gateways, one of mine is OK and it comes right up, I had disabled gateway monitoring on the other pair because it was interfering with the packet captures I was taking when diagnosing some of the other traffic issues above.