Routed IPsec using if_ipsec VTI interfaces

jimp

Granted I haven't tried it on an HA pair but nothing has changed in a couple days that would affect that. When was the last snapshot you had working there?

Can you show the conXXXX entry from /var/etc/ipsec/ipsec.conf for that tunnel?

obrienmd

Non-HA side:

conn con2
        fragmentation = yes
        keyexchange = ikev2
        reauth = yes
        forceencaps = no
        mobike = no

        rekey = yes
        installpolicy = no

        dpdaction = restart
        dpddelay = 10s
        dpdtimeout = 60s
        auto = start
        left = {non_ha_side_wan_ip}
        right = {ha_side_wan_ip}
        leftid = {non_ha_side_wan_ip}
        ikelifetime = 28800s
        lifetime = 3600s
        ike = aes256-sha1-modp1024!
        esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
        leftauth = psk
        rightauth = psk
        rightid = {ha_side_wan_ip}
        rightsubnet = 10.90.91.1
        leftsubnet = 10.90.91.2/30

HA side:

conn con1
        fragmentation = yes
        keyexchange = ikev2
        reauth = yes
        forceencaps = no
        mobike = no

        rekey = yes
        installpolicy = no

        dpdaction = restart
        dpddelay = 10s
        dpdtimeout = 60s
        auto = start
        left = {ha_side_wan_ip}
        right = {non_ha_side_wan_ip}
        leftid = {ha_side_wan_ip}
        ikelifetime = 28800s
        lifetime = 3600s
        ike = aes256-sha1-modp1024!
        esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
        leftauth = psk
        rightauth = psk
        rightid = {non_ha_side_wan_ip}
        rightsubnet = 10.90.91.2
        leftsubnet = 10.90.91.1/30

jimp

Those don't look quite right, there is no reqid in those blocks like there should be. Are the ipsecX interfaces actually present?

Might be due to the tunnel being IKEv2, I think all three of my test systems here have been IKEv1. I'll try to spin up a v2 set.

obrienmd

Yup, ipsec1000/2000 (depending on box) ints are there, and show proper /30s.

jimp

I did see one problem come up that I just pushed a fix for, but I didn't see that specific error you had unless I had an IKEv1/IKEv2 mismatch between the peers.

The fix I made only touches two lines, you can easily apply it manually to test: https://github.com/pfsense/pfsense/commit/d4b43c48ed1636d3fcd6e47d73ba721bd63d883a

With that I just switched both sides from IKEv1 to IKEv2 and it came right back up.

obrienmd

Yep, nailed it. Looking good with that change.

Because of your warning on frr, I'm testing with static routing right now. After everything was fixed and I disabled / re-enabled the interfaces to get traffic flowing, static routes were showing in the route table but set to hn1 rather than the ipsec interface. Editing and re-saving the static route resolved the issue.

With dynamic routing I bet I won't see that in the future, but if there's some resiliency code somewhere to reset interfaces on static routes when gateways disappear/appear, go up/down, go pending, etc... Perhaps something needs to get tweaked there.

jimp

I need to work a bit on static routes yet. I had it solved and working on reboot but somewhere in my changes this week that appears to have broken again as I am not seeing my routes in the table after it boots up. I need to investigate more and open another issue up for that.

FRR should be better next week, see my updates on https://redmine.pfsense.org/issues/8449#note-2

obrienmd

Great, thanks Jim.

obrienmd

Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

jimp

@obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

Not without loading it up and seeing what's in /etc/version.lastcommit. Servers are using CDT.

jimp

Static routes should be OK now. I'm not quite sure how it worked before, given the changes I had to make, but it's working now.

https://github.com/pfsense/pfsense/commit/0aa52fb21a21f58035f2e2fe3b9328a9c175ffb5

I think that might be most if not all of the functional issues. There are still some anti-foot-shooting measures I need to take like preventing removing an IPsec tunnel or P2 used as a VTI interface.

obrienmd

On latest devel for factory and CE, everything functionally is looking great. Had to restart *pinger (I forget which one is used these days) for gateways to get out of pending after initial interface bring-up, but packets are all flowing, no weird state issues, very solid :)

jimp

@obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

On latest devel for factory and CE, everything functionally is looking great. Had to restart *pinger (I forget which one is used these days) for gateways to get out of pending after initial interface bring-up, but packets are all flowing, no weird state issues, very solid :)

Great! I'll have to check back on the gateways, one of mine is OK and it comes right up, I had disabled gateway monitoring on the other pair because it was interfering with the packet captures I was taking when diagnosing some of the other traffic issues above.