Routed IPsec using if_ipsec VTI interfaces


  • Rebel Alliance Developer Netgate

    The next round of 2.4.4 snapshots will include support for routed IPsec using FreeBSD's if_ipsec(4) virtual tunnel interfaces (VTI).

    With a VTI setup, you define the local/remote network in a P2 entry set to VTI, and these addresses become the local and remote addresses of an ipsecX interface. This interface can then be assigned and enabled (Set IP type to None, just like OpenVPN), and then you can setup static routes, it gets a gateway to use with policy routing, and you can use routing protocols with it as well. Though that last part is not well-tested since FRR has a zebra crash issue going on at the moment on 2.4.4 snaps. You also get interface-specific IPsec firewall rules and NAT, you can packet capture on the interface, and get traffic data.

    As things get tested/fixed we'll have a better write-up on how to set it all up, but it's fairly straightforward.



  • Hi Jim, I tested in 2.4.4 snapshot (6/2), no connectivity yet - posted my results / some info in the redmine feature:

    https://redmine.pfsense.org/issues/8544


  • Rebel Alliance Developer Netgate

    Let's keep the discussion here and not on Redmine.

    Can you show me:

    • The output of ifconfig -a or at least the ipsecX interface (where X = that tunnels reqid, which looks to be 1 from your redmine comment)
    • The contents of the state table for those IP addresses (e.g. pfctl -ss | grep 192.168.240)
    • The config.xml contents for the IPsec tunnel P1 and P2, and the assigned interface

    On mine, the interface does not show as a /24 and it shouldn't since it's a peer-to-peer interface:

    ipsec2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
            tunnel inet 203.0.113.14 --> 203.0.113.2
            inet6 fe80::20d:b9ff:fe33:f70%ipsec2 prefixlen 64 scopeid 0xb 
            inet 10.2.222.2 --> 10.2.222.1 netmask 0xff000000 
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            reqid: 2
            groups: ipsec
    

    I do have a static route on the interface, however:

    : netstat -rn | egrep '(ipsec2|10.2.222)'
    10.2.0.0/24        10.2.222.1         UGS      ipsec2
    10.2.222.1         link#11            UH       ipsec2
    10.2.222.2         link#11            UHS         lo0
    fe80::%ipsec2/64                  link#11                       U        ipsec2
    fe80::20d:b9ff:fe33:f70%ipsec2    link#11                       UHS         lo0
    


  • Thanks Jim,

    Yours shows the same netmask mine does when I use a 10. for the tunnel IPs... I think it's setting some default based on which RFC1918 type space it's in - like Windows auto-populates a /24 netmask for 192.168 addresses and a /8 for 10. addresses (strange, but true).

    Here's the info - note, I changed the addresses to 192.168.241.1(Texas) and 192.168.242.1 (Washington) for troubleshooting, but here goes:

    There was no tunnel inet when i checked the Texas side's ifconfig. I disabled and re-enabled the interface, and it showed up... Still not passing traffic but pings just timed out, rather than giving the sendto error. Firewall logs don't show any packets blocked on WAN or ipsec (I have both basically open between those IPs for testing).

    ifconfig:

    #Washington
    ipsec2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
            tunnel inet {washington_wan_ip) --> {texas_wan_ip)
            inet6 fe80::208:a2ff:fe0a:7291%ipsec2 prefixlen 64 scopeid 0x9
            inet 192.168.241.1 --> 192.168.242.1 netmask 0xffffff00
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            reqid: 2
            groups: ipsec
    
    #Texas - strange, no tunnel inet
    ipsec1: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1400
            inet 192.168.242.1 --> 192.168.241.1 netmask 0xffffff00
            inet6 fe80::215:5dff:fe1f:d816%ipsec1 prefixlen 64 tentative scopeid 0x8
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            reqid: 1
            groups: ipsec
    

    states:

    #Washington
    ipsec2 icmp 192.168.241.1:56811 -> 192.168.242.1:56811       0:0
    
    #Texas
    ipsec1 icmp 192.168.242.1:25691 -> 192.168.241.1:25691       0:0
    

    config.xml:

    #Washington
            <interfaces>
                    <opt1>
                            <descr><![CDATA[Texas]]></descr>
                            <if>ipsec2</if>
                            <spoofmac></spoofmac>
                            <enable></enable>
                    </opt1>
            </interfaces>
            <ipsec>
                    <phase1>
                            <ikeid>2</ikeid>
                            <iketype>ikev2</iketype>
                            <interface>wan</interface>
                            <remote-gateway>{texas_wan_ip}</remote-gateway>
                            <protocol>inet</protocol>
                            <myid_type>myaddress</myid_type>
                            <myid_data></myid_data>
                            <peerid_type>peeraddress</peerid_type>
                            <peerid_data></peerid_data>
                            <encryption>
                                    <item>
                                            <encryption-algorithm>
                                                    <name>aes</name>
                                                    <keylen>256</keylen>
                                            </encryption-algorithm>
                                            <hash-algorithm>sha256</hash-algorithm>
                                            <dhgroup>2</dhgroup>
                                    </item>
                            </encryption>
                            <lifetime>28800</lifetime>
                            <pre-shared-key>{my_pre_shared_key}</pre-shared-key>
                            <private-key></private-key>
                            <certref></certref>
                            <caref></caref>
                            <authentication_method>pre_shared_key</authentication_method>
                            <descr><![CDATA[TXDC]]></descr>
                            <nat_traversal>on</nat_traversal>
                            <mobike>off</mobike>
                            <margintime></margintime>
                            <dpd_delay>10</dpd_delay>
                            <dpd_maxfail>5</dpd_maxfail>
                    </phase1>
                    <client></client>
                    <logging>
                            <dmn>1</dmn>
                            <mgr>1</mgr>
                            <ike>2</ike>
                            <chd>2</chd>
                            <job>1</job>
                            <cfg>2</cfg>
                            <knl>1</knl>
                            <net>1</net>
                            <asn>1</asn>
                            <enc>1</enc>
                            <imc>1</imc>
                            <imv>1</imv>
                            <pts>1</pts>
                            <tls>1</tls>
                            <esp>1</esp>
                            <lib>1</lib>
                    </logging>
                    <makebeforebreak></makebeforebreak>
                    <uniqueids>yes</uniqueids>
                    <phase2>
                            <ikeid>2</ikeid>
                            <uniqid>5b141a502eaff</uniqid>
                            <mode>vti</mode>
                            <reqid>1</reqid>
                            <localid>
                                    <type>address</type>
                                    <address>192.168.241.1</address>
                            </localid>
                            <remoteid>
                                    <type>address</type>
                                    <address>192.168.242.1</address>
                            </remoteid>
                            <protocol>esp</protocol>
                            <encryption-algorithm-option>
                                    <name>aes256gcm</name>
                                    <keylen>auto</keylen>
                            </encryption-algorithm-option>
                            <hash-algorithm-option>hmac_sha256</hash-algorithm-option>
                            <pfsgroup>14</pfsgroup>
                            <lifetime>3600</lifetime>
                            <pinghost></pinghost>
                            <descr><![CDATA[TXDC]]></descr>
                    </phase2>
            </ipsec>
    
    #Texas
            <interfaces>
                    <opt2>
                            <descr><![CDATA[Washington]]></descr>
                            <if>ipsec1</if>
                            <spoofmac></spoofmac>
                            <enable></enable>
                    </opt2>
            </interfaces>
            <ipsec>
                    <phase1>
                            <ikeid>1</ikeid>
                            <iketype>ikev2</iketype>
                            <interface>wan</interface>
                            <remote-gateway>{washington_wan_ip}</remote-gateway>
                            <protocol>inet</protocol>
                            <myid_type>myaddress</myid_type>
                            <myid_data></myid_data>
                            <peerid_type>peeraddress</peerid_type>
                            <peerid_data></peerid_data>
                            <encryption>
                                    <item>
                                            <encryption-algorithm>
                                                    <name>aes</name>
                                                    <keylen>256</keylen>
                                            </encryption-algorithm>
                                            <hash-algorithm>sha256</hash-algorithm>
                                            <dhgroup>2</dhgroup>
                                    </item>
                            </encryption>
                            <lifetime>28800</lifetime>
                            <pre-shared-key>{my_pre_shared_key}</pre-shared-key>
                            <private-key></private-key>
                            <certref></certref>
                            <caref></caref>
                            <authentication_method>pre_shared_key</authentication_method>
                            <descr><![CDATA[Washington]]></descr>
                            <nat_traversal>on</nat_traversal>
                            <mobike>off</mobike>
                            <margintime></margintime>
                            <dpd_delay>10</dpd_delay>
                            <dpd_maxfail>5</dpd_maxfail>
                    </phase1>
                    <client></client>
                    <logging>
                            <dmn>1</dmn>
                            <mgr>1</mgr>
                            <ike>2</ike>
                            <chd>2</chd>
                            <job>1</job>
                            <cfg>2</cfg>
                            <knl>1</knl>
                            <net>1</net>
                            <asn>1</asn>
                            <enc>1</enc>
                            <imc>1</imc>
                            <imv>1</imv>
                            <pts>1</pts>
                            <tls>1</tls>
                            <esp>1</esp>
                            <lib>1</lib>
                    </logging>
                    <makebeforebreak></makebeforebreak>
                    <uniqueids>yes</uniqueids>
                    <phase2>
                            <ikeid>1</ikeid>
                            <uniqid>5b141a51cd2c8</uniqid>
                            <mode>vti</mode>
                            <reqid>1</reqid>
                            <localid>
                                    <type>address</type>
                                    <address>192.168.242.1</address>
                            </localid>
                            <remoteid>
                                    <type>address</type>
                                    <address>192.168.241.1</address>
                            </remoteid>
                            <protocol>esp</protocol>
                            <encryption-algorithm-option>
                                    <name>aes256gcm</name>
                                    <keylen>auto</keylen>
                            </encryption-algorithm-option>
                            <hash-algorithm-option>hmac_sha256</hash-algorithm-option>
                            <pfsgroup>14</pfsgroup>
                            <lifetime>3600</lifetime>
                            <pinghost></pinghost>
                            <descr><![CDATA[Auburn]]></descr>
                    </phase2>
            </ipsec>
    


  • Thought this was interesting, here's a tcpdump of the Washington WAN interface for all packets going to the Texas WAN IP.

    During this dump, I was pinging the Texas side of the tunnel from the Washington side (both internal tunnel IPs, not WAN to WAN), but it seems that no ESP traffic was being sent.

    #Washington
    #tcpdump -vvnni {wan_int} "dst host {texas_wan_ip}"
    tcpdump: listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
    09:11:28.527012 IP (tos 0x0, ttl 64, id 48076, offset 0, flags [none], proto UDP (17), length 108)
        {washington_wan_ip}.500 > {texas_wan_ip}.500: [udp sum ok] isakmp 2.0 msgid 00000543 cookie 976f62e3b6408d81->db4ad4f0b86ca2a2: child_sa  inf2:
        (v2e: len=48)
    09:11:38.550511 IP (tos 0x0, ttl 64, id 19398, offset 0, flags [none], proto UDP (17), length 108)
        {washington_wan_ip}.500 > {texas_wan_ip}.500: [udp sum ok] isakmp 2.0 msgid 00000544 cookie 976f62e3b6408d81->db4ad4f0b86ca2a2: child_sa  inf2:
        (v2e: len=48)
    09:11:48.626281 IP (tos 0x0, ttl 64, id 22543, offset 0, flags [none], proto UDP (17), length 108)
        {washington_wan_ip}.500 > {texas_wan_ip}.500: [udp sum ok] isakmp 2.0 msgid 00000545 cookie 976f62e3b6408d81->db4ad4f0b86ca2a2: child_sa  inf2:
        (v2e: len=48)
    09:11:58.715107 IP (tos 0x0, ttl 64, id 48507, offset 0, flags [none], proto UDP (17), length 108)
        {washington_wan_ip}.500 > {texas_wan_ip}.500: [udp sum ok] isakmp 2.0 msgid 00000546 cookie 976f62e3b6408d81->db4ad4f0b86ca2a2: child_sa  inf2:
        (v2e: len=48)
    09:11:58.731185 IP (tos 0x0, ttl 64, id 19883, offset 0, flags [none], proto UDP (17), length 108)
        {washington_wan_ip}.500 > {texas_wan_ip}.500: [udp sum ok] isakmp 2.0 msgid 000005a3 cookie 976f62e3b6408d81->db4ad4f0b86ca2a2: child_sa  inf2[R]:
        (v2e: len=48)
    

  • Rebel Alliance Developer Netgate

    The missing tunnel inet line is probably part of it, but I'm also wary of the netmask it added. I believe that should be /32 (0xFFFFFFFF) but I need to look into why it's ending up like it is. Possibly because of a default per block like you mentioned.

    It can take an actual subnet there and not just a single address, so I may need to reconsider locking down the GUI controls like I have.

    I'll work up a fix and push it shortly



  • Thanks, ready to test as soon as it's in snapshots!



  • Note, the reason I set them in different /24s is that I was hoping it would cause the traffic to "route" rather than going off into the ether somewhere (thinking the addy was in-net maybe and trying to just broadcast it), given it was destined for a different subnet, and the default it was giving me for 192.168/16 RFC1918 addresses was /24.

    However, I still see traffic exiting the interface in tcpdump, but don't see any ESP going out the WAN. It's weird, I have no idea where the packets are going :)

    Also, yours worked, and they were adjacent addresses in a /8 per your ifconfig entries, so that's a mystery :)


  • Rebel Alliance Developer Netgate

    The far side of mine wasn't pfSense, but TNSR, and it was set to /24, pfSense was set to /8 (incorrectly) but yeah it was working, oddly enough.

    I do see ESP on mine, yours is only showing IKE traffic so maybe your tunnel isn't even connecting?

    Here are the changes I just pushed so that it will respect the subnet mask set for the local side, and so it applies the interface changes when IPsec is applied: https://github.com/pfsense/pfsense/commit/65767828cb2c6d14648bd86aced1b325b172ce43

    They should show up in snapshots in a few hours or by the morning, but you can apply that with the system patches package for now.



  • OK, patches applied and both are now /32s in ifconfig :)

    Still no traffic, will start digging on ipsec logs - P1s are "ESTABLISHED" and P2s show in IPSec/Overview. I'm not an ipsec guru, but I'll let you know what I find.

    ipsec --statusall on each box:

    #Washington
    Status of IKE charon daemon (strongSwan 5.6.2, FreeBSD 11.2-RC1, amd64):
      uptime: 26 hours, since Jun 03 09:49:40 2018
      worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
      loaded plugins: charon unbound aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey ipseckey pem openssl fips-prf curve25519 xcbc cmac hmac curl attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-sim eap-md5 eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap whitelist addrblock counters
    Listening IP addresses:
      {all system IPs}
    Connections:
            con2:  {washington_wan_ip}...{texas_wan_ip}  IKEv2, dpddelay=10s
            con2:   local:  [{washington_wan_ip}] uses pre-shared key authentication
            con2:   remote: [{texas_wan_ip}] uses pre-shared key authentication
            con2:   child:  192.168.241.1/32|/0 === 192.168.242.1/32|/0 TUNNEL, dpdaction=restart
    Security Associations (1 up, 0 connecting):
            con2[8]: ESTABLISHED 114 minutes ago, {washington_wan_ip}[{washington_wan_ip}]...{texas_wan_ip}[{texas_wan_ip}]
            con2[8]: IKEv2 SPIs: 7a626268c7c2243f_i 94e70afd7a796de2_r*, pre-shared key reauthentication in 5 hours
            con2[8]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
            con2{48}:  INSTALLED, TUNNEL, reqid 3, ESP SPIs: c659a939_i c37c85a6_o
            con2{48}:  AES_GCM_16_256/MODP_2048, 77812 bytes_i, 0 bytes_o, rekeying in 22 minutes
            con2{48}:   192.168.241.1/32|/0 === 192.168.242.1/32|/0
    
    #Texas
    Status of IKE charon daemon (strongSwan 5.6.2, FreeBSD 11.2-RC1, amd64):
      uptime: 24 hours, since Jun 03 11:20:24 2018
      worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
      loaded plugins: charon unbound aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey ipseckey pem openssl fips-prf curve25519 xcbc cmac hmac curl attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-sim eap-md5 eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap whitelist addrblock counters
    Listening IP addresses:
      {all system IPs}
    Connections:
       bypasslan:  %any...%any  IKEv1/2
       bypasslan:   local:  uses public key authentication
       bypasslan:   remote: uses public key authentication
       bypasslan:   child: {local_lan}/24|/0 === {local_lan}/24|/0 PASS
            con1:  {texas_wan_ip}...{washington_wan_ip}  IKEv2, dpddelay=10s
            con1:   local:  [{texas_wan_ip}] uses pre-shared key authentication
            con1:   remote: [{washington_wan_ip}] uses pre-shared key authentication
            con1:   child:  192.168.242.1/32|/0 === 192.168.241.1/32|/0 TUNNEL, dpdaction=restart
    Shunted Connections:
       bypasslan:  {local_lan}/24|/0 === {local_lan}/24|/0 PASS
    Security Associations (1 up, 0 connecting):
            con1[4]: ESTABLISHED 114 minutes ago,{texas_wan_ip[{texas_wan_ip}]...{washington_wan_ip}[{washington_wan_ip}]
            con1[4]: IKEv2 SPIs: 7a626268c7c2243f_i* 94e70afd7a796de2_r, pre-shared key reauthentication in 5 hours
            con1[4]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
            con1{37}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c37c85a6_i c659a939_o
            con1{37}:  AES_GCM_16_256/MODP_2048, 0 bytes_i, 235872 bytes_o, rekeying in 23 minutes
            con1{37}:   192.168.242.1/32|/0 === 192.168.241.1/32|/0
    
    

  • Rebel Alliance Developer Netgate

    Try putting them into a unique common /30, like you would an OpenVPN tunnel network.

    If you need to route LAN-to-LAN then setup static routes or try a dynamic routing protocol (OSPF, BGP), though admittedly the routing protocol part has not been tested yet, only static routes.



  • Sure, I'm pretty comfortable with routing, just not the ipsec side. I'll try a shared /30.

    Right now, status is very strange. Gateways are showing up (i.e. they can ping each other, and I see that in tcpdump), but when I try to ping from CLI, even using -S (to set source in the same subnet, just in case), I get nothing in reply. Very odd.

    States are weird, some are going through enc0, some on the actual int.

    Washington:
    enc0 icmp 10.91.110.2:8558 <- 10.91.110.1:8558       0:0
    ipsec1 icmp 10.91.110.2:10026 -> 10.91.110.1:10026       0:0
    ipsec1 icmp 10.91.110.2:33717 -> 10.91.110.1:33717       0:0
    
    Texas:
    ipsec1 icmp 10.91.110.1:8558 -> 10.91.110.2:8558       0:0
    enc0 icmp 10.91.110.1:33717 <- 10.91.110.2:33717       0:0
    


  • Eureka - putting them in a shared /30 seems to work. Now to see if all the GRE weirdness in previous uses get me with VTI :)

    State matching: https://redmine.pfsense.org/issues/4479
    Another: GREs sometimes open states before ipsec, then can't "get going" until states are cleared


  • Rebel Alliance Developer Netgate

    @obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

    Eureka - putting them in a shared /30 seems to work. Now to see if all the GRE weirdness in previous uses get me with VTI :)

    State matching: https://redmine.pfsense.org/issues/4479

    I haven't tested TCP much but it shouldn't have the same issues.

    Another: GREs sometimes open states before ipsec, then can't "get going" until states are cleared

    That you can solve by putting floating rules outbound on WAN to stop your traffic from leaking and making incorrect states.



  • Thanks for all the help thus far Jim!

    This is super exciting - it seems to work great thus far. The floating rules thing was always a little iffy for us (one in 5-6 reboots would get bad states even though non-IKE/ESP traffic was forbidden), though I'm with you in principle.

    One last (I hope) weird issue: Firewall-originated traffic targeting anything outside the ipsec tunnel ip of the far firewall goes out the ipsec interface (i.e. route works as expected), but a dump of the far side interface doesn't show the traffic incoming. So:

    • From Texas LAN host to Washington firewall - pings, services work
    • From Texas LAN host to Washington LAN host - pings, services work
    • From Texas firewall to Washington firewall ipsec tunnel IP - pings, services work
    • From Texas firewall to Washington firewall LAN IP - pings, services fail (see outbound in tcpdump, not inbound on far side)
    • From Texas firewall to Washington LAN host - pings, services fail (see outbound in tcpdump, not inbound on far side)

  • Rebel Alliance Developer Netgate

    I'll have to setup a better test to try that out, but I found an issue with the interface numbering/reqids that I need to fix before getting back to that.



  • @jimp yep, I think I'm seeing the reqid issue myself. Every few reboots, I get this complaint and no traffic flows:

    Jun 5 08:08:59	charon		12[KNL] received an SADB_ACQUIRE with policy id 2 but no matching policy found
    Jun 5 08:08:59	charon		12[KNL] creating acquire job for policy {near_wan_ip}/32|/0 === {far_wan_ip}/32|/0 with reqid {0}
    Jun 5 08:08:59	charon		14[CFG] trap not found, unable to acquire reqid 0
    

  • Rebel Alliance Developer Netgate

    If it works at all, it probably isn't the same issue. In my case I'm seeing the interface end up with one number but the reqid in the ipsec config has a different number, so no traffic ever reaches the interface due to the mismatch. That should be an all-or-nothing situation.

    If you only have one P1/P2 or even only one P2 per P1 then it should be OK as-is, just by coincidence.



  • When I see those errors, it really doesn't work at all. I have connected P1s and P2s, but traffic isn't flowing at all (not the previous situation two posts up).


  • Rebel Alliance Developer Netgate

    I just pushed some changes to how the IPsec interfaces are formed. The numbering of the interfaces has changed, so to be safe you should unassign the interface before upgrading. I'll work on some upgrade code in the morning, but it should hopefully now align better in terms of how strongswan forms the reqid vs how the if_ipsec interfaces want it so everything will line up better. I need to do some more testing, but it should be better.



  • Sweet - I'll test as soon as the snapshots show up and let you know how it goes!

    I did notice that the official Netgate boxes I'm working with (SG-2440s mostly) seem to "see" snapshots for 2.4.4 later than the community edition installs I'm also testing with.


  • Rebel Alliance Developer Netgate

    The factory snapshots happen on a different schedule than the CE snapshots so they won't ever be exactly the same. Close, but not the same. Also depends on how things get merged back into factory and if there are any conflicts.


  • Rebel Alliance Developer Netgate

    Looks like it's better and worse. I can pass traffic between the hosts that failed before, but the gateways are not being generated properly so I need to fix that up, so static routes won't work and so on.

    I see the code blocks I didn't update to the new style so I'll fix those up in the morning.



  • Thanks Jim!


  • Rebel Alliance Developer Netgate

    OK, I just pushed the updated gateway code and it's working well for me now.

    I do, however, see the same behavior you did where the firewall can't reach a routed network on the far side using the ipsec interface address as the source. It does work if I set the source to be the LAN, however.

    Using the ipsecX interface address as the source:

    : ping -S 10.6.106.1 10.7.0.1
    PING 10.7.0.1 (10.7.0.1) from 10.6.106.1: 56 data bytes
    ^C
    --- 10.7.0.1 ping statistics ---
    2 packets transmitted, 0 packets received, 100.0% packet loss
    

    Going LAN to LAN from the firewall:

    : ping -S 10.6.0.1 10.7.0.1
    PING 10.7.0.1 (10.7.0.1) from 10.6.0.1: 56 data bytes
    64 bytes from 10.7.0.1: icmp_seq=0 ttl=64 time=0.802 ms
    64 bytes from 10.7.0.1: icmp_seq=1 ttl=64 time=0.883 ms
    64 bytes from 10.7.0.1: icmp_seq=2 ttl=64 time=0.716 ms
    ^C
    --- 10.7.0.1 ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    

    Routes all look correct, on the source node traffic appears on the ipsecX and enc0 interface but the counters on the child SA do not increase and no ESP leaves, so somehow it isn't making its way to that connection. I'll keep poking at it, but it's not the end of the world since the same situation also didn't work on plain IPsec, though we hoped routed IPsec would be a cure for that.


  • Rebel Alliance Developer Netgate

    Since that firewall-to-LAN routing issue is not a flaw in the VTI code that I can see, I've split that off into https://redmine.pfsense.org/issues/8551



  • Makes perfect sense to me - as soon as the daily build hits pfSense factory -devel, I'll start testing again!


  • Rebel Alliance Developer Netgate

    OK, I think I have that nailed down. Apparently it does not get along with pf route-to directly on the interface. It works fine for LAN traffic but not traffic exiting from the firewall itself. I pushed a fix, should be in snaps soonish.



  • Cool, thanks!

    Question - and this might be sacrilege - can I set my Factory boxes to download CE snapshots? I poked around repos but when it looks like just swapping the pfSense.conf one didn't really work out on a test box :)


  • Rebel Alliance Developer Netgate

    @obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

    Cool, thanks!

    Question - and this might be sacrilege - can I set my Factory boxes to download CE snapshots? I poked around repos but when it looks like just swapping the pfSense.conf one didn't really work out on a test box :)

    Not easily, several things need adjusted and it's just not worth the hassle to downgrade like that in-place. All the changes I made today, including the fix for that route-to issue, have been synchronized to Factory so it should show up in snapshots for both CE and Factory by the morning.



  • Much appreciated.

    This is going to help in a lot of places... Now I just have to get Verizon to terminate mobile private network tunnels as VTI :) Wish me luck...



  • Hrm, with the new devel builds, phase 1s are coming up (and SAs show, with no inbound traffic), and I'm seeing this in ipsec logs, I think the key lines being SADB_ACQUIRE and 'unable to acquire reqid'.

    CARP is enabled on WANs of one of the sides, and I'm using the CARP IP for 'Interface' on the P1 locally and 'remote gateway' on the P1 remotely. Might this have something to do with it?

    The other pair I've been testing most with is that where one side is factory, and we had some success earlier, but neither side is HA/CARP. I'll be testing again with that one today as soon as factory images come out.

    Jun 8 05:18:00	charon		10[CFG] vici client 360 registered for: list-sa
    Jun 8 05:18:00	charon		10[CFG] vici client 360 requests: list-sas
    Jun 8 05:18:00	charon		10[CFG] vici client 360 disconnected
    Jun 8 05:18:00	charon		10[KNL] received an SADB_ACQUIRE with policy id 2 but no matching policy found
    Jun 8 05:18:00	charon		10[KNL] creating acquire job for policy {local_wan_ip}/32|/0 === {remote_wan_ip}/32|/0 with reqid {0}
    Jun 8 05:18:00	charon		15[CFG] trap not found, unable to acquire reqid 0
    Jun 8 05:18:03	charon		15[KNL] <con2|2> querying policy {local_tunnel_ip}/32|/0 === {remote_tunnel_ip}/32|/0 in failed, not found
    Jun 8 05:18:03	charon		15[KNL] <con2|2> querying policy {remote_tunnel_ip)/32|/0 === {local_tunnel_ip}/32|/0 in failed, not found
    Jun 8 05:18:03	charon		15[IKE] <con2|2> sending DPD request
    Jun 8 05:18:03	charon		15[IKE] <con2|2> queueing IKE_DPD task
    Jun 8 05:18:03	charon		15[IKE] <con2|2> activating new tasks
    Jun 8 05:18:03	charon		15[IKE] <con2|2> activating IKE_DPD task
    Jun 8 05:18:03	charon		15[ENC] <con2|2> generating INFORMATIONAL request 253 [ ]
    Jun 8 05:18:03	charon		15[NET] <con2|2> sending packet: from {local_ip}[500] to {remote_ip}[500] (76 bytes)
    Jun 8 05:18:03	charon		10[NET] <con2|2> received packet: from {remote_ip)[500] to {local_ip}[500] (76 bytes)
    Jun 8 05:18:03	charon		10[ENC] <con2|2> parsed INFORMATIONAL response 253 [ ]
    Jun 8 05:18:03	charon		10[IKE] <con2|2> activating new tasks
    Jun 8 05:18:03	charon		10[IKE] <con2|2> nothing to initiate
    

  • Rebel Alliance Developer Netgate

    Granted I haven't tried it on an HA pair but nothing has changed in a couple days that would affect that. When was the last snapshot you had working there?

    Can you show the conXXXX entry from /var/etc/ipsec/ipsec.conf for that tunnel?



  • Non-HA side:

    conn con2
            fragmentation = yes
            keyexchange = ikev2
            reauth = yes
            forceencaps = no
            mobike = no
    
            rekey = yes
            installpolicy = no
    
            dpdaction = restart
            dpddelay = 10s
            dpdtimeout = 60s
            auto = start
            left = {non_ha_side_wan_ip}
            right = {ha_side_wan_ip}
            leftid = {non_ha_side_wan_ip}
            ikelifetime = 28800s
            lifetime = 3600s
            ike = aes256-sha1-modp1024!
            esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
            leftauth = psk
            rightauth = psk
            rightid = {ha_side_wan_ip}
            rightsubnet = 10.90.91.1
            leftsubnet = 10.90.91.2/30
    
    

    HA side:

    conn con1
            fragmentation = yes
            keyexchange = ikev2
            reauth = yes
            forceencaps = no
            mobike = no
    
            rekey = yes
            installpolicy = no
    
            dpdaction = restart
            dpddelay = 10s
            dpdtimeout = 60s
            auto = start
            left = {ha_side_wan_ip}
            right = {non_ha_side_wan_ip}
            leftid = {ha_side_wan_ip}
            ikelifetime = 28800s
            lifetime = 3600s
            ike = aes256-sha1-modp1024!
            esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
            leftauth = psk
            rightauth = psk
            rightid = {non_ha_side_wan_ip}
            rightsubnet = 10.90.91.2
            leftsubnet = 10.90.91.1/30
    
    

  • Rebel Alliance Developer Netgate

    Those don't look quite right, there is no reqid in those blocks like there should be. Are the ipsecX interfaces actually present?

    Might be due to the tunnel being IKEv2, I think all three of my test systems here have been IKEv1. I'll try to spin up a v2 set.



  • Yup, ipsec1000/2000 (depending on box) ints are there, and show proper /30s.


  • Rebel Alliance Developer Netgate

    I did see one problem come up that I just pushed a fix for, but I didn't see that specific error you had unless I had an IKEv1/IKEv2 mismatch between the peers.

    The fix I made only touches two lines, you can easily apply it manually to test: https://github.com/pfsense/pfsense/commit/d4b43c48ed1636d3fcd6e47d73ba721bd63d883a

    With that I just switched both sides from IKEv1 to IKEv2 and it came right back up.



  • Yep, nailed it. Looking good with that change.

    Because of your warning on frr, I'm testing with static routing right now. After everything was fixed and I disabled / re-enabled the interfaces to get traffic flowing, static routes were showing in the route table but set to hn1 rather than the ipsec interface. Editing and re-saving the static route resolved the issue.

    With dynamic routing I bet I won't see that in the future, but if there's some resiliency code somewhere to reset interfaces on static routes when gateways disappear/appear, go up/down, go pending, etc... Perhaps something needs to get tweaked there.


  • Rebel Alliance Developer Netgate

    I need to work a bit on static routes yet. I had it solved and working on reboot but somewhere in my changes this week that appears to have broken again as I am not seeing my routes in the table after it boots up. I need to investigate more and open another issue up for that.

    FRR should be better next week, see my updates on https://redmine.pfsense.org/issues/8449#note-2



  • Great, thanks Jim.