Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Routed IPsec using if_ipsec VTI interfaces

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    45 Posts 2 Posters 11.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      The far side of mine wasn't pfSense, but TNSR, and it was set to /24, pfSense was set to /8 (incorrectly) but yeah it was working, oddly enough.

      I do see ESP on mine, yours is only showing IKE traffic so maybe your tunnel isn't even connecting?

      Here are the changes I just pushed so that it will respect the subnet mask set for the local side, and so it applies the interface changes when IPsec is applied: https://github.com/pfsense/pfsense/commit/65767828cb2c6d14648bd86aced1b325b172ce43

      They should show up in snapshots in a few hours or by the morning, but you can apply that with the system patches package for now.

      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      O 1 Reply Last reply Reply Quote 0
      • O
        obrienmd @jimp
        last edited by

        OK, patches applied and both are now /32s in ifconfig :)

        Still no traffic, will start digging on ipsec logs - P1s are "ESTABLISHED" and P2s show in IPSec/Overview. I'm not an ipsec guru, but I'll let you know what I find.

        ipsec --statusall on each box:

        #Washington
        Status of IKE charon daemon (strongSwan 5.6.2, FreeBSD 11.2-RC1, amd64):
          uptime: 26 hours, since Jun 03 09:49:40 2018
          worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
          loaded plugins: charon unbound aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey ipseckey pem openssl fips-prf curve25519 xcbc cmac hmac curl attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-sim eap-md5 eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap whitelist addrblock counters
        Listening IP addresses:
          {all system IPs}
        Connections:
                con2:  {washington_wan_ip}...{texas_wan_ip}  IKEv2, dpddelay=10s
                con2:   local:  [{washington_wan_ip}] uses pre-shared key authentication
                con2:   remote: [{texas_wan_ip}] uses pre-shared key authentication
                con2:   child:  192.168.241.1/32|/0 === 192.168.242.1/32|/0 TUNNEL, dpdaction=restart
        Security Associations (1 up, 0 connecting):
                con2[8]: ESTABLISHED 114 minutes ago, {washington_wan_ip}[{washington_wan_ip}]...{texas_wan_ip}[{texas_wan_ip}]
                con2[8]: IKEv2 SPIs: 7a626268c7c2243f_i 94e70afd7a796de2_r*, pre-shared key reauthentication in 5 hours
                con2[8]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
                con2{48}:  INSTALLED, TUNNEL, reqid 3, ESP SPIs: c659a939_i c37c85a6_o
                con2{48}:  AES_GCM_16_256/MODP_2048, 77812 bytes_i, 0 bytes_o, rekeying in 22 minutes
                con2{48}:   192.168.241.1/32|/0 === 192.168.242.1/32|/0
        
        #Texas
        Status of IKE charon daemon (strongSwan 5.6.2, FreeBSD 11.2-RC1, amd64):
          uptime: 24 hours, since Jun 03 11:20:24 2018
          worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
          loaded plugins: charon unbound aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey ipseckey pem openssl fips-prf curve25519 xcbc cmac hmac curl attr kernel-pfkey kernel-pfroute resolve socket-default stroke vici updown eap-identity eap-sim eap-md5 eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap whitelist addrblock counters
        Listening IP addresses:
          {all system IPs}
        Connections:
           bypasslan:  %any...%any  IKEv1/2
           bypasslan:   local:  uses public key authentication
           bypasslan:   remote: uses public key authentication
           bypasslan:   child: {local_lan}/24|/0 === {local_lan}/24|/0 PASS
                con1:  {texas_wan_ip}...{washington_wan_ip}  IKEv2, dpddelay=10s
                con1:   local:  [{texas_wan_ip}] uses pre-shared key authentication
                con1:   remote: [{washington_wan_ip}] uses pre-shared key authentication
                con1:   child:  192.168.242.1/32|/0 === 192.168.241.1/32|/0 TUNNEL, dpdaction=restart
        Shunted Connections:
           bypasslan:  {local_lan}/24|/0 === {local_lan}/24|/0 PASS
        Security Associations (1 up, 0 connecting):
                con1[4]: ESTABLISHED 114 minutes ago,{texas_wan_ip[{texas_wan_ip}]...{washington_wan_ip}[{washington_wan_ip}]
                con1[4]: IKEv2 SPIs: 7a626268c7c2243f_i* 94e70afd7a796de2_r, pre-shared key reauthentication in 5 hours
                con1[4]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
                con1{37}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c37c85a6_i c659a939_o
                con1{37}:  AES_GCM_16_256/MODP_2048, 0 bytes_i, 235872 bytes_o, rekeying in 23 minutes
                con1{37}:   192.168.242.1/32|/0 === 192.168.241.1/32|/0
        
        
        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          Try putting them into a unique common /30, like you would an OpenVPN tunnel network.

          If you need to route LAN-to-LAN then setup static routes or try a dynamic routing protocol (OSPF, BGP), though admittedly the routing protocol part has not been tested yet, only static routes.

          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • O
            obrienmd
            last edited by

            Sure, I'm pretty comfortable with routing, just not the ipsec side. I'll try a shared /30.

            Right now, status is very strange. Gateways are showing up (i.e. they can ping each other, and I see that in tcpdump), but when I try to ping from CLI, even using -S (to set source in the same subnet, just in case), I get nothing in reply. Very odd.

            States are weird, some are going through enc0, some on the actual int.

            Washington:
            enc0 icmp 10.91.110.2:8558 <- 10.91.110.1:8558       0:0
            ipsec1 icmp 10.91.110.2:10026 -> 10.91.110.1:10026       0:0
            ipsec1 icmp 10.91.110.2:33717 -> 10.91.110.1:33717       0:0
            
            Texas:
            ipsec1 icmp 10.91.110.1:8558 -> 10.91.110.2:8558       0:0
            enc0 icmp 10.91.110.1:33717 <- 10.91.110.2:33717       0:0
            
            1 Reply Last reply Reply Quote 0
            • O
              obrienmd
              last edited by

              Eureka - putting them in a shared /30 seems to work. Now to see if all the GRE weirdness in previous uses get me with VTI :)

              State matching: https://redmine.pfsense.org/issues/4479
              Another: GREs sometimes open states before ipsec, then can't "get going" until states are cleared

              jimpJ 1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate @obrienmd
                last edited by

                @obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

                Eureka - putting them in a shared /30 seems to work. Now to see if all the GRE weirdness in previous uses get me with VTI :)

                State matching: https://redmine.pfsense.org/issues/4479

                I haven't tested TCP much but it shouldn't have the same issues.

                Another: GREs sometimes open states before ipsec, then can't "get going" until states are cleared

                That you can solve by putting floating rules outbound on WAN to stop your traffic from leaking and making incorrect states.

                Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • O
                  obrienmd
                  last edited by

                  Thanks for all the help thus far Jim!

                  This is super exciting - it seems to work great thus far. The floating rules thing was always a little iffy for us (one in 5-6 reboots would get bad states even though non-IKE/ESP traffic was forbidden), though I'm with you in principle.

                  One last (I hope) weird issue: Firewall-originated traffic targeting anything outside the ipsec tunnel ip of the far firewall goes out the ipsec interface (i.e. route works as expected), but a dump of the far side interface doesn't show the traffic incoming. So:

                  • From Texas LAN host to Washington firewall - pings, services work
                  • From Texas LAN host to Washington LAN host - pings, services work
                  • From Texas firewall to Washington firewall ipsec tunnel IP - pings, services work
                  • From Texas firewall to Washington firewall LAN IP - pings, services fail (see outbound in tcpdump, not inbound on far side)
                  • From Texas firewall to Washington LAN host - pings, services fail (see outbound in tcpdump, not inbound on far side)
                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    I'll have to setup a better test to try that out, but I found an issue with the interface numbering/reqids that I need to fix before getting back to that.

                    Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    O 1 Reply Last reply Reply Quote 0
                    • O
                      obrienmd @jimp
                      last edited by

                      @jimp yep, I think I'm seeing the reqid issue myself. Every few reboots, I get this complaint and no traffic flows:

                      Jun 5 08:08:59	charon		12[KNL] received an SADB_ACQUIRE with policy id 2 but no matching policy found
                      Jun 5 08:08:59	charon		12[KNL] creating acquire job for policy {near_wan_ip}/32|/0 === {far_wan_ip}/32|/0 with reqid {0}
                      Jun 5 08:08:59	charon		14[CFG] trap not found, unable to acquire reqid 0
                      
                      1 Reply Last reply Reply Quote 0
                      • jimpJ
                        jimp Rebel Alliance Developer Netgate
                        last edited by

                        If it works at all, it probably isn't the same issue. In my case I'm seeing the interface end up with one number but the reqid in the ipsec config has a different number, so no traffic ever reaches the interface due to the mismatch. That should be an all-or-nothing situation.

                        If you only have one P1/P2 or even only one P2 per P1 then it should be OK as-is, just by coincidence.

                        Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                        Need help fast? Netgate Global Support!

                        Do not Chat/PM for help!

                        1 Reply Last reply Reply Quote 0
                        • O
                          obrienmd
                          last edited by

                          When I see those errors, it really doesn't work at all. I have connected P1s and P2s, but traffic isn't flowing at all (not the previous situation two posts up).

                          1 Reply Last reply Reply Quote 0
                          • jimpJ
                            jimp Rebel Alliance Developer Netgate
                            last edited by

                            I just pushed some changes to how the IPsec interfaces are formed. The numbering of the interfaces has changed, so to be safe you should unassign the interface before upgrading. I'll work on some upgrade code in the morning, but it should hopefully now align better in terms of how strongswan forms the reqid vs how the if_ipsec interfaces want it so everything will line up better. I need to do some more testing, but it should be better.

                            Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                            Need help fast? Netgate Global Support!

                            Do not Chat/PM for help!

                            1 Reply Last reply Reply Quote 0
                            • O
                              obrienmd
                              last edited by

                              Sweet - I'll test as soon as the snapshots show up and let you know how it goes!

                              I did notice that the official Netgate boxes I'm working with (SG-2440s mostly) seem to "see" snapshots for 2.4.4 later than the community edition installs I'm also testing with.

                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                The factory snapshots happen on a different schedule than the CE snapshots so they won't ever be exactly the same. Close, but not the same. Also depends on how things get merged back into factory and if there are any conflicts.

                                Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • jimpJ
                                  jimp Rebel Alliance Developer Netgate
                                  last edited by

                                  Looks like it's better and worse. I can pass traffic between the hosts that failed before, but the gateways are not being generated properly so I need to fix that up, so static routes won't work and so on.

                                  I see the code blocks I didn't update to the new style so I'll fix those up in the morning.

                                  Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                  Need help fast? Netgate Global Support!

                                  Do not Chat/PM for help!

                                  1 Reply Last reply Reply Quote 0
                                  • O
                                    obrienmd
                                    last edited by

                                    Thanks Jim!

                                    1 Reply Last reply Reply Quote 0
                                    • jimpJ
                                      jimp Rebel Alliance Developer Netgate
                                      last edited by

                                      OK, I just pushed the updated gateway code and it's working well for me now.

                                      I do, however, see the same behavior you did where the firewall can't reach a routed network on the far side using the ipsec interface address as the source. It does work if I set the source to be the LAN, however.

                                      Using the ipsecX interface address as the source:

                                      : ping -S 10.6.106.1 10.7.0.1
                                      PING 10.7.0.1 (10.7.0.1) from 10.6.106.1: 56 data bytes
                                      ^C
                                      --- 10.7.0.1 ping statistics ---
                                      2 packets transmitted, 0 packets received, 100.0% packet loss
                                      

                                      Going LAN to LAN from the firewall:

                                      : ping -S 10.6.0.1 10.7.0.1
                                      PING 10.7.0.1 (10.7.0.1) from 10.6.0.1: 56 data bytes
                                      64 bytes from 10.7.0.1: icmp_seq=0 ttl=64 time=0.802 ms
                                      64 bytes from 10.7.0.1: icmp_seq=1 ttl=64 time=0.883 ms
                                      64 bytes from 10.7.0.1: icmp_seq=2 ttl=64 time=0.716 ms
                                      ^C
                                      --- 10.7.0.1 ping statistics ---
                                      3 packets transmitted, 3 packets received, 0.0% packet loss
                                      

                                      Routes all look correct, on the source node traffic appears on the ipsecX and enc0 interface but the counters on the child SA do not increase and no ESP leaves, so somehow it isn't making its way to that connection. I'll keep poking at it, but it's not the end of the world since the same situation also didn't work on plain IPsec, though we hoped routed IPsec would be a cure for that.

                                      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                      Need help fast? Netgate Global Support!

                                      Do not Chat/PM for help!

                                      1 Reply Last reply Reply Quote 0
                                      • jimpJ
                                        jimp Rebel Alliance Developer Netgate
                                        last edited by

                                        Since that firewall-to-LAN routing issue is not a flaw in the VTI code that I can see, I've split that off into https://redmine.pfsense.org/issues/8551

                                        Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                        Need help fast? Netgate Global Support!

                                        Do not Chat/PM for help!

                                        1 Reply Last reply Reply Quote 0
                                        • O
                                          obrienmd
                                          last edited by

                                          Makes perfect sense to me - as soon as the daily build hits pfSense factory -devel, I'll start testing again!

                                          1 Reply Last reply Reply Quote 0
                                          • jimpJ
                                            jimp Rebel Alliance Developer Netgate
                                            last edited by

                                            OK, I think I have that nailed down. Apparently it does not get along with pf route-to directly on the interface. It works fine for LAN traffic but not traffic exiting from the firewall itself. I pushed a fix, should be in snaps soonish.

                                            Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                            Need help fast? Netgate Global Support!

                                            Do not Chat/PM for help!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.