Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Routed IPsec using if_ipsec VTI interfaces

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    45 Posts 2 Posters 11.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      OK, I just pushed the updated gateway code and it's working well for me now.

      I do, however, see the same behavior you did where the firewall can't reach a routed network on the far side using the ipsec interface address as the source. It does work if I set the source to be the LAN, however.

      Using the ipsecX interface address as the source:

      : ping -S 10.6.106.1 10.7.0.1
      PING 10.7.0.1 (10.7.0.1) from 10.6.106.1: 56 data bytes
      ^C
      --- 10.7.0.1 ping statistics ---
      2 packets transmitted, 0 packets received, 100.0% packet loss
      

      Going LAN to LAN from the firewall:

      : ping -S 10.6.0.1 10.7.0.1
      PING 10.7.0.1 (10.7.0.1) from 10.6.0.1: 56 data bytes
      64 bytes from 10.7.0.1: icmp_seq=0 ttl=64 time=0.802 ms
      64 bytes from 10.7.0.1: icmp_seq=1 ttl=64 time=0.883 ms
      64 bytes from 10.7.0.1: icmp_seq=2 ttl=64 time=0.716 ms
      ^C
      --- 10.7.0.1 ping statistics ---
      3 packets transmitted, 3 packets received, 0.0% packet loss
      

      Routes all look correct, on the source node traffic appears on the ipsecX and enc0 interface but the counters on the child SA do not increase and no ESP leaves, so somehow it isn't making its way to that connection. I'll keep poking at it, but it's not the end of the world since the same situation also didn't work on plain IPsec, though we hoped routed IPsec would be a cure for that.

      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Since that firewall-to-LAN routing issue is not a flaw in the VTI code that I can see, I've split that off into https://redmine.pfsense.org/issues/8551

        Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • O
          obrienmd
          last edited by

          Makes perfect sense to me - as soon as the daily build hits pfSense factory -devel, I'll start testing again!

          1 Reply Last reply Reply Quote 0
          • jimpJ
            jimp Rebel Alliance Developer Netgate
            last edited by

            OK, I think I have that nailed down. Apparently it does not get along with pf route-to directly on the interface. It works fine for LAN traffic but not traffic exiting from the firewall itself. I pushed a fix, should be in snaps soonish.

            Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 0
            • O
              obrienmd
              last edited by

              Cool, thanks!

              Question - and this might be sacrilege - can I set my Factory boxes to download CE snapshots? I poked around repos but when it looks like just swapping the pfSense.conf one didn't really work out on a test box :)

              1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate
                last edited by

                @obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

                Cool, thanks!

                Question - and this might be sacrilege - can I set my Factory boxes to download CE snapshots? I poked around repos but when it looks like just swapping the pfSense.conf one didn't really work out on a test box :)

                Not easily, several things need adjusted and it's just not worth the hassle to downgrade like that in-place. All the changes I made today, including the fix for that route-to issue, have been synchronized to Factory so it should show up in snapshots for both CE and Factory by the morning.

                Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • O
                  obrienmd
                  last edited by

                  Much appreciated.

                  This is going to help in a lot of places... Now I just have to get Verizon to terminate mobile private network tunnels as VTI :) Wish me luck...

                  1 Reply Last reply Reply Quote 0
                  • O
                    obrienmd
                    last edited by

                    Hrm, with the new devel builds, phase 1s are coming up (and SAs show, with no inbound traffic), and I'm seeing this in ipsec logs, I think the key lines being SADB_ACQUIRE and 'unable to acquire reqid'.

                    CARP is enabled on WANs of one of the sides, and I'm using the CARP IP for 'Interface' on the P1 locally and 'remote gateway' on the P1 remotely. Might this have something to do with it?

                    The other pair I've been testing most with is that where one side is factory, and we had some success earlier, but neither side is HA/CARP. I'll be testing again with that one today as soon as factory images come out.

                    Jun 8 05:18:00	charon		10[CFG] vici client 360 registered for: list-sa
                    Jun 8 05:18:00	charon		10[CFG] vici client 360 requests: list-sas
                    Jun 8 05:18:00	charon		10[CFG] vici client 360 disconnected
                    Jun 8 05:18:00	charon		10[KNL] received an SADB_ACQUIRE with policy id 2 but no matching policy found
                    Jun 8 05:18:00	charon		10[KNL] creating acquire job for policy {local_wan_ip}/32|/0 === {remote_wan_ip}/32|/0 with reqid {0}
                    Jun 8 05:18:00	charon		15[CFG] trap not found, unable to acquire reqid 0
                    Jun 8 05:18:03	charon		15[KNL] <con2|2> querying policy {local_tunnel_ip}/32|/0 === {remote_tunnel_ip}/32|/0 in failed, not found
                    Jun 8 05:18:03	charon		15[KNL] <con2|2> querying policy {remote_tunnel_ip)/32|/0 === {local_tunnel_ip}/32|/0 in failed, not found
                    Jun 8 05:18:03	charon		15[IKE] <con2|2> sending DPD request
                    Jun 8 05:18:03	charon		15[IKE] <con2|2> queueing IKE_DPD task
                    Jun 8 05:18:03	charon		15[IKE] <con2|2> activating new tasks
                    Jun 8 05:18:03	charon		15[IKE] <con2|2> activating IKE_DPD task
                    Jun 8 05:18:03	charon		15[ENC] <con2|2> generating INFORMATIONAL request 253 [ ]
                    Jun 8 05:18:03	charon		15[NET] <con2|2> sending packet: from {local_ip}[500] to {remote_ip}[500] (76 bytes)
                    Jun 8 05:18:03	charon		10[NET] <con2|2> received packet: from {remote_ip)[500] to {local_ip}[500] (76 bytes)
                    Jun 8 05:18:03	charon		10[ENC] <con2|2> parsed INFORMATIONAL response 253 [ ]
                    Jun 8 05:18:03	charon		10[IKE] <con2|2> activating new tasks
                    Jun 8 05:18:03	charon		10[IKE] <con2|2> nothing to initiate
                    
                    1 Reply Last reply Reply Quote 0
                    • jimpJ
                      jimp Rebel Alliance Developer Netgate
                      last edited by

                      Granted I haven't tried it on an HA pair but nothing has changed in a couple days that would affect that. When was the last snapshot you had working there?

                      Can you show the conXXXX entry from /var/etc/ipsec/ipsec.conf for that tunnel?

                      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      1 Reply Last reply Reply Quote 0
                      • O
                        obrienmd
                        last edited by

                        Non-HA side:

                        conn con2
                                fragmentation = yes
                                keyexchange = ikev2
                                reauth = yes
                                forceencaps = no
                                mobike = no
                        
                                rekey = yes
                                installpolicy = no
                        
                                dpdaction = restart
                                dpddelay = 10s
                                dpdtimeout = 60s
                                auto = start
                                left = {non_ha_side_wan_ip}
                                right = {ha_side_wan_ip}
                                leftid = {non_ha_side_wan_ip}
                                ikelifetime = 28800s
                                lifetime = 3600s
                                ike = aes256-sha1-modp1024!
                                esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
                                leftauth = psk
                                rightauth = psk
                                rightid = {ha_side_wan_ip}
                                rightsubnet = 10.90.91.1
                                leftsubnet = 10.90.91.2/30
                        
                        

                        HA side:

                        conn con1
                                fragmentation = yes
                                keyexchange = ikev2
                                reauth = yes
                                forceencaps = no
                                mobike = no
                        
                                rekey = yes
                                installpolicy = no
                        
                                dpdaction = restart
                                dpddelay = 10s
                                dpdtimeout = 60s
                                auto = start
                                left = {ha_side_wan_ip}
                                right = {non_ha_side_wan_ip}
                                leftid = {ha_side_wan_ip}
                                ikelifetime = 28800s
                                lifetime = 3600s
                                ike = aes256-sha1-modp1024!
                                esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
                                leftauth = psk
                                rightauth = psk
                                rightid = {non_ha_side_wan_ip}
                                rightsubnet = 10.90.91.2
                                leftsubnet = 10.90.91.1/30
                        
                        
                        1 Reply Last reply Reply Quote 0
                        • jimpJ
                          jimp Rebel Alliance Developer Netgate
                          last edited by jimp

                          Those don't look quite right, there is no reqid in those blocks like there should be. Are the ipsecX interfaces actually present?

                          Might be due to the tunnel being IKEv2, I think all three of my test systems here have been IKEv1. I'll try to spin up a v2 set.

                          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                          Need help fast? Netgate Global Support!

                          Do not Chat/PM for help!

                          1 Reply Last reply Reply Quote 0
                          • O
                            obrienmd
                            last edited by

                            Yup, ipsec1000/2000 (depending on box) ints are there, and show proper /30s.

                            1 Reply Last reply Reply Quote 0
                            • jimpJ
                              jimp Rebel Alliance Developer Netgate
                              last edited by

                              I did see one problem come up that I just pushed a fix for, but I didn't see that specific error you had unless I had an IKEv1/IKEv2 mismatch between the peers.

                              The fix I made only touches two lines, you can easily apply it manually to test: https://github.com/pfsense/pfsense/commit/d4b43c48ed1636d3fcd6e47d73ba721bd63d883a

                              With that I just switched both sides from IKEv1 to IKEv2 and it came right back up.

                              Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                              Need help fast? Netgate Global Support!

                              Do not Chat/PM for help!

                              1 Reply Last reply Reply Quote 0
                              • O
                                obrienmd
                                last edited by

                                Yep, nailed it. Looking good with that change.

                                Because of your warning on frr, I'm testing with static routing right now. After everything was fixed and I disabled / re-enabled the interfaces to get traffic flowing, static routes were showing in the route table but set to hn1 rather than the ipsec interface. Editing and re-saving the static route resolved the issue.

                                With dynamic routing I bet I won't see that in the future, but if there's some resiliency code somewhere to reset interfaces on static routes when gateways disappear/appear, go up/down, go pending, etc... Perhaps something needs to get tweaked there.

                                1 Reply Last reply Reply Quote 0
                                • jimpJ
                                  jimp Rebel Alliance Developer Netgate
                                  last edited by

                                  I need to work a bit on static routes yet. I had it solved and working on reboot but somewhere in my changes this week that appears to have broken again as I am not seeing my routes in the table after it boots up. I need to investigate more and open another issue up for that.

                                  FRR should be better next week, see my updates on https://redmine.pfsense.org/issues/8449#note-2

                                  Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                  Need help fast? Netgate Global Support!

                                  Do not Chat/PM for help!

                                  1 Reply Last reply Reply Quote 0
                                  • O
                                    obrienmd
                                    last edited by

                                    Great, thanks Jim.

                                    1 Reply Last reply Reply Quote 0
                                    • O
                                      obrienmd
                                      last edited by

                                      Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

                                      jimpJ 1 Reply Last reply Reply Quote 0
                                      • jimpJ
                                        jimp Rebel Alliance Developer Netgate @obrienmd
                                        last edited by

                                        @obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

                                        Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

                                        Not without loading it up and seeing what's in /etc/version.lastcommit. Servers are using CDT.

                                        Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                        Need help fast? Netgate Global Support!

                                        Do not Chat/PM for help!

                                        1 Reply Last reply Reply Quote 0
                                        • jimpJ
                                          jimp Rebel Alliance Developer Netgate
                                          last edited by

                                          Static routes should be OK now. I'm not quite sure how it worked before, given the changes I had to make, but it's working now.

                                          https://github.com/pfsense/pfsense/commit/0aa52fb21a21f58035f2e2fe3b9328a9c175ffb5

                                          I think that might be most if not all of the functional issues. There are still some anti-foot-shooting measures I need to take like preventing removing an IPsec tunnel or P2 used as a VTI interface.

                                          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          1 Reply Last reply Reply Quote 0
                                          • O
                                            obrienmd
                                            last edited by

                                            On latest devel for factory and CE, everything functionally is looking great. Had to restart *pinger (I forget which one is used these days) for gateways to get out of pending after initial interface bring-up, but packets are all flowing, no weird state issues, very solid :)

                                            jimpJ 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.