Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Routed IPsec using if_ipsec VTI interfaces

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    45 Posts 2 Posters 11.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      Looks like it's better and worse. I can pass traffic between the hosts that failed before, but the gateways are not being generated properly so I need to fix that up, so static routes won't work and so on.

      I see the code blocks I didn't update to the new style so I'll fix those up in the morning.

      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • O
        obrienmd
        last edited by

        Thanks Jim!

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          OK, I just pushed the updated gateway code and it's working well for me now.

          I do, however, see the same behavior you did where the firewall can't reach a routed network on the far side using the ipsec interface address as the source. It does work if I set the source to be the LAN, however.

          Using the ipsecX interface address as the source:

          : ping -S 10.6.106.1 10.7.0.1
          PING 10.7.0.1 (10.7.0.1) from 10.6.106.1: 56 data bytes
          ^C
          --- 10.7.0.1 ping statistics ---
          2 packets transmitted, 0 packets received, 100.0% packet loss
          

          Going LAN to LAN from the firewall:

          : ping -S 10.6.0.1 10.7.0.1
          PING 10.7.0.1 (10.7.0.1) from 10.6.0.1: 56 data bytes
          64 bytes from 10.7.0.1: icmp_seq=0 ttl=64 time=0.802 ms
          64 bytes from 10.7.0.1: icmp_seq=1 ttl=64 time=0.883 ms
          64 bytes from 10.7.0.1: icmp_seq=2 ttl=64 time=0.716 ms
          ^C
          --- 10.7.0.1 ping statistics ---
          3 packets transmitted, 3 packets received, 0.0% packet loss
          

          Routes all look correct, on the source node traffic appears on the ipsecX and enc0 interface but the counters on the child SA do not increase and no ESP leaves, so somehow it isn't making its way to that connection. I'll keep poking at it, but it's not the end of the world since the same situation also didn't work on plain IPsec, though we hoped routed IPsec would be a cure for that.

          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • jimpJ
            jimp Rebel Alliance Developer Netgate
            last edited by

            Since that firewall-to-LAN routing issue is not a flaw in the VTI code that I can see, I've split that off into https://redmine.pfsense.org/issues/8551

            Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 0
            • O
              obrienmd
              last edited by

              Makes perfect sense to me - as soon as the daily build hits pfSense factory -devel, I'll start testing again!

              1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate
                last edited by

                OK, I think I have that nailed down. Apparently it does not get along with pf route-to directly on the interface. It works fine for LAN traffic but not traffic exiting from the firewall itself. I pushed a fix, should be in snaps soonish.

                Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • O
                  obrienmd
                  last edited by

                  Cool, thanks!

                  Question - and this might be sacrilege - can I set my Factory boxes to download CE snapshots? I poked around repos but when it looks like just swapping the pfSense.conf one didn't really work out on a test box :)

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    @obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

                    Cool, thanks!

                    Question - and this might be sacrilege - can I set my Factory boxes to download CE snapshots? I poked around repos but when it looks like just swapping the pfSense.conf one didn't really work out on a test box :)

                    Not easily, several things need adjusted and it's just not worth the hassle to downgrade like that in-place. All the changes I made today, including the fix for that route-to issue, have been synchronized to Factory so it should show up in snapshots for both CE and Factory by the morning.

                    Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • O
                      obrienmd
                      last edited by

                      Much appreciated.

                      This is going to help in a lot of places... Now I just have to get Verizon to terminate mobile private network tunnels as VTI :) Wish me luck...

                      1 Reply Last reply Reply Quote 0
                      • O
                        obrienmd
                        last edited by

                        Hrm, with the new devel builds, phase 1s are coming up (and SAs show, with no inbound traffic), and I'm seeing this in ipsec logs, I think the key lines being SADB_ACQUIRE and 'unable to acquire reqid'.

                        CARP is enabled on WANs of one of the sides, and I'm using the CARP IP for 'Interface' on the P1 locally and 'remote gateway' on the P1 remotely. Might this have something to do with it?

                        The other pair I've been testing most with is that where one side is factory, and we had some success earlier, but neither side is HA/CARP. I'll be testing again with that one today as soon as factory images come out.

                        Jun 8 05:18:00	charon		10[CFG] vici client 360 registered for: list-sa
                        Jun 8 05:18:00	charon		10[CFG] vici client 360 requests: list-sas
                        Jun 8 05:18:00	charon		10[CFG] vici client 360 disconnected
                        Jun 8 05:18:00	charon		10[KNL] received an SADB_ACQUIRE with policy id 2 but no matching policy found
                        Jun 8 05:18:00	charon		10[KNL] creating acquire job for policy {local_wan_ip}/32|/0 === {remote_wan_ip}/32|/0 with reqid {0}
                        Jun 8 05:18:00	charon		15[CFG] trap not found, unable to acquire reqid 0
                        Jun 8 05:18:03	charon		15[KNL] <con2|2> querying policy {local_tunnel_ip}/32|/0 === {remote_tunnel_ip}/32|/0 in failed, not found
                        Jun 8 05:18:03	charon		15[KNL] <con2|2> querying policy {remote_tunnel_ip)/32|/0 === {local_tunnel_ip}/32|/0 in failed, not found
                        Jun 8 05:18:03	charon		15[IKE] <con2|2> sending DPD request
                        Jun 8 05:18:03	charon		15[IKE] <con2|2> queueing IKE_DPD task
                        Jun 8 05:18:03	charon		15[IKE] <con2|2> activating new tasks
                        Jun 8 05:18:03	charon		15[IKE] <con2|2> activating IKE_DPD task
                        Jun 8 05:18:03	charon		15[ENC] <con2|2> generating INFORMATIONAL request 253 [ ]
                        Jun 8 05:18:03	charon		15[NET] <con2|2> sending packet: from {local_ip}[500] to {remote_ip}[500] (76 bytes)
                        Jun 8 05:18:03	charon		10[NET] <con2|2> received packet: from {remote_ip)[500] to {local_ip}[500] (76 bytes)
                        Jun 8 05:18:03	charon		10[ENC] <con2|2> parsed INFORMATIONAL response 253 [ ]
                        Jun 8 05:18:03	charon		10[IKE] <con2|2> activating new tasks
                        Jun 8 05:18:03	charon		10[IKE] <con2|2> nothing to initiate
                        
                        1 Reply Last reply Reply Quote 0
                        • jimpJ
                          jimp Rebel Alliance Developer Netgate
                          last edited by

                          Granted I haven't tried it on an HA pair but nothing has changed in a couple days that would affect that. When was the last snapshot you had working there?

                          Can you show the conXXXX entry from /var/etc/ipsec/ipsec.conf for that tunnel?

                          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                          Need help fast? Netgate Global Support!

                          Do not Chat/PM for help!

                          1 Reply Last reply Reply Quote 0
                          • O
                            obrienmd
                            last edited by

                            Non-HA side:

                            conn con2
                                    fragmentation = yes
                                    keyexchange = ikev2
                                    reauth = yes
                                    forceencaps = no
                                    mobike = no
                            
                                    rekey = yes
                                    installpolicy = no
                            
                                    dpdaction = restart
                                    dpddelay = 10s
                                    dpdtimeout = 60s
                                    auto = start
                                    left = {non_ha_side_wan_ip}
                                    right = {ha_side_wan_ip}
                                    leftid = {non_ha_side_wan_ip}
                                    ikelifetime = 28800s
                                    lifetime = 3600s
                                    ike = aes256-sha1-modp1024!
                                    esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
                                    leftauth = psk
                                    rightauth = psk
                                    rightid = {ha_side_wan_ip}
                                    rightsubnet = 10.90.91.1
                                    leftsubnet = 10.90.91.2/30
                            
                            

                            HA side:

                            conn con1
                                    fragmentation = yes
                                    keyexchange = ikev2
                                    reauth = yes
                                    forceencaps = no
                                    mobike = no
                            
                                    rekey = yes
                                    installpolicy = no
                            
                                    dpdaction = restart
                                    dpddelay = 10s
                                    dpdtimeout = 60s
                                    auto = start
                                    left = {ha_side_wan_ip}
                                    right = {non_ha_side_wan_ip}
                                    leftid = {ha_side_wan_ip}
                                    ikelifetime = 28800s
                                    lifetime = 3600s
                                    ike = aes256-sha1-modp1024!
                                    esp = aes256gcm128-sha256-modp2048,aes256gcm96-sha256-modp2048,aes256gcm64-sha256-modp2048!
                                    leftauth = psk
                                    rightauth = psk
                                    rightid = {non_ha_side_wan_ip}
                                    rightsubnet = 10.90.91.2
                                    leftsubnet = 10.90.91.1/30
                            
                            
                            1 Reply Last reply Reply Quote 0
                            • jimpJ
                              jimp Rebel Alliance Developer Netgate
                              last edited by jimp

                              Those don't look quite right, there is no reqid in those blocks like there should be. Are the ipsecX interfaces actually present?

                              Might be due to the tunnel being IKEv2, I think all three of my test systems here have been IKEv1. I'll try to spin up a v2 set.

                              Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                              Need help fast? Netgate Global Support!

                              Do not Chat/PM for help!

                              1 Reply Last reply Reply Quote 0
                              • O
                                obrienmd
                                last edited by

                                Yup, ipsec1000/2000 (depending on box) ints are there, and show proper /30s.

                                1 Reply Last reply Reply Quote 0
                                • jimpJ
                                  jimp Rebel Alliance Developer Netgate
                                  last edited by

                                  I did see one problem come up that I just pushed a fix for, but I didn't see that specific error you had unless I had an IKEv1/IKEv2 mismatch between the peers.

                                  The fix I made only touches two lines, you can easily apply it manually to test: https://github.com/pfsense/pfsense/commit/d4b43c48ed1636d3fcd6e47d73ba721bd63d883a

                                  With that I just switched both sides from IKEv1 to IKEv2 and it came right back up.

                                  Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                  Need help fast? Netgate Global Support!

                                  Do not Chat/PM for help!

                                  1 Reply Last reply Reply Quote 0
                                  • O
                                    obrienmd
                                    last edited by

                                    Yep, nailed it. Looking good with that change.

                                    Because of your warning on frr, I'm testing with static routing right now. After everything was fixed and I disabled / re-enabled the interfaces to get traffic flowing, static routes were showing in the route table but set to hn1 rather than the ipsec interface. Editing and re-saving the static route resolved the issue.

                                    With dynamic routing I bet I won't see that in the future, but if there's some resiliency code somewhere to reset interfaces on static routes when gateways disappear/appear, go up/down, go pending, etc... Perhaps something needs to get tweaked there.

                                    1 Reply Last reply Reply Quote 0
                                    • jimpJ
                                      jimp Rebel Alliance Developer Netgate
                                      last edited by

                                      I need to work a bit on static routes yet. I had it solved and working on reboot but somewhere in my changes this week that appears to have broken again as I am not seeing my routes in the table after it boots up. I need to investigate more and open another issue up for that.

                                      FRR should be better next week, see my updates on https://redmine.pfsense.org/issues/8449#note-2

                                      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                      Need help fast? Netgate Global Support!

                                      Do not Chat/PM for help!

                                      1 Reply Last reply Reply Quote 0
                                      • O
                                        obrienmd
                                        last edited by

                                        Great, thanks Jim.

                                        1 Reply Last reply Reply Quote 0
                                        • O
                                          obrienmd
                                          last edited by

                                          Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

                                          jimpJ 1 Reply Last reply Reply Quote 0
                                          • jimpJ
                                            jimp Rebel Alliance Developer Netgate @obrienmd
                                            last edited by

                                            @obrienmd said in Routed IPsec using if_ipsec VTI interfaces:

                                            Is there a simple way to map a devel release, e.g. 2.4.4.a.20180608.1025 for Factory or 2.4.4.a.20180608.0718 for CE, against a git commit? I don't want to assume it will be build using all commits immediately prior to that (and I don't know which time zones these are based on).

                                            Not without loading it up and seeing what's in /etc/version.lastcommit. Servers are using CDT.

                                            Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                            Need help fast? Netgate Global Support!

                                            Do not Chat/PM for help!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.