IPSEC VTI Tunnels



  • Has anyone managed to get tan IPSEC VTI tunnel up and running?

    I've been testing with version 2.4.4.a.20180718.2128 and after several hours i saw a tunnel appear for about 30 seconds with packets passing through the tunnels, although my client still didn't see it.

    I've followed the IPSEC VTI example and still haven't managed to get this working.

    I see a number of errors in the log as shown below:

    /rc.newipsecdns: The command '/sbin/ifconfig 'ipsec1000' create reqid '1000'' returned exit code '1', the output was 'ifconfig: create: bad value'

    Jul 19 21:12:00 charon: 08[KNL] <con1000|13> querying policy 10.6.106.2/32|/0 === 10.6.106.0/30|/0 in failed, not found
    Jul 19 21:12:00 charon: 08[KNL] <con1000|13> querying policy 10.6.106.0/30|/0 === 10.6.106.2/32|/0 out failed, not found

    I've also noticed that pfsense creates a /30 on the local side of the tunnel and a /32 on the oppsite, which may be correct, but worth noting as i expected both to be /30.


  • Rebel Alliance Developer Netgate

    I've done the setup on 5-6 different pairs of VMs and it worked for me every time here.

    Did you follow the documented method here: https://www.netgate.com/docs/pfsense/vpn/ipsec/ipsec-routed.html ?

    That's what I followed for most of the setups I did, since I was confirming the instructions worked while writing them.

    I haven't seen that particular error before. Can you post screenshots of your IPsec P1/P2 config, the assigned interface, IPsec tab firewall rules, your routing setup for the IPsec interface, and any other parts of your config that reference the VTI interface?



  • @jimp Thanks for the fast reponse, i'll give more details.

    I have followed the instructions at the link you mentioned and completed the steps in exactly the same manner. I should note that i have no problem with a standard tunnel and that comes up immediately and stays up. The issue only arises when i switch to VTI.

    I am creating a link between a PFSense and Unifi USG device. I see p1/p2 come up but no traffic passes from the pfsense across the tunnel.

    Running a trace on the VTI interface on my USG i see my LAN traffic hit the router and pass across the VTI. I see the traffic on the PFSense arrive on the IPSEC interface (tagged as authentic, confidential). I see the PFSense respond to the ping and send out a ping reply which never crosses the tunnel. I have set a static route on the pfsense for the LAN traffic in the other site to route over IPSEC interface. The still has no effect.

    The only thing i noticed was that when the tunnel is created it sets a /30 on the pfsense and a /32 on the USG device. I am unable to change the /32 in the PFSense settings as the netmask is disabled in the GUI.

    8_1532053760316_Capture9.PNG 7_1532053760315_Capture8.PNG 6_1532053760314_Capture7.PNG 5_1532053760312_Capture6.PNG 4_1532053760311_Capture5.PNG 3_1532053760310_Capture4.PNG 2_1532053760308_Capture3.PNG 1_1532053760307_Capture2.PNG 0_1532053760305_Capture.PNG

    I have tried a variety of IP addresses on the local and remote in the VTI config, including the one in the online example.


  • Rebel Alliance Developer Netgate

    That's all rather hard to read/parse jumbled up like that with so much blocked out and cut down, but there is still a bit of info missing.

    I still need to see:

    • The P2 settings (from the edit page, not the summary/list)
    • The interface config on Interfaces > OPT1
    • The output of ifconfig ipsec1000
    • The routing table entries for the ipsec1000 interface under Diagnostics > Routes

    On all of mine no matter what I set in the P2 settings for the mask on the local network interface it makes /32 security associations in both directions. I have one pair set for /30 and another set for /24 and both work and show /32 for both ends in the SA list.

    I don't have a USG to test against so there may be some quirk there.

    Also, if you are going to mask/hide info in screenshots it's much better to use a contrasting color so it's obvious where the image has been altered. Use black for a white background, white for a dark background, etc. If it's masked out with the same color as the background it's difficult to discern what is hidden vs what might be incorrect/broken.



  • I've had this problem while trying to set up a VTI tunnel to an EdgeRouter Lite (similar to the USG in hardware, and I'd bet a little bit in software too). I changed the remote network from a single address to a /30 like my local network, and traffic started passing through once the new P2 SA was up.

    But since the webConfigurator won't allow me to put in a network for the remote end, I wonder if this is the "correct" way to do things...



  • Hello, I'm trying to do the same as Zhongfu, but without success.

    @zhongfu what did you do? Where did you change the remote address to a /30, in the pfsense or in the Edgerouter?

    Thanks



  • @abounde on pfSense.

    You could probably just enable the network type field in the phase2 configuration page (with "Inspect Element"?) then configure accordingly, or you could try this in the pfSense shell (assuming you've only got one phase2 config -- if you've got more, just change the [0]s to the appropriate value)

    parse_config(true);
    $config['ipsec']['phase2'][0]['remoteid']['type'] = "network";
    $config['ipsec']['phase2'][0]['remoteid']['netbits'] = 30;
    write_config();
    

  • Rebel Alliance Developer Netgate

    If that worked I can change the GUI to not lock down the remote.

    I originally locked it down because of how the man page for if_ipsec(4) described configuring the interface, it only had a mask on the local component not the remote. If the local is set to /30 it should have the right mask for the interface as a whole. Is that not happening for you?



  • @jimp I think it might be because strongswan doesn't know where to route the packets (?) -- just changing the strongswan configuration to remote_ip/30 for the remote allows traffic to go through



  • @zhongfu thanks for your help, I did the changes (with "Inspect Element"?) and now my VPN Phase 2 setup is this:

    0_1535723585419_028a08b2-c8f9-4489-8383-269cb2b063d9-image.png

    Now, I can ping from the LAN behing the EdgeRouter to the LAN Interface of the pfSense, but I can't ping to a host inside the LAN behind the pfSense.

    In the opposite by, from the LAN behind the pfSense to the LAN behind the EdgeRouter I can't ping anything. But again, I can ping the LAN Interface of the EdgeRouter.

    The firewall rules in both LAN interfaces are complete open.

    My route table in the pfSense is this:
    0_1535723860089_b51a97b6-179a-455d-bd31-c2ca054d2e1c-image.png

    And in my EdgeRouter is this:
    0_1535723934449_23ae1de7-43d7-4ae9-a434-1bc283237f69-image.png

    Could you check in your network/lab if you can ping from host to host, passing by the VPN tunnel.?

    Thanks


  • Rebel Alliance Developer Netgate

    I pushed a change to allow the remote type to be changed to Network. It defaults to address, but once set to Network it will retain that value.



  • @zhongfu could you paste here the EdgeRouter configuration?

    I think my issue now is on the EdgeRouter side.

    Thanks



  • @abounde don't think I can help you there -- I'm using BGP for my setup, and the ERL on the other end isn't mine.

    maybe you'd want your gateway to be the VTI IP of the other end though? (e.g. 10.6.106.2 instead of ipsec1000 for the pfSense site, etc)



  • @jimp It looks like the VTI interface won't come up now, because if_ipsec(4) won't take a subnet mask for the remote component. However, strongswan doesn't seem to route packets between local and remote if the mask for rightsubnet isn't the same as that for leftsubnet

    Perhaps it might be a good idea to strip the subnet mask from the remote before running ifconfig ipsecX local_ip/30 remote_ip, or something similar like only accepting an address for the remote but adding the subnet mask from local_ip to remote_ip in the strongswan config?


  • Rebel Alliance Developer Netgate

    I reverted the commit I made to allow changed on that field for now, since it was broken.

    There seems to be two possible paths here:

    1. Still allow the field to be changed but (a) add input validation to prevent different masks and (b) ignore the mask bits when running ifconfig -- this could be confusing to the user though
    2. Prevent the field from being changed and inject the local mask bits into rightsubnet in the strongSwan config.

    Option #2 is much easier, but I am left to wonder how well that will interact with third party implementations that work now when the remote is an address. It may be fine, but needs testing.

    If you want to try that, use the system patches package to revert da54e84ae79328a87b4a319239bb1b14d7ed2ce6 and then add the attached patch as another entry.
    0_1536597185830_vti_force_rightsubnet_bits.diff



  • @jimp said in IPSEC VTI Tunnels:

    Option #2 is much easier, but I am left to wonder how well that will interact with third party implementations that work now when the remote is an address. It may be fine, but needs testing.

    Hello @jimp, just to inform here is another user wating for a fix.

    SonicWall needs a network as local network. For this i am getting this error:

    IKEv2 Responder: Peer's destination network does not match VPN Policy's [Local Network]
    VPN Policy: XXXNAMEOFVPNXXX; Proposed network: 172.27.3.1-172.27.3.1
    

    It would be fine, if we could select a network as remote network on pfSense. Thanks a lot!


  • Rebel Alliance Developer Netgate

    Palo Alto seems unhappy as well. I have a new patch to test but it does need testing. It comes up and works for me but I don't have access to any of these other devices (ubnt, sonicwall, PA, etc). Also need to be sure it doesn't interfere with other non-IPsec traffic and other non-VTI IPsec tunnels.

    From my other post:

    Try the attached patch and see if it helps. I could not get the VTI to come up and pass traffic with only 0.0.0.0/0 in the rightsubnet and leftsubnet, but it did seem to connect and work with the attached patch that has both the VTI endpoints and all zeroes. I haven't testing to see if it interferes with anything else yet, though, just VTI itself (BGP connects and exchanges routes, traffic passes)

    0_1538745996158_ipsec-vti-0.0.0.0.diff

    Use the System Patches package to apply the diff, or make the changes by hand. After applying the patch, stop IPsec, then edit/save/apply the IPsec VTI P1 or P2 and it should restart with the new policy in place.



  • Hi Jim,

    Long time pfSense user here.

    Thought I would sign up to the forum to contribute to this. I have just installed patch 0_1538745996158_ipsec-vti-0.0.0.0.diff and setup a VTI between the pfSense and an EdgeRouter 4 (running the latest firmware) and I can report that the VPN is now working correctly. I'll let you know if I come across any subsequent strange behaviors, but everything is looking good so far.



  • @jimp I've encountered a similar issue (I could ping the tunnel IP addresses but nothing else) by doing a pfsense-debian buster ipsec connection. I can prepare a testcase on a vultr VM pair if required and ship you the credentials.

    EDIT: I still see these in the log file(s) when I go to status-ipsec:

    Oct 8 13:04:37 192.168.100.1 charon: 04[KNL] <con3000|3> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found
    Oct 8 13:04:37 192.168.100.1 charon: 04[KNL] <con3000|3> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 out failed, not found
    Oct 8 13:04:37 192.168.100.1 charon: 04[KNL] <con2000|2> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found
    Oct 8 13:04:37 192.168.100.1 charon: 04[KNL] <con2000|2> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 out failed, not found
    Oct 8 13:04:37 192.168.100.1 charon: 04[KNL] <con1000|1> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found
    Oct 8 13:04:37 192.168.100.1 charon: 04[KNL] <con1000|1> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 out failed, not found

    But the tunnels are up and passing traffic.


  • Rebel Alliance Developer Netgate

    @fsamareanu said in IPSEC VTI Tunnels:

    @jimp I've encountered a similar issue (I could ping the tunnel IP addresses but nothing else) by doing a pfsense-debian buster ipsec connection. I can prepare a testcase on a vultr VM pair if required and ship you the credentials.

    But the tunnels are up and passing traffic.

    Is this with the new patch applied? If not, apply that patch.

    EDIT: I still see these in the log file(s) when I go to status-ipsec:

    Oct 8 13:04:37 192.168.100.1 charon: 04[KNL] <con3000|3> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found

    I'm not terribly surprised there, since VTI doesn't actually install the policy in the kernel since it isn't needed. That may be prohibitively difficult to suppress that warning but if I do end up committing this patch we can look into it after.



  • @jimp the warning is with the patch applied. The error was there before as well, just showing the /30 subnet and the corresponding remote tunnel ip.

    I have not tested the pfsense-Linux ipsec tunnel after the pfsense patch. Will get to it tomorrow and update here.



  • @turbulence said in IPSEC VTI Tunnels:

    Hi Jim,

    Long time pfSense user here.

    Thought I would sign up to the forum to contribute to this. I have just installed patch 0_1538745996158_ipsec-vti-0.0.0.0.diff and setup a VTI between the pfSense and an EdgeRouter 4 (running the latest firmware) and I can report that the VPN is now working correctly. I'll let you know if I come across any subsequent strange behaviors, but everything is looking good so far.

    Mind sharing your configuration?
    I am trying to get IPSec VTI running between PfSense and EdgeRouter X but i'm not able to get it working. (I already applied latest patch)
    alt text
    alt text
    alt text

    Logging:

    Oct 12 09:11:40	charon		12[KNL] creating acquire job for policy X.X.X.X/32|/0 === X.X.X.X/32|/0 with reqid {0}
    Oct 12 09:11:40	charon		12[KNL] received an SADB_ACQUIRE with policy id 8936 but no matching policy found
    Oct 12 09:11:38	charon		06[CFG] vici client 148 disconnected
    Oct 12 09:11:38	charon		16[CFG] vici client 148 requests: list-sas
    Oct 12 09:11:38	charon		16[CFG] vici client 148 registered for: list-sa
    Oct 12 09:11:38	charon		08[CFG] vici client 148 connected
    


  • Sure thing.

    Here's the ER4 config to start with. BTW, you need to be using IKEV2.

    ==PEER CONFIG==
    show vpn ipsec site-to-site peer X.X.X.X
    authentication {
    mode pre-shared-secret
    pre-shared-secret SECRETGOESHERE
    }
    connection-type initiate
    description TUNNEL-NAME-HERE
    ike-group FOO4
    ikev2-reauth yes
    local-address Y.Y.Y.Y
    vti {
    bind vti4
    esp-group FOO4
    }

    ==ESP CONFIG==
    show vpn ipsec esp-group FOO4
    compression disable
    lifetime 28800
    mode tunnel
    pfs dh-group14
    proposal 1 {
    encryption aes256
    hash sha256
    }

    ==IKE CONFIG==
    show vpn ipsec ike-group FOO4
    ikev2-reauth yes
    key-exchange ikev2
    lifetime 28800
    proposal 1 {
    dh-group 14
    encryption aes256
    hash sha256
    }

    ==VTI CONFIG==
    show interfaces vti vti4
    address 10.10.202.2/30
    mtu 1436

    ==ROUTE CONFIG==
    show protocols static interface-route 172.24.16.0/24
    next-hop-interface vti4 {
    description TUNNEL-NAME-HERE
    }



  • And here's the PFSense configuration.

    Let me know if you need any further assistance!

    2_1539329677025_Routes.PNG 1_1539329677024_Phase2.png 0_1539329677024_Phase1.png



  • @turbulence said in IPSEC VTI Tunnels:

    And here's the PFSense configuration.

    Let me know if you need any further assistance!

    Thank you for the information. Will try it out later today and report back.



  • @turbulence said in IPSEC VTI Tunnels:

    And here's the PFSense configuration.

    Let me know if you need any further assistance!

    Ok, I tried to do it with IKEv2 and instead of Type 'Network' on Remote Network in PfSense Phase 2 setting, I used Type 'Address'.

    With those settings the tunnel will come online. But I'm still not able to pass traffic.
    What I found out is the following:
    When I start a packetdump on PfSense I see ICMP traffic. (192.168.111.1 --> 192.168.111.2)
    On EdgeRouter X side I see the ICMP messages arrive and the router also responds, but the response packets never reach the IPSEC1000 interface on the PfSense.

    PfSense
    alt text

    EdgeRouter
    alt text



  • Hello Everyone,
    I got vti setup working between pfsense and edgerouter pro with ebgp in place . Patch for 0.0.0.0/0 is required.

    0_1539434435515_vti_pfsense.png

    key point was Address in phase 2 and on edge router firewall allow BGP in VPN zone. VPN configuration on edge router is standard vti setup.

    Phase 2

    0_1539435267226_vti_phase2.png

    The only one think is that tunnel goes down time to time. In logs I see

    Oct 13 08:48:58 php-fpm 35862 /rc.newipsecdns: Gateway, none 'available' for inet6, use the first one configured. ''
    Oct 13 08:48:58 php-fpm 35862 /rc.newipsecdns: The command '/sbin/ifconfig 'ipsec3000' create reqid '3000'' returned exit code '1', the output was 'ifconfig: create: bad value'
    Oct 13 08:48:57 check_reload_status Reloading filter
    Oct 13 08:48:57 php-fpm 35862 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Oct 13 08:48:42 php-fpm 61417 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
    Oct 13 08:48:41 check_reload_status Reloading filter



  • My problem is solved. On the other side (EdgeRouter side) I use Dual-WAN with LB. I'm migrating this tunnel from VyOS to PfSense. The PfSense public IP was being loadbalanced. This caused the IPSec traffic to go over the wrong outside interface. After adding the public IP to the LB Exclude list, things started working.

    Now I am running into another (NAT) problem. Maybe one of you can test if you get the same result. If I add an NAT rule to masquerade traffic with IP-address of VTI interface I am not able to reach anything on the EdgeRouter side.

    What I am doing to test:
    Ping from PfSense CLI to 192.168.111.2 (EdgeRouter). This works without the NAT rule. When I enable the NAT rule it stops working.

    So far my observations are:
    Traffic is leaving IPsec1000 interface on PfSense with correct NAT address (192.168.111.1). Traffic arrives on interface EdgeRouter and EdgeRouter sends traffic back which also arrives back at the IPsec1000 interface. But PfSense never gets reply on ICMP-echo request.

    NAT rule looks like this. (Interface is VTI interface)
    alt text

    @turbulence



  • I am watching tunnel and it not 100% stable. Sometimes pfsense stop reply to traffic from vpn tunnel. I see traffic arrive on vti interface, but never send reply.


  • Rebel Alliance Developer Netgate

    @bluray said in IPSEC VTI Tunnels:

    Now I am running into another (NAT) problem. Maybe one of you can test if you get the same result. If I add an NAT rule to masquerade traffic with IP-address of VTI interface I am not able to reach anything on the EdgeRouter side.

    That's a known issue, NAT doesn't work with VTI currently. There is some weirdness between if_ipsec/VTI and pf where the traffic hits both the ipsecX interface and enc0. You might try putting the NAT rule on the "IPsec" interface in the GUI and not the assigned VTI interface.



  • @jimp said in IPSEC VTI Tunnels:

    @bluray said in IPSEC VTI Tunnels:

    Now I am running into another (NAT) problem. Maybe one of you can test if you get the same result. If I add an NAT rule to masquerade traffic with IP-address of VTI interface I am not able to reach anything on the EdgeRouter side.

    That's a known issue, NAT doesn't work with VTI currently. There is some weirdness between if_ipsec/VTI and pf where the traffic hits both the ipsecX interface and enc0. You might try putting the NAT rule on the "IPsec" interface in the GUI and not the assigned VTI interface.

    Hello Jim,

    Thank you for your feedback.
    Unfortunately that doesn't work in my case. Is there a bugreport for this so I can track this issue? If not how can I make one?


  • Rebel Alliance Developer Netgate

    I don't recall if I made an issue for it on Redmine, but it's not one that we can address. It will need to be taken upstream to FreeBSD directly.



  • I already searched in Redmine, but I couldn't find one. Your explanation explains why there is no. (Due to it being a bug in FreeBSD IPSec implementation which must be fixed by FreeBSD Developers)

    I'm not familiair with reporting bugs. Will do some research how to submit one. Hopefully this can be resolved in the future.