Anyone using VTI with ASR1000 on other end

obrienmd

Loving new VTI functionality! We have a few firewalls already using 2.4.4 with VTI between them, and getting the benefits of IPSec (mostly perf) with routed interfaces has been awesome.

In one deployment, we just tested moving from Transport P2 + GRE tunnels, and all their state weirdness, to VTI. The other end is a Cisco ASR1000. We did the following to make the change:

Cisco side reconfigured their tunnels / interfaces
Removed GRE interfaces
Removed GRE tunnels
Changed P2 from transport to VTI and configured per pfSense docs (/30 local, address remote)
Added interface)
Configured a wide open firewall rule on the ipsec interface

Strangely, the P1 part of the tunnels now seems to drop/reconnect every minute - gateway packet loss goes up to 100% average, then comes back down when it reconnects, again and again. When it is up, the VTI tunnel works great - BGP sessions come up, routes come across both sides, traffic flows, etc.

Local ip is 10.10.110.9/30
Remote ip is 10.10.110.8

In system log, I see (might just be a side-effect of the bouncing):
Oct 11 15:43:41 php-fpm 85870 /rc.newipsecdns: The command '/sbin/ifconfig 'ipsec5000' create reqid '5000'' returned exit code '1', the output was 'ifconfig: create: bad value'
Oct 11 15:43:41 check_reload_status Reloading filter
Oct 11 15:43:41 php-fpm 85870 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
Oct 11 15:43:26 php-fpm 353 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use PFSENSEINTERFACENAME_VTIV4.
Oct 11 15:43:25 check_reload_status Reloading filter
Oct 11 15:43:25 check_reload_status Restarting OpenVPN tunnels/interfaces
Oct 11 15:43:25 check_reload_status Restarting ipsec tunnels
Oct 11 15:43:25 check_reload_status updating dyndns PFSENSEINTERFACENAME_VTIV4
Oct 11 15:43:25 rc.gateway_alarm 6908 >>> Gateway alarm: PFSENSEINTERFACENAME_VTIV4 (Addr:10.10.110.9 Alarm:1 RTT:34.345ms RTTsd:.549ms Loss:22%)

In ipsec log, I see lots of:
07[KNL] <con5000|192> querying policy 10.10.110.9/32|/0 === 10.10.110.8/30|/0 in failed, not found

Anyone succesfully using VTI against an ASR1k or other Cisco kit?

volga629

I think the issue that routing based vpn like vti left and right subnet should be set 0.0.0.0/0 not vti ip address
that break the tunnel
in config set
rightsubnet = 192.168.252.2
leftsubnet = 192.168.252.1/30
that incorrect tunnel will be never established because ipip is on top vti and based on encrypted domain ip addresses where normally it public ip's from side A and side B.

From 192.168.252.2 icmp_seq=2 Destination Host Unreachable

I am trying vti tunnel with ubnt edge pro router.
PFSENSE devs please confirm.

jimp

Try with this patch:

https://redmine.pfsense.org/issues/8859#note-9

I'd also be curious to know if it hurts your other working setups in any way. I don't expect it to, I haven't seen any negative side effects thus far, but I haven't seen enough feedback to be confident in it yet.

volga629

Thank you let me apply patch

in order avoid overlap or conflict you have to use pass through for local subnet

conn dev-host
type=passthrough
authby=never
left=my lan subnet list
right=my lan subnet list
auto=route

jimp

You should not need any of that with VTI, just the stock lines + this patch that adds 0.0.0.0/0 both ways.

volga629

Is it possible set just 0.0.0.0/0 ?
In config I see

rightsubnet = 192.168.252.2,0.0.0.0/0
leftsubnet = 192.168.252.1/30,0.0.0.0/0

jimp

FreeBSD requires the VTI endpoints be there. Without that, it doesn't work, even with just 0.0.0.0/0. That's the smallest possible set I could get to work.

volga629

Ok, but it not resolve the issue. Vti still not coming up.

volga629@canlrt01:~$ ping 192.168.252.1
PING 192.168.252.1 (192.168.252.1) 56(84) bytes of data.
From 192.168.252.2 icmp_seq=1 Destination Host Unreachable
From 192.168.252.2 icmp_seq=2 Destination Host Unreachable
From 192.168.252.2 icmp_seq=3 Destination Host Unreachable
From 192.168.252.2 icmp_seq=4 Destination Host Unreachable
^C
--- 192.168.252.1 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 2997ms

jimp

There is already a different thread for problems with ubnt/edgerouters though, if you need help, post there or start a thread of your own. This one is specifically for an ASR1000

obrienmd

Testing patch from https://redmine.pfsense.org/attachments/2624/ipsec-vti-0.0.0.0.diff now :)

obrienmd

@jimp Good to go against ASR1000 with that patch - tunnel is no longer bouncing. There's a secondary tunnel on the same box not coming up, but I think that's on their side (P1 up, P2 not returning any traffic though pings are going out).

Regardless, I'll drop in next week after our maintenance window to let you know if they've fixed it.

I don't have a good test case against other ipsec site-to-site or mobile tunnels on this test box - the only one with those is on prod :) I will do my best to spin something up on this so that you can be more confident in the patch going into -p1 :)

Thanks Jim!