IPsec VTI does not pass traffic on 2.6.0
-
Yes same problem here - was running 2.5.2 (which broke the IPSEC connection viewing status)
Upgraded to 2.6.0 39 of 40 IPSEC connections connected first time (I hoped it was just a glitch)Woke up this morning to a nightmare of tunnels just not connecting about 8 were just a mess and not functional they were connecting back to versions 2.5.1 mostly)
Luckily it was a VM so rolled back to 2.5.2 and all tunnels came up immediately.
-
@jimp If you're interested I'm able to fire up the VM with 2.6 on it and you can see what its doing? It will need to be outside of business hours Australia AEST which might suit you? This was a clean upgrade from 2.5.2 to 2.6
-
What are your IPsec firewall rules like?
Generic open rule / Allow all traffic
Do you see anything get dropped in the firewall log?
Nothing for IPsec.
Any errors in the IPsec log?
2022-02-06 00:29:54.678808+00:00 charon 29338 09[KNL] <con1|2> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found
/var/etc/ipsec/swanctl.conf
for one of the routers.# This file is automatically generated. Do not edit connections { bypass { remote_addrs = 127.0.0.1 children { bypasslan { local_ts = 10.0.0.0/22,fc00::/7 remote_ts = 10.0.0.0/22,fc00::/7 mode = pass start_action = trap } } } con1 { # P1 (ikeid 1): <redacted> fragmentation = yes unique = replace version = 2 proposals = aes256gcm128-sha512-modp8192 dpd_delay = 10s rekey_time = 25920s reauth_time = 0s over_time = 2880s rand_time = 2880s encap = no mobike = no local_addrs = <redacted> remote_addrs = <redacted> local { id = fqdn:<redacted> auth = psk } remote { id = fqdn:<redacted> auth = psk } children { con1 { # P2 (reqid 2) policies = no life_time = 3600s rekey_time = 3240s rand_time = 360s start_action = start remote_ts = 10.0.100.2,0.0.0.0/0,::/0 local_ts = 10.0.100.1,0.0.0.0/0,::/0 reqid = 5001 esp_proposals = aes256gcm128-modp8192 close_action = start dpd_action = restart } } } } secrets { ike-0 { secret = <redacted>= id-0 = %any id-1 = fqdn:<redacted> } }
-
@thatsysadmin The IPSEC firewalls do have quite a bit going on however nothing specific to the sites that didnt come back online.
The sites that didnt reconnect had errors about already being connected but were trying to reconnect. (sorry thats hopeless but I didnt record it - gave up and reverted to 2.5)
Can do some testing in 6 hours. 8pm AEST if it will help
-
Just to add...my config uses an FQDN as the remote gateway and I noticed @thatsysadmin also mentions
fqdn
in his config.Just throwing it out there whether this has anything to do with it in comparison to @jimp's VTI tunnels in the lab.
-
@kev-i-n no FQDN in my configs (my IP/Peer address)
(they all initially connected except one on boot - they just didnt reconnect) - Stopped IPSEC service on both and started again.
Same sites remained downGenerally
IKEv2
P1 AES256-CGM 128 / AES-XCBC - HD 14
P2 ESP AES128/AES256-CGM
Key groups offDPD enabled
Unique ID (replace)
Filter IPSEC Tunnel and VTI on IPSEC tab (ENC0) anything there? -
@kev-i-n
I still had the same issue with the "IP option" I changed it as a troubleshooting measure. -
There appear to be a couple different problems here getting lumped together.
If your VTI interfaces connect but do not pass traffic, this is the correct thread.
If you have any other issue on 22.01/2.6.0 such as tunnels failing to connect or reconnect, please start a new thread and post your information there. Be sure to include log entries from both sides as well as the information requested previously in this thread (IPsec config for the tunnel from config.xml and swanctl.conf, ifconfig output, etc).
Now, back to connected VTIs not passing traffic:
I'm still not getting the whole picture from these and the thread has gotten a bit mixed up with the other unrelated info. With one of these VTI tunnels in a state that fails to pass traffic I need the following information:
- The output of
ifconfig -a
for theipsecX
interface(s) - The output of
setkey -D
andsetkey -DP
- The contents of
/var/etc/ipsec/swanctl.conf
for the tunnel(s) - The contents of
config.xml
for the tunnel(s) -- both P1 and all P2s
As with before you can mask private info but try to keep it consistent. For example, replace unique IP addresses consistently, e.g. 1.2.3.4 with A.A.A.A and 9.8.7.6 with B.B.B.B so I can tell they are different IP addresses in the output. Mask or remove any private things like PSKs or identifiers if they are sensitive.
If both sides are running pfSense software then having the info from both sides would help.
- The output of
-
@jimp Hi very happy to start a new thread - lets call it 2.6.0 ipsec is broken - dont upgrade (yet).
Seriously - in the few days there appears to a lot of activity around ipsec having issues on 2.6 and people here trying to assist and fix their broken networks (remember this is a release version)
As a netgate developer how about you start a new thread for Systems that have had ipsec fail after upgrading from 2.5 to 2.6 and from there you can be particular about what information you want to put where.
Whether it be the VTI is broken, the tunnel is broken, the routing is broken - hey it just stopped working after upgrading. Most customers are coming here looking or answers on why it stopped after upgrading.
My offer is still open for you to look at a working 2.5 - we can revert to 2.6 and you can diagnose it anyway you want. You can have remote access; Im keen for this to be resolved rather than sitting back concerned im posting into the wrong thread about a general ipsec failure.
-
It is working for the vast majority of people, such a dire warning is unwarranted. It has been thoroughly tested internally and over the last six months or more in snapshots including heavy use on Netgate infrastructure used by all of our employees.
Keep your thread subjects relevant, for example "IPsec VTI tunnel will not reconnect on 2.6.0" or similar. Do not assume it's happening to anyone but you, and do not assume your problem is identical to others. Only after diagnosing the problem can such a determination be made.
It's important to keep each report separate so that the details do not get confused.
-
This post is deleted! -
@jimp
Here's the outputs.
https://github.com/thatsysadmin/pfsense_2.6.0_IPsec_troubleshootingOne thing;
/var/etc/swanctl.conf
doesn't exist, did you mean/var/etc/ipsec/strongswan.conf
? -
@thatsysadmin said in IPsec VTI does not pass traffic on 2.6.0:
@jimp
Here's the outputs.
https://github.com/thatsysadmin/pfsense_2.6.0_IPsec_troubleshootingOne thing;
/var/etc/swanctl.conf
doesn't exist, did you mean/var/etc/ipsec/strongswan.conf
?I meant
/var/etc/ipsec/swanctl.conf
, I've edited the post above. I'll check out the other info and see if anything stands out. -
You have two P2s on that tunnel, one tunnel mode and one VTI. That isn't a valid configuration and it's unnecessary. You should remove the tunnel mode entry (not just disable it). If you do that and then stop IPsec, then start IPsec, it might start working.
That would explain the reqid mismatching which is likely why traffic isn't passing. In
setkey
it shows it's looking for reqid5001
but inifconfig
the interface is set for5002
. -
@jimp
After a reconfiguration of the interfaces, it works. Thanks for all your help.But why would having one of the phase 2s disabled break the whole thing though; shouldn't it be disregarded if it was disabled?
-
@thatsysadmin ahh interesting - I think I might have some disabled too.. After 2.5.2 left the status on P2 messed up i've not been actively monitoring them. If you have a testing setup does disabling them at EITHER end break 2.6
-
I too also had some lingering Phase 2's configured for tunnel mode, albeit in a disabled state as they were from before I made the switch to route-based.
After wrestling to remove these within 2.5.2 - which required reconfiguring the VTI setup - the VPN eventually started passing traffic again. After this, I took the plunge to 2.6.0 and it came back up successfully without any changes needed.
So it does seem like the common issue here in that if using routed mode (VTI) with old tunnel mode phase 2's still setup - even if they're disabled - prevents the s2s from passing traffic.
Glad we got there in the end :) Thanks for your assistance @jimp
-
@thatsysadmin said in IPsec VTI does not pass traffic on 2.6.0:
But why would having one of the phase 2s disabled break the whole thing though; shouldn't it be disregarded if it was disabled?
It could probably handle that better, but it's not a valid combination to have a mix of tunnel and VTI even if some are disabled. They should all be the same type, and really there should be at most one VTI P2 per address family (so one IPv4, one IPv6). I'm not sure if we have validation which actively checks for and prevents that yet, though.