pfSense 2.5.0 broke all IPSec VPNs

A Former User

I got pretty excited when I saw 2.5.0 released so I upgraded my lab installation. This lab installation has several IPSec VPNs, going to a Unifi site, OPNSense, and several other pfSense sites, all running 2.4.5-p1. When the upgrade completed, I had to remote into a console within my lab as the IPSec tunnel to my lab never came back up. Once in, pfSense wasn't bringing up any of the IPSec tunnels. Before I reverted my snapshot back, I got the following out of the logs:

Feb 17 12:14:33	charon	93047	07[IKE] <con100000|386> sending retransmit 2 of request message ID 0, seq 1
Feb 17 12:14:33	charon	93047	07[NET] <con100000|386> sending packet: from Z.Z.Z.Z[500] to Y.Y.Y.Y[500] (180 bytes)
Feb 17 12:14:36	charon	93047	07[NET] <391> received packet: from A.A.A.A[500] to Z.Z.Z.Z[500] (180 bytes)
Feb 17 12:14:36	charon	93047	07[ENC] <391> parsed ID_PROT request 0 [ SA V V V V V ]
Feb 17 12:14:36	charon	93047	07[CFG] <391> looking for an IKEv1 config for Z.Z.Z.Z...A.A.A.A
Feb 17 12:14:36	charon	93047	07[IKE] <391> no IKE config found for Z.Z.Z.Z...A.A.A.A, sending NO_PROPOSAL_CHOSEN
Feb 17 12:14:36	charon	93047	07[ENC] <391> generating INFORMATIONAL_V1 request 2574132598 [ N(NO_PROP) ]
Feb 17 12:14:36	charon	93047	07[NET] <391> sending packet: from Z.Z.Z.Z[500] to A.A.A.A[500] (40 bytes)
Feb 17 12:14:36	charon	93047	07[IKE] <391> IKE_SA (unnamed)[391] state change: CREATED => DESTROYING
Feb 17 12:14:44	charon	93047	15[IKE] <con600000|376> sending retransmit 4 of request message ID 0, seq 1
Feb 17 12:14:44	charon	93047	15[NET] <con600000|376> sending packet: from Z.Z.Z.Z[500] to A.A.A.A[500] (180 bytes)
Feb 17 12:14:44	charon	93047	15[NET] <392> received packet: from Y.Y.Y.Y[500] to Z.Z.Z.Z[500] (180 bytes)
Feb 17 12:14:44	charon	93047	15[ENC] <392> parsed ID_PROT request 0 [ SA V V V V V ]
Feb 17 12:14:44	charon	93047	15[CFG] <392> looking for an IKEv1 config for Z.Z.Z.Z...Y.Y.Y.Y
Feb 17 12:14:44	charon	93047	15[IKE] <392> no IKE config found for Z.Z.Z.Z...Y.Y.Y.Y, sending NO_PROPOSAL_CHOSEN
Feb 17 12:14:44	charon	93047	15[ENC] <392> generating INFORMATIONAL_V1 request 291910428 [ N(NO_PROP) ]
Feb 17 12:14:44	charon	93047	15[NET] <392> sending packet: from Z.Z.Z.Z[500] to Y.Y.Y.Y[500] (40 bytes)
Feb 17 12:14:44	charon	93047	15[IKE] <392> IKE_SA (unnamed)[392] state change: CREATED => DESTROYING
Feb 17 12:14:46	charon	93047	15[IKE] <con100000|386> sending retransmit 3 of request message ID 0, seq 1
Feb 17 12:14:46	charon	93047	15[NET] <con100000|386> sending packet: from V.V.V.V[500] to Y.Y.Y.Y[500] (180 bytes)
Feb 17 12:14:47	charon	93047	15[NET] <393> received packet: from W.W.W.W[500] to Z.Z.Z.Z[500] (156 bytes)
Feb 17 12:14:47	charon	93047	15[ENC] <393> parsed ID_PROT request 0 [ SA V V V V ]
Feb 17 12:14:47	charon	93047	15[CFG] <393> looking for an IKEv1 config for Z.Z.Z.Z...W.W.W.W
Feb 17 12:14:47	charon	93047	15[IKE] <393> no IKE config found for Z.Z.Z.Z...W.W.W.W, sending NO_PROPOSAL_CHOSEN
Feb 17 12:14:47	charon	93047	15[ENC] <393> generating INFORMATIONAL_V1 request 2478788730 [ N(NO_PROP) ]
Feb 17 12:14:47	charon	93047	15[NET] <393> sending packet: from Z.Z.Z.Z[500] to W.W.W.W[500] (40 bytes)
Feb 17 12:14:47	charon	93047	15[IKE] <393> IKE_SA (unnamed)[393] state change: CREATED => DESTROYING
Feb 17 12:14:48	charon	93047	15[NET] <394> received packet: from X.X.X.X[500] to Z.Z.Z.Z[500] (180 bytes)
Feb 17 12:14:48	charon	93047	15[ENC] <394> parsed ID_PROT request 0 [ SA V V V V V ]
Feb 17 12:14:48	charon	93047	15[CFG] <394> looking for an IKEv1 config for Z.Z.Z.Z...X.X.X.X
Feb 17 12:14:48	charon	93047	15[IKE] <394> no IKE config found for Z.Z.Z.Z...X.X.X.X, sending NO_PROPOSAL_CHOSEN
Feb 17 12:14:48	charon	93047	15[ENC] <394> generating INFORMATIONAL_V1 request 4171928100 [ N(NO_PROP) ]
Feb 17 12:14:48	charon	93047	15[NET] <394> sending packet: from Z.Z.Z.Z[500] to X.X.X.X[500] (40 bytes)
Feb 17 12:14:48	charon	93047	15[IKE] <394> IKE_SA (unnamed)[394] state change: CREATED => DESTROYING
Feb 17 12:14:49	charon	93047	15[NET] <395> received packet: from Y.Y.Y.Y[500] to Z.Z.Z.Z[500] (180 bytes)
Feb 17 12:14:49	charon	93047	15[ENC] <395> parsed ID_PROT request 0 [ SA V V V V V ]
Feb 17 12:14:49	charon	93047	15[CFG] <395> looking for an IKEv1 config for Z.Z.Z.Z...Y.Y.Y.Y
Feb 17 12:14:49	charon	93047	15[IKE] <395> no IKE config found for Z.Z.Z.Z...Y.Y.Y.Y, sending NO_PROPOSAL_CHOSEN
Feb 17 12:14:49	charon	93047	15[ENC] <395> generating INFORMATIONAL_V1 request 3792136085 [ N(NO_PROP) ]
Feb 17 12:14:49	charon	93047	15[NET] <395> sending packet: from Z.Z.Z.Z[500] to Y.Y.Y.Y[500] (40 bytes)
Feb 17 12:14:49	charon	93047	15[IKE] <395> IKE_SA (unnamed)[395] state change: CREATED => DESTROYING
Feb 17 12:14:52	charon	93047	15[IKE] <con300000|380> sending retransmit 4 of request message ID 0, seq 1
Feb 17 12:14:52	charon	93047	15[NET] <con300000|380> sending packet: from V.V.V.V[500] to C.C.C.C[500] (180 bytes)
Feb 17 12:14:57	charon	93047	15[NET] <396> received packet: from Y.Y.Y.Y[500] to Z.Z.Z.Z[500] (180 bytes)
Feb 17 12:14:57	charon	93047	15[ENC] <396> parsed ID_PROT request 0 [ SA V V V V V ]
Feb 17 12:14:57	charon	93047	15[CFG] <396> looking for an IKEv1 config for Z.Z.Z.Z...Y.Y.Y.Y
Feb 17 12:14:57	charon	93047	15[IKE] <396> no IKE config found for Z.Z.Z.Z...Y.Y.Y.Y, sending NO_PROPOSAL_CHOSEN
Feb 17 12:14:57	charon	93047	15[ENC] <396> generating INFORMATIONAL_V1 request 2015748929 [ N(NO_PROP) ]
Feb 17 12:14:57	charon	93047	15[NET] <396> sending packet: from Z.Z.Z.Z[500] to Y.Y.Y.Y[500] (40 bytes)
Feb 17 12:14:57	charon	93047	15[IKE] <396> IKE_SA (unnamed)[396] state change: CREATED => DESTROYING

Just to add, the tunnels all were fine and working before the upgrade, and they were again working just fine when I reverted the snapshot, so there's definitely a breaking change in the 2.5.0 release.

Any idea why 2.5.0 would break these IPSec tunnels?

Thanks!

jimp

"no IKE config found" means it isn't matching your Phase 1 info. Most likely due to your Phase 1 identifier configuration.

On 2.4.x there were some problems with identifiers not using the correct types, which are fixed on 2.5.0. Notably, Key ID is affected for example. On 2.4.x you could incorrectly choose Key ID and then strongSwan would have treated it as whatever type it wanted (but not a Key ID). So check and correct your Phase 1 identifiers to make sure the chosen type matches what the contents are. For example if it's an IP address, hostname, or FQDN, Key ID would be the wrong choice.

612brokeaf

I can confirm the exact same issue, no single IPsec tunnel came up after an upgrade from 2.4.5 to 2.5.0, they go to pfSense (2.4.5) and Cisco. It all works fine on 2.4.5. I also suspected peer IDs, but this is pretty simple, they are all:

My id: my ip address
Peer id: peer ip address

I tried manually setting the values to IPs; no joy. What do you suggest to configure then? Please note that I use DNS resolution for all peers, as in the remote g/w is specified as hostname, because some are on dynamic IPs. The resulting Strongswan config looks pretty much the same at first glance, minus formatting, between 2.4.5 and 2.5.0.

Not that I'm a Cisco fan (not at all), but given the maturity of their implementation, I would say that if it's worked with Cisco for ten plus years the way it was configured, and now it stopped, then is Cisco also misusing key types?

jimp

The IP address types are fine, mostly Key ID was the problem. If you aren't using that, then it probably isn't identifiers.

We'll need a lot more information than "it didn't work" to diagnose it, though. Starting with logs.

612brokeaf

@jimp thank you - yes, obviously, logs. Problem is that all I am seeing is:

no IKE config found for Z.Z.Z.Z...A.A.A.A, sending NO_PROPOSAL_CHOSEN

...which IIRC is not the same as prop not chosen due to transforms/algos not being accepted; it doesn't even get to that point.

This is IKEv2 by the way.

jimp

Compare the contents of /var/etc/ipsec/swanctl.conf and the output of swanctl --list-conns and see if the contents line up.

That log must mean that somehow it's not matching the P1. Without more details from the logs it's impossible to say why.

612brokeaf

@jimp ah, now we're getting somewhere...

swanctl --list_conns gives me zero output (empty), as if the config was not loaded properly. On 2.4.5 it does.

There is also a long pause when trying to look up SAs/SPDs under status->IPSec.

jimp

Do you get any output from swanctl --load-all --debug ?

612brokeaf

@jimp It doesn't like the debug option alone it seems, but otherwise:

swanctl --load-all --debug 5
created thread 01 [80079e500]
started worker thread 01
no events, waiting
created thread 03 [80079ef00]
started worker thread 03
  watching 7 for reading
watcher going to poll() 2 fds
no files found matching '/usr/local/etc/swanctl/conf.d/*.conf'
created thread 04 [80079f400]
started worker thread 04
created thread 02 [80079ea00]
started worker thread 02
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
no authorities found, 0 unloaded
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
no pools found, 0 unloaded
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
no connections found, 0 unloaded
watcher going to poll() 2 fds
terminated worker thread 01
terminated worker thread 03

However this is pretty much the same as what I get from the working 2.4.5.

612brokeaf

OK - in 2.4.5, in /usr/local/etc/strongswan.conf I have:

starter {
	load_warning = no
	config_file = /var/etc/ipsec/ipsec.conf
}

While on 2.5.0 I have:

starter {
	load_warning = no
}

Would this be it? Or have things been moved around?

612brokeaf

Yeah I think this isn't getting me anywhere. swanctl --load-conns shows "no connections found" on both 2.4.5. and 2.5.0, so this is probably not where I should be looking.

jimp

2.5.0 does not use starter, it uses swanctl/VICI. That isn't relevant.

What is in your /var/etc/ipsec/swanctl.conf file? You can obscure private info like iP addresses, identifiers, and keys but leave all the structure and names in place.

612brokeaf

@travisn Sorry to hijack your thread here. But - I think I got something that could also be useful for you. There may be some errors in the config that cause strongswan to fail to load it or parts of it.

Try swanctl --list-conns and see if you get a list. If not, try swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1 and see if you get any errors.

A Former User

@612brokeaf Hey no worries! I'm just glad I wasn't the only one. I'm going to run through this thread tonight to see if I can get closer. I'm just setting up another 2.4.5-p1 instance in another lab that doesn't have anything else running to try and cut down on the noise.

A Former User

Well this is interesting. After installing 2.4.5-p1 in a quieter environment so I could have cleaner logs, I added a single IPSec VPN the same way I setup my other ones. I upgraded to 2.5.0 and the VPN came back up and I was able to access the remote site just fine. I'm going to try re-upgrading my original site to see if I can reproduce it again or if it was just some strange fluke.

A Former User

Alright, this must have been a strange fluke. I upgraded the original firewall I upgraded this morning and it completed successfully. No idea what happened

bbrendon

@jimp said in pfSense 2.5.0 broke all IPSec VPNs:

On 2.4.x there were some problems with identifiers not using the correct types,

Using IPs instead of distinguished name fixed it for me. Not sure what I'll do when IPs change but I'm up for now.

A Former User

Alright, I hit another one. I upgraded 3 successfully, the 4th (the one where I wasn't monitoring the console, of course), decided to have the same problem.

Tried what @612brokeaf suggested by running swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1 and didn't get any errors, but my tunnels came right up. Of course it did not survive a reboot but again, no errors. I don't have any pools, so I guess that's a good thing?

At this point, I'm not sure what I should be looking for. There's no errors, no warnings, no light at the end of the tunnel. I'm not going to wipe this one so if anyone has any further suggestions, I'm open. Otherwise, it seems to be a shot in the dark whether or not IPSec VPNs survive the upgrade.

vergilis

Hi. Just to review. Distinguished names do not work in 2.5 and when changing it to IP address authentication everything works?

m0nji

i also have problems with my ipsec tunnels after upgrading to 2.5.
i have 5 tunnels which all are not working anymore. an output of

swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1

gave me no clue

the only thing which i see in the log is:

which probably means, the key does not match in P1. but they are definitly correct! i also tried to change the keys on both sites with no success.

right now, the only workaround for me was, to recreate the tunnels (P1+P2) with the EXACT SAME settings as before. with that, the first tunnel came up right away. i am also using distinquished name for most of the tunnels.

i wait now for maybe some more hints or instructions to test, before i recreate all the other tunnels.

btw: is the "status --> ipsec" page for you all that slow? it takes around 10sec before it shows me the status.