pfSense 2.5.0 broke all IPSec VPNs

612brokeaf

@jimp It doesn't like the debug option alone it seems, but otherwise:

swanctl --load-all --debug 5
created thread 01 [80079e500]
started worker thread 01
no events, waiting
created thread 03 [80079ef00]
started worker thread 03
  watching 7 for reading
watcher going to poll() 2 fds
no files found matching '/usr/local/etc/swanctl/conf.d/*.conf'
created thread 04 [80079f400]
started worker thread 04
created thread 02 [80079ea00]
started worker thread 02
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
no authorities found, 0 unloaded
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
watcher going to poll() 2 fds
no pools found, 0 unloaded
watched FD 7 ready to read
watcher going to poll() 1 fds
watcher got notification, rebuilding
  watching 7 for reading
no connections found, 0 unloaded
watcher going to poll() 2 fds
terminated worker thread 01
terminated worker thread 03

However this is pretty much the same as what I get from the working 2.4.5.

612brokeaf

OK - in 2.4.5, in /usr/local/etc/strongswan.conf I have:

starter {
	load_warning = no
	config_file = /var/etc/ipsec/ipsec.conf
}

While on 2.5.0 I have:

starter {
	load_warning = no
}

Would this be it? Or have things been moved around?

612brokeaf

Yeah I think this isn't getting me anywhere. swanctl --load-conns shows "no connections found" on both 2.4.5. and 2.5.0, so this is probably not where I should be looking.

jimp

2.5.0 does not use starter, it uses swanctl/VICI. That isn't relevant.

What is in your /var/etc/ipsec/swanctl.conf file? You can obscure private info like iP addresses, identifiers, and keys but leave all the structure and names in place.

612brokeaf

@travisn Sorry to hijack your thread here. But - I think I got something that could also be useful for you. There may be some errors in the config that cause strongswan to fail to load it or parts of it.

Try swanctl --list-conns and see if you get a list. If not, try swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1 and see if you get any errors.

A Former User

@612brokeaf Hey no worries! I'm just glad I wasn't the only one. I'm going to run through this thread tonight to see if I can get closer. I'm just setting up another 2.4.5-p1 instance in another lab that doesn't have anything else running to try and cut down on the noise.

A Former User

Well this is interesting. After installing 2.4.5-p1 in a quieter environment so I could have cleaner logs, I added a single IPSec VPN the same way I setup my other ones. I upgraded to 2.5.0 and the VPN came back up and I was able to access the remote site just fine. I'm going to try re-upgrading my original site to see if I can reproduce it again or if it was just some strange fluke.

A Former User

Alright, this must have been a strange fluke. I upgraded the original firewall I upgraded this morning and it completed successfully. No idea what happened

bbrendon

@jimp said in pfSense 2.5.0 broke all IPSec VPNs:

On 2.4.x there were some problems with identifiers not using the correct types,

Using IPs instead of distinguished name fixed it for me. Not sure what I'll do when IPs change but I'm up for now.

A Former User

Alright, I hit another one. I upgraded 3 successfully, the 4th (the one where I wasn't monitoring the console, of course), decided to have the same problem.

Tried what @612brokeaf suggested by running swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1 and didn't get any errors, but my tunnels came right up. Of course it did not survive a reboot but again, no errors. I don't have any pools, so I guess that's a good thing?

At this point, I'm not sure what I should be looking for. There's no errors, no warnings, no light at the end of the tunnel. I'm not going to wipe this one so if anyone has any further suggestions, I'm open. Otherwise, it seems to be a shot in the dark whether or not IPSec VPNs survive the upgrade.

vergilis

Hi. Just to review. Distinguished names do not work in 2.5 and when changing it to IP address authentication everything works?

m0nji

i also have problems with my ipsec tunnels after upgrading to 2.5.
i have 5 tunnels which all are not working anymore. an output of

swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1

gave me no clue

the only thing which i see in the log is:

which probably means, the key does not match in P1. but they are definitly correct! i also tried to change the keys on both sites with no success.

right now, the only workaround for me was, to recreate the tunnels (P1+P2) with the EXACT SAME settings as before. with that, the first tunnel came up right away. i am also using distinquished name for most of the tunnels.

i wait now for maybe some more hints or instructions to test, before i recreate all the other tunnels.

btw: is the "status --> ipsec" page for you all that slow? it takes around 10sec before it shows me the status.

thiagocrepaldi

Maybe that helps, but my IPsec tunnel broke after upgrading to pfsense 2.5.0 because the "Peer identifier" was set to Any in both sides. By changing it to "IP address" 0.0.0.0, things got working again.

My IPSec uses dynamic ip in both ends, so i cant use real IPs here

612brokeaf

@m0nji said in [pfSense 2.5.0 broke all IPSec VPNs]

btw: is the "status --> ipsec" page for you all that slow? it takes around 10sec before it shows me the status.

Yes, same here.

A Former User

@thiagocrepaldi I have it set to "IP Address". It's like that on all of my firewalls, so it's strange that out of 5 upgrades, 2 failed (1 done twice).

cliobrando

Same problem here, but cannot make it work, I have deleted all the config and created a single phase1 using the same configuration that worked in 2.4.5-p1:

EDIT: Nevermind, erased everything, and started from zero fixed the problem.

Thanks!

jimp

Seems there are several issues here all getting confused.

Identifier issues with "Distinguished Name" (Which is a bug -- see https://redmine.pfsense.org/issues/11442 -- for a quick workaround, apply the patch there or just set your IDs to KeyID in the meantime)
Identifier issues from incorrect use of Key ID in the past (which fell back to automatic guessing at the type, so may not now match a remote not set specifically to Key ID) -- To fix this, set the right ID type and value on both sides to match
Configuration issues where the configuration is failing to load (with errors)
Tunnels loading but not connecting
Other things that haven't yet been identified

Having one thread for all of this is a giant mess that's hard to follow. It's better for the moment if everyone makes their own thread here in the IPsec category and includes as much detail as possible.

If someone else does have a thread for the exact same root issue then you can combine those threads, but this one is far too generic to be useful.

For those of you who say re-creating the tunnel worked, be sure to grab the config.xml and compare before/after as well as /var/etc/ipsec/swanctl.conf -- something must be different if it suddenly started working, and if it's something done by the upgrade process then we can identify and fix it.

For troubleshooting, first apply patches to fix known issues which have already been resolved:

ead6515637a34ce6e170e2d2b0802e4fa1e63a00 #11435
57beb9ad8ca11703778fc483c7cba0f6770657ac #11435
10eb04259fd139c62e08df8de877b71fdd0eedc8 #11442
ded7970ba57a99767e08243103e55d8a58edfc35 #11486
afffe759c4fd19fe6b8311196f4b6d5e288ea4fb #11487
2fe5cc52bd881ed26723a81e0eed848fd505fba6 #11488

After that, edit/save/apply an IPsec tunnel, then stop and start (not restart) the IPsec daemon, or reboot instead.

If problems persist, do the following:

Edit/save a tunnel
Apply changes
Go to Status > Services and stop, then start the IPsec service (don't click restart)
Go to Status > IPsec on one end and attempt to initiate the tunnel if it doesn't come up automatically.

If it works, great. If not:

Run swanctl --list-conns to see what the IPsec daemon loaded for the connections
Run swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1 and see if it reports any problems
Get the config from /var/etc/ipsec/swanctl.conf
Get the most recent logs from both sides

With that in hand, check for an existing thread which matches the symptoms exactly. If one exists, post there. If there isn't one, create one.

Locking this so it doesn't keep growing and making things more confusing.