IPSec tunnels are not coming up since last update

luphi

All,

I'm running 2.5 since several month without issues, until I updated to 2.5.0.a.20200102.0720
(Last update before was around 12/15/2019)

After the reboot IPSec tunnels were not coming up anymore.

/var/log/ipsec.log is showing the same 4 lines every couple of seconds:

Jan  2 16:19:18 eagle1 charon[17945]: 11[CFG] vici client 127 connected
Jan  2 16:19:18 eagle1 charon[17945]: 11[CFG] vici client 127 registered for: list-sa
Jan  2 16:19:18 eagle1 charon[17945]: 12[CFG] vici client 127 requests: list-sas
Jan  2 16:19:18 eagle1 charon[17945]: 11[CFG] vici client 127 disconnected

Cheers,
luphi

jimp

Those log entries are just the GUI displaying the status (widget or Status > IPsec).

You'll need to post a lot more information about your setup for there to be any hope of tracking down what your problem might be.

EDIT: For reference, the VMs I have updated to a current snapshot connect just fine, so it doesn't appear to be a general issue.

luphi

Thank you jimp,

the tunnels were running smooth and I'm not aware of any config change but the update.
Which information do you need for further troubleshooting?

Here is the anonymized config:
ipsec.xml

Cheers,
luphi

jimp

That config wasn't as anonymized as it should have been, your PSKs are still visible, so you should change those ASAP.

Nothing stood out there, what we'd need to see are the full ipsec.logs from startup and after, plus at least /var/etc/ipsec/swanctl.conf though you can omit the secrets {} block.

luphi

thank you for the hint.

ipsec.log.txt
swanctl.conf.txt

jimp

Nothing stands out there. They are both set to start with trap policies so they won't connect until traffic tries to pass. Have you tried passing traffic over the tunnel?

luphi

yes, I tried. That was showing me, the tunnels are down.
Is there a way to roll back to a version of let's say 2 weeks ago?

luphi

Not sure if that is important but I'm running a cluster.
All logs are of course from the primary, active node.

jimp

No, there isn't a way to roll back. And HA shouldn't matter unless your secondary is receiving the traffic and not the primary. Check the CARP status to be sure.

Are you certain your traffic that should enter the tunnel is doing so? Maybe it's being misrouted by something like a rule on LAN with a gateway set?

Do the tunnels connect if you click the button to connect them on the status page?

Does it show down on the other end?

luphi

all interface are in master state.
To be on the safe side, I took down the 2nd node.
Routing is okay.
Internet access is working fine
When I click the "connect" button, a tcpdump shows nothing on the Internet facing interface on port 500.

jimp

Do you have a default route in the routing table?

luphi

yes, I have.

jimp

Any state table entries for the remote peer address on any port? It could be using 4500 and not 500.

Might also help to see the output of:

swanctl --list-conns
swanctl --list-sas
swanctl --list-pols
setkey -D
setkey -DP
route -n get x.x.x.x (where x.x.x.x is the remote peer address)

luphi

# swanctl --list-conns
bypass: IKEv1/2, no reauthentication, rekeying every 14400s
  local:  %any
  remote: 127.0.0.1
  local unspecified authentication:
  remote unspecified authentication:
  bypass: PASS, no rekeying
    local:  172.23.1.0/24|/0
    remote: 172.23.1.0/24|/0
# swanctl --list-sas
# swanctl --list-pols
bypass/bypass, PASS
  local:  172.23.1.0/24|/0
  remote: 172.23.1.0/24|/0
# setkey -D
No SAD entries.
# setkey -DP
172.23.1.0/24[any] 172.23.1.0/24[any] any
	in none
	created: Jan  2 15:59:26 2020  lastused: Jan  3 10:01:52 2020
	lifetime: 9223372036854775807(s) validtime: 0(s)
	spid=2 seq=1 pid=24554 scope=global 
	refcnt=2
172.23.1.0/24[any] 172.23.1.0/24[any] any
	out none
	created: Jan  2 15:59:26 2020  lastused: Jan  3 10:01:52 2020
	lifetime: 9223372036854775807(s) validtime: 0(s)
	spid=1 seq=0 pid=24554 scope=global 
	refcnt=3

routing is ok, pointing to the Internet router

jimp

Interesting. The connections are not loaded at all, neither are the policies. But the LAN bypass is.

What does this show:
swanctl --load-conns --file /var/etc/ipsec/swanctl.conf

luphi

We are getting closer...

 # swanctl --load-conns --file /var/etc/ipsec/swanctl.conf
loaded connection 'bypass'
loading connection 'con1000' failed: invalid value for: proposals, config discarded
loading connection 'con2000' failed: invalid value for: proposals, config discarded
loaded 1 of 3 connections, 2 failed to load, 0 unloaded

luphi

When I check the GUI (VPN -> IPsec -> Tunnels -> Edit Phase 1),
the "Phase1 proposal (Encryption Algorithm)" box shown twice:

jimp

That's a known issue that I just pushed a fix for: https://redmine.pfsense.org/issues/10151

luphi

thought, it might be related to my issue

jimp

Doesn't appear to be. Looks like it's something with the way the P1 proposal is being formed with certain combinations of options. I finally found one tunnel on one of my VMs that is doing the same thing.

Looks like what actually broke it is https://redmine.pfsense.org/issues/9726

Not working: proposals = aes128gcm128-curve448
Working: proposals = aes128gcm128-aesxcbc-curve448

I reverted the change from the PR on there and it came up and started working again.