Cannot run two OpenVPN clients reliable

matsan

Hello,
We have the following setup connecting HQ and a Remote Office using OSPF for fail-over:

The Remote Office SG-3100 has two OpenVPN Clients defined connecting to two servers on HQ SG-3100 (on separate ports 1195 and 1196). The setup can work for weeks but the starts failing with errors related to ifconfig:

Jun 7 06:55:07	openvpn	23736	Exiting due to fatal error
Jun 7 06:55:07	openvpn	23736	FreeBSD ifconfig failed: external program exited with error status: 1
Jun 7 06:55:07	openvpn	23736	/sbin/ifconfig ovpnc3 10.3.102.2/24 mtu 1500 up
Jun 7 06:55:07	openvpn	23736	TUN/TAP device /dev/tap3 opened
Jun 7 06:55:07	openvpn	23736	TUN/TAP device ovpnc3 exists previously, keep at program end
Jun 7 06:55:07	openvpn	23736	[site2site.xxx.yyy.net] Peer Connection Initiated with [AF_INET]<IP_LTE_ROUTER>:1196
Jun 7 06:55:07	openvpn	23736	UDPv4 link remote: [AF_INET]<IP_LTE_ROUTER>:1196
Jun 7 06:55:07	openvpn	23736	UDPv4 link local (bound): [AF_INET]<IP_COMHEM2>:0
Jun 7 06:55:07	openvpn	23736	TCP/UDP: Preserving recently used remote address: [AF_INET]<IP_LTE_ROUTER>:1196
Jun 7 06:55:07	openvpn	23736	WARNING: experimental option --capath /var/etc/openvpn/client3/ca
Jun 7 06:55:07	openvpn	23736	Initializing OpenSSL support for engine 'devcrypto'
Jun 7 06:55:07	openvpn	23736	NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 7 06:55:07	openvpn	23456	DCO version: FreeBSD 14.0-CURRENT #0 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:25:32 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/armv7/mMnBJzEV/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/sources/FreeBS
Jun 7 06:55:07	openvpn	23456	library versions: OpenSSL 1.1.1t-freebsd 7 Feb 2023, LZO 2.10
Jun 7 06:55:07	openvpn	23456	OpenVPN 2.6.2 armv7-portbld-freebsd14.0 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD] [DCO]

Log was captured this morning after being up for three weeks. Getting everything to work again required removing the failed OpenVPN Client configuration, rebooting the SG-3100 and recreating the OpenVPN Client connection.

The OpenVPN Client connections are configured with Peer-to-Peer (SSL/TLS), TAP, TLS Authentication, same Peer Certificate but different Client Certificates. No tunnel network.

OSPF picks up both routes when it works and shows in OSPF Neighbors and OSPF Routes.

Remote Office is running 23.05 with FRR 1.2_3
HQ is running 23.01 with FRR 1.2_3
(I cannot get HQ unit to update to 23.05 even after reboot and let it sit on the Update page for hours).
However the HQ SG-3100 have both OpenVPN Servers up and running.

This setup worked reliably until April when we started seeing this issue - https://forum.netgate.com/topic/179614/50-days-after-upgrading-openvpn-now-starts-failing?_=1686115195213

Any help greatly appreciated.
/Mattias