SG-3100 Site to Site VPN dies under heavy load
-
I've got two sites with a site to site VPN configured. I've done this plenty of times and the basic setup is working fine.
One site is an old Dell Server (overkill) and the other site is a Netgate SG-3100. Both sites have dual WAN.
I'm doing backups from one site to the other and performance is ok, but the backups keep failing and then the far site (SG-3100) shows that all OpenVPN Servers and Clients have died. The service will not restart until I reboot the SG-3100.
Originally, this was a simple site to site VPN over just the WAN 1 connection (100 mbps fiber). When it originally failed, I thought I would set up multi-WAN site to site using OSPF. This was working fine. Each site ran two servers and two clients. (WAN 1 to WAN 1, WAN 2 to WAN 2, WAN 1 to WAN 2, WAN 2 to WAN 1). With these 4 connections, I could lose any WAN at either site and the VPN should stay up. Like before, during "normal" operation, this is working perfectly, but if I start putting a heavy load across the VPN (replicating VM's from one site to the other), all VPN Servers and Clients will fail on the SG-3100.
The basic setup of all 4 VPN connections is as follows.
-Peer to Peer (Shared Key)
-UDP on IPv4 only
-tun
-Encryption Algorithm AES-128-CDC
-Enable Negotiable Crypto
-AES-128-GCM
-Auth Digest SHA256 (256 bit)
-Hardware Crypto (BSD cryptodev engine)
-Compression - Omit Preference (Use OpenVPN Default)Should I be disabling hardware crypto or disabling compression or trying different crypto or something?
I will paste the last 100 lines of my OpenVPN log below:
Feb 25 08:44:52 openvpn 31338 /sbin/ifconfig ovpnc5 10.2.194.2 10.2.194.1 mtu 1500 netmask 255.255.255.255 up
Feb 25 08:44:52 openvpn 31338 /usr/local/sbin/ovpn-linkup ovpnc5 1500 1572 10.2.194.2 10.2.194.1 init
Feb 25 08:44:52 openvpn 31338 TCP/UDP: Preserving recently used remote address: [AF_INET]50.240.111.97:1199
Feb 25 08:44:52 openvpn 31338 UDPv4 link local (bound): [AF_INET]23.31.175.81:0
Feb 25 08:44:52 openvpn 31338 UDPv4 link remote: [AF_INET]50.240.111.97:1199
Feb 25 08:44:52 openvpn 31338 cipher_ctx_update: EVP_CipherUpdate() failed
Feb 25 08:44:52 openvpn 31338 Exiting due to fatal error
Feb 25 08:44:52 openvpn 31338 /usr/local/sbin/ovpn-linkdown ovpnc5 1500 1572 10.2.194.2 10.2.194.1 init
Feb 25 08:44:53 openvpn 72869 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Feb 25 08:44:53 openvpn 72869 OpenVPN 2.4.6 armv6-portbld-freebsd11.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Sep 5 2018
Feb 25 08:44:53 openvpn 72869 library versions: OpenSSL 1.0.2o-freebsd 27 Mar 2018, LZO 2.10
Feb 25 08:44:53 openvpn 74422 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 25 08:44:53 openvpn 74422 Initializing OpenSSL support for engine 'cryptodev'
Feb 25 08:44:53 openvpn 74422 TUN/TAP device ovpns7 exists previously, keep at program end
Feb 25 08:44:53 openvpn 74422 TUN/TAP device /dev/tun7 opened
Feb 25 08:44:53 openvpn 74422 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Feb 25 08:44:53 openvpn 74422 /sbin/ifconfig ovpns7 10.2.197.1 10.2.197.2 mtu 1500 netmask 255.255.255.255 up
Feb 25 08:44:53 openvpn 74422 /usr/local/sbin/ovpn-linkup ovpns7 1500 1572 10.2.197.1 10.2.197.2 init
Feb 25 08:44:53 openvpn 74422 UDPv4 link local (bound): [AF_INET]66.83.239.114:1197
Feb 25 08:44:53 openvpn 74422 UDPv4 link remote: [AF_UNSPEC]
Feb 25 08:44:53 openvpn 74422 cipher_ctx_update: EVP_CipherUpdate() failed
Feb 25 08:44:53 openvpn 74422 Exiting due to fatal error
Feb 25 08:44:53 openvpn 74422 /usr/local/sbin/ovpn-linkdown ovpns7 1500 1572 10.2.197.1 10.2.197.2 init
Feb 25 08:44:54 openvpn 17104 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Feb 25 08:44:54 openvpn 17104 OpenVPN 2.4.6 armv6-portbld-freebsd11.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Sep 5 2018
Feb 25 08:44:54 openvpn 17104 library versions: OpenSSL 1.0.2o-freebsd 27 Mar 2018, LZO 2.10
Feb 25 08:44:54 openvpn 18255 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 25 08:44:54 openvpn 18255 Initializing OpenSSL support for engine 'cryptodev'
Feb 25 08:44:54 openvpn 18255 TUN/TAP device ovpns6 exists previously, keep at program end
Feb 25 08:44:54 openvpn 18255 TUN/TAP device /dev/tun6 opened
Feb 25 08:44:54 openvpn 18255 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Feb 25 08:44:54 openvpn 18255 /sbin/ifconfig ovpns6 10.2.198.1 10.2.198.2 mtu 1500 netmask 255.255.255.255 up
Feb 25 08:44:54 openvpn 18255 /usr/local/sbin/ovpn-linkup ovpns6 1500 1572 10.2.198.1 10.2.198.2 init
Feb 25 08:44:54 openvpn 18255 UDPv4 link local (bound): [AF_INET]23.31.175.81:1200
Feb 25 08:44:54 openvpn 18255 UDPv4 link remote: [AF_UNSPEC]
Feb 25 08:44:54 openvpn 18255 cipher_ctx_update: EVP_CipherUpdate() failed
Feb 25 08:44:54 openvpn 18255 Exiting due to fatal error
Feb 25 08:44:54 openvpn 18255 /usr/local/sbin/ovpn-linkdown ovpns6 1500 1572 10.2.198.1 10.2.198.2 init
Feb 25 08:45:01 openvpn 12434 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Feb 25 08:45:01 openvpn 12434 OpenVPN 2.4.6 armv6-portbld-freebsd11.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Sep 5 2018
Feb 25 08:45:01 openvpn 12434 library versions: OpenSSL 1.0.2o-freebsd 27 Mar 2018, LZO 2.10
Feb 25 08:45:01 openvpn 12588 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 25 08:45:01 openvpn 12588 Initializing OpenSSL support for engine 'cryptodev'
Feb 25 08:45:01 openvpn 12588 TUN/TAP device ovpnc4 exists previously, keep at program end
Feb 25 08:45:01 openvpn 12588 TUN/TAP device /dev/tun4 opened
Feb 25 08:45:01 openvpn 12588 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Feb 25 08:45:01 openvpn 12588 /sbin/ifconfig ovpnc4 10.2.193.2 10.2.193.1 mtu 1500 netmask 255.255.255.255 up
Feb 25 08:45:01 openvpn 12588 /usr/local/sbin/ovpn-linkup ovpnc4 1500 1572 10.2.193.2 10.2.193.1 init
Feb 25 08:45:01 openvpn 12588 TCP/UDP: Preserving recently used remote address: [AF_INET]70.43.27.130:1198
Feb 25 08:45:01 openvpn 12588 UDPv4 link local (bound): [AF_INET]66.83.239.114:0
Feb 25 08:45:01 openvpn 12588 UDPv4 link remote: [AF_INET]70.43.27.130:1198
Feb 25 08:45:01 openvpn 12588 cipher_ctx_update: EVP_CipherUpdate() failed
Feb 25 08:45:01 openvpn 12588 Exiting due to fatal error
Feb 25 08:45:01 openvpn 12588 /usr/local/sbin/ovpn-linkdown ovpnc4 1500 1572 10.2.193.2 10.2.193.1 init
Feb 25 08:45:01 openvpn 37299 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Feb 25 08:45:01 openvpn 37299 OpenVPN 2.4.6 armv6-portbld-freebsd11.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Sep 5 2018
Feb 25 08:45:01 openvpn 37299 library versions: OpenSSL 1.0.2o-freebsd 27 Mar 2018, LZO 2.10
Feb 25 08:45:01 openvpn 37574 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 25 08:45:01 openvpn 37574 Initializing OpenSSL support for engine 'cryptodev'
Feb 25 08:45:01 openvpn 37574 TUN/TAP device ovpnc5 exists previously, keep at program end
Feb 25 08:45:01 openvpn 37574 TUN/TAP device /dev/tun5 opened
Feb 25 08:45:01 openvpn 37574 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Feb 25 08:45:01 openvpn 37574 /sbin/ifconfig ovpnc5 10.2.194.2 10.2.194.1 mtu 1500 netmask 255.255.255.255 up
Feb 25 08:45:01 openvpn 37574 /usr/local/sbin/ovpn-linkup ovpnc5 1500 1572 10.2.194.2 10.2.194.1 init
Feb 25 08:45:01 openvpn 37574 TCP/UDP: Preserving recently used remote address: [AF_INET]50.240.111.97:1199
Feb 25 08:45:01 openvpn 37574 UDPv4 link local (bound): [AF_INET]23.31.175.81:0
Feb 25 08:45:01 openvpn 37574 UDPv4 link remote: [AF_INET]50.240.111.97:1199
Feb 25 08:45:01 openvpn 37574 cipher_ctx_update: EVP_CipherUpdate() failed
Feb 25 08:45:01 openvpn 37574 Exiting due to fatal error
Feb 25 08:45:01 openvpn 37574 /usr/local/sbin/ovpn-linkdown ovpnc5 1500 1572 10.2.194.2 10.2.194.1 init
Feb 25 08:45:02 openvpn 82254 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Feb 25 08:45:02 openvpn 82254 OpenVPN 2.4.6 armv6-portbld-freebsd11.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Sep 5 2018
Feb 25 08:45:02 openvpn 82254 library versions: OpenSSL 1.0.2o-freebsd 27 Mar 2018, LZO 2.10
Feb 25 08:45:02 openvpn 85884 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 25 08:45:02 openvpn 85884 Initializing OpenSSL support for engine 'cryptodev'
Feb 25 08:45:02 openvpn 85884 TUN/TAP device ovpns6 exists previously, keep at program end
Feb 25 08:45:02 openvpn 85884 TUN/TAP device /dev/tun6 opened
Feb 25 08:45:02 openvpn 85884 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Feb 25 08:45:02 openvpn 85884 /sbin/ifconfig ovpns6 10.2.198.1 10.2.198.2 mtu 1500 netmask 255.255.255.255 up
Feb 25 08:45:02 openvpn 85884 /usr/local/sbin/ovpn-linkup ovpns6 1500 1572 10.2.198.1 10.2.198.2 init
Feb 25 08:45:02 openvpn 85884 UDPv4 link local (bound): [AF_INET]23.31.175.81:1200
Feb 25 08:45:02 openvpn 85884 UDPv4 link remote: [AF_UNSPEC]
Feb 25 08:45:02 openvpn 85884 cipher_ctx_update: EVP_CipherUpdate() failed
Feb 25 08:45:02 openvpn 85884 Exiting due to fatal error
Feb 25 08:45:02 openvpn 85884 /usr/local/sbin/ovpn-linkdown ovpns6 1500 1572 10.2.198.1 10.2.198.2 init
Feb 25 08:45:03 openvpn 34617 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Feb 25 08:45:03 openvpn 34617 OpenVPN 2.4.6 armv6-portbld-freebsd11.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Sep 5 2018
Feb 25 08:45:03 openvpn 34617 library versions: OpenSSL 1.0.2o-freebsd 27 Mar 2018, LZO 2.10
Feb 25 08:45:03 openvpn 34710 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 25 08:45:03 openvpn 34710 Initializing OpenSSL support for engine 'cryptodev'
Feb 25 08:45:03 openvpn 34710 TUN/TAP device ovpns7 exists previously, keep at program end
Feb 25 08:45:03 openvpn 34710 TUN/TAP device /dev/tun7 opened
Feb 25 08:45:03 openvpn 34710 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Feb 25 08:45:03 openvpn 34710 /sbin/ifconfig ovpns7 10.2.197.1 10.2.197.2 mtu 1500 netmask 255.255.255.255 up
Feb 25 08:45:03 openvpn 34710 /usr/local/sbin/ovpn-linkup ovpns7 1500 1572 10.2.197.1 10.2.197.2 init
Feb 25 08:45:04 openvpn 34710 UDPv4 link local (bound): [AF_INET]66.83.239.114:1197
Feb 25 08:45:04 openvpn 34710 UDPv4 link remote: [AF_UNSPEC]
Feb 25 08:45:04 openvpn 34710 cipher_ctx_update: EVP_CipherUpdate() failed
Feb 25 08:45:04 openvpn 34710 Exiting due to fatal error
Feb 25 08:45:04 openvpn 34710 /usr/local/sbin/ovpn-linkdown ovpns7 1500 1572 10.2.197.1 10.2.197.2 init -
Hmmm I have some SG-3100 with 100 Mbit/s fiber pipes but never managed to kill the OpenVPN process, also under heavy load.
I use PKI, AES-256-GCM (fixed with NCP off), SHA256, TLS Encryption and Authentication, No Hardware Crypto Acceleration, lz4-v2 compression.
OpenVPN peak is around 85 Mbit/s and stable since months.-Rico
-
Same problem with our setup, we also use the hardware crypto - so maybe that's it?
-
Here are the logs from when it crashed for us:
ntpd.log:Mar 4 05:01:45 lumospfsense ntpd[23171]: Deleting interface #11 ovpnc1, fe80::208:a2ff:fe0d:20fa%13#123, interface stats: received=0, sent=0, dropped=0, active_time=9598404 secs
ntpd.log:Mar 4 05:01:45 lumospfsense ntpd[23171]: Deleting interface #12 ovpnc1, 192.168.228.6#123, interface stats: received=0, sent=0, dropped=0, active_time=9598404 secs
openvpn.log:Mar 4 05:01:43 lumospfsense openvpn[14004]: cipher_ctx_update: EVP_CipherUpdate() failed
openvpn.log:Mar 4 05:01:43 lumospfsense openvpn[14004]: Exiting due to fatal error
openvpn.log:Mar 4 05:01:43 lumospfsense openvpn[14004]: /usr/local/sbin/ovpn-linkdown ovpnc1 1500 1561 192.168.228.6 192.168.228.5 init
system.log:Mar 4 05:01:43 lumospfsense kernel: ovpnc1: link state changed to DOWN
system.log:Mar 4 05:01:43 lumospfsense check_reload_status: Reloading filter -
I ended up just using IPSec site to site tunnels, instead of OpenVPN. At some point, I'll go test it again and try turning off hardware crypto to see if that resolves it.
My biggest reason for using OpenVPN is that I wanted to tie it into OSPF so I would have resiliency in a multi-WAN connection between sites with two WAN connections at each site.
I have since been able to get that functionality working over two IPSec connections PLUS two OpenVPN connection (at another two sites) all tied together with OSPF.
-
By the way, I have since gone back to OpenVPN tunnels and they have been rock solid now that I’ve NOT enabled hardware crypto.
-
Thanks for posting this. I was having the same issue with hardware crypto enabled on my SG-3100. Disabling seems to have resolved the issue though it certainly hasn't helped my CPU load.