5100/22.05 - Degraded OpenVPN client performance

[Myster_fr]

Hello,

I proceeded yesterday evening with a 22.01 => 22.05 upgrade.
All in all, it went pretty well except for the squid issue a lot seem to encounter.
Quick fix :

cp /usr/local/etc/squid/mime.conf.sample  /usr/local/etc/squid/mime.conf

and restart squid.

What annoys me the most, is the significant drop of performance witnessed on my OpenVPN client.

I had a client configuration in place that I've been using for years.
Through this tunnel, I know be experience that I'm capped at about 300-350 Mbps/s (roughly 30 MB/s), while the SG-5100 used to show 10-15% CPU usage under 22.01.

Since the upgrade, I barely hit 50 Mb/s (so 5MB/s), so about 6 times less than before, while the CPU hovers ~45% usage.
I tried connecting from a different device just to confirm the issue does not lay on the server side, and could reach my usual 30 MB/s download speed.

I increased log level to determine if something odd was happening but could not find any suspicious messages.

Did anyone else encounter this kind of issue ?

Here are some information regarding the OpenVPN client configuration :

dev ovpnc3
disable-dco
verb 1
dev-type tun
dev-node /dev/tun3
writepid /var/run/openvpn_client3.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.xx.yy
engine rdrand
tls-client
lport 0
management /var/etc/openvpn/client3/sock unix
remote my.vpn.provider.server 1194 udp4
pull
auth-user-pass /var/etc/openvpn/client3/up
auth-retry nointeract
remote-cert-tls server
capath /var/etc/openvpn/client3/ca
tls-auth /var/etc/openvpn/client3/tls-auth 1
data-ciphers AES-256-GCM:AES-128-CBC:AES-128-GCM
data-ciphers-fallback AES-256-GCM
allow-compression asym
compress 
resolv-retry infinite

## Custom options
#ncp-disable  ## Had to disable this following the upgrade to 22.05 otherwise tunnel would not come up.
remote-cert-tls server
client
comp-lzo
auth-nocache
connection retry -1
route-delay
keepalive 3 10
route-nopull

Nothing specific in the logs :

Oct 10 16:48:17 sg-5100 openvpn[91751]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Oct 10 16:48:17 sg-5100 openvpn[91751]: Initializing OpenSSL support for engine 'rdrand'
Oct 10 16:48:17 sg-5100 openvpn[91751]: WARNING: experimental option --capath /var/etc/openvpn/client3/ca
Oct 10 16:48:22 sg-5100 openvpn[91751]: TCP/UDP: Preserving recently used remote address: [AF_INET] xx.xx.xx.xx:1194
Oct 10 16:48:22 sg-5100 openvpn[91751]: UDPv4 link local (bound): [AF_INET]192.168.xx.yy:0
Oct 10 16:48:22 sg-5100 openvpn[91751]: UDPv4 link remote: [AF_INET] xx.xx.xx.xx:1194
Oct 10 16:48:23 sg-5100 openvpn[91751]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1570'
Oct 10 16:48:23 sg-5100 openvpn[91751]: WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA256'
Oct 10 16:48:23 sg-5100 openvpn[91751]: [PrivateVPN] Peer Connection Initiated with [AF_INET]xx.xx.xx.xx:1194
Oct 10 16:48:24 sg-5100 openvpn[91751]: Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])
Oct 10 16:48:24 sg-5100 openvpn[91751]: Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Oct 10 16:48:24 sg-5100 openvpn[91751]: Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Oct 10 16:48:24 sg-5100 openvpn[91751]: Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Oct 10 16:48:24 sg-5100 openvpn[91751]: TUN/TAP device ovpnc3 exists previously, keep at program end
Oct 10 16:48:24 sg-5100 openvpn[91751]: TUN/TAP device /dev/tun3 opened
Oct 10 16:48:24 sg-5100 openvpn[91751]: /sbin/ifconfig ovpnc3 yyy.yyy.yyy.yyy yyy.yyy.yyy.yyy mtu 1500 netmask 255.255.255.192 up
Oct 10 16:48:24 sg-5100 openvpn[91751]: /usr/local/sbin/ovpn-linkup ovpnc3 1500 0 yyy.yyy.yyy.yyy 255.255.255.192 init
Oct 10 16:48:24 sg-5100 openvpn[91751]: Initialization Sequence Completed
Oct 10 17:48:23 sg-5100 openvpn[91751]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1570'
Oct 10 17:48:23 sg-5100 openvpn[91751]: WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA256'

For what it's worth, I tried enabling DCO but :

• my VPN provider doesn't seem to support AES-256-GCM
• openvpn crashes on VPN connection attempt without any other message, even with logging level set to the most verbose.
  As I know it's still experimental, I didn't fiddle any further.

[Myster_fr]

I realize something is not clear from my initial post : the OpenVPN client is my SG-5100, connecting to a VPN provider.

Therefore, please consider all elements shared previously as ccoming from the SG-5100 itself

[Myster_fr]

So, I kept testing.

• when I test the bandwidth through the VPN, I get capped to ~6-8MB/s, and I see the openvpn process on the SG-5100 hit 50% CPU (running top in SSH session)
• I checked the hardware acceleration settings, and it was set to "Intel QAT". I tried changing it to AES-NI, but it had no impact on the bandwidth limitation.
• still on the SG-5100, in the OpenVPN client settings, the "Hardware Crypto" setting is set to "Intel RDRAND engine - RAND".

Is there anything else I should be checking ?
Am I the only one witnessing this performance drop ?

[Myster_fr]

Humm,

I ended up shutting down pfBlocker-ng and immediately got all BW back !
So nothing to do with OpenVPN in the end...

[Gertjan]

@myster_fr

@myster_fr said in 5100/22.05 - Degraded OpenVPN client performance:

pfBlocker-ng

Or pfblockerng-devel 3.1.0_6 ? (new version since .... yesterday).

But, pfblockerng-devel can 'do things' with DNS requests.
pfblockerng-devel can not change or impact throughput like you've shown.
It doesn't create a traffic shaper / limiter or something like that.

And if DNS has difficulties to "go out", then only DNS resolution will suffer. Connections are always IP based, and have nothing to do with 'DNS'.
pfblockerng-devel doesn't interact with IP traffic.
( well, you could use IP pfblockerng-devel feeds that block access to known sites that host adds etc, but that traffic would be "zero" as it will get blocked, and never reach the VPN connection )

[Myster_fr]

@gertjan yes, it was pfblockerng-devel v3.1.0_6.

I have blocklists set to prevent traffic coming from "non friendly countries", basically, asia region, russia, some northern countries + africa.

But I agree, it is quite weird.
I've now made several tests with pfblocker-ng enabled/disabled, etc.. and always see the BW drop when pfblocker-ng is enabled.