6100 Poor Performance With Openvpn

[khodorb]

Hi,

i had an SG-4860 and had to upgrade it to 6100-MAX, i migrated the configuration,
my SG-4860 was working greate with openvpn setup for 50 user and it had a acceptable performance while using MS teams over VPN,

i was able to migrate the configuration and had only to adjust the intefaces,
initially i am facing a problem with in/out errors increasing on WAN interface , in addition to the disruption, lagging and voice cutting on ms teams calls.

i have tried the following :
enabled AES-NI encryption
set firewall to conservative mode.
tested the following combination for flow control and hardware checksum offloading

Flow_Control HW_Checksum Idrop Ipackets
Case 1(default settings) enabled disabled 1017 84218694
Case 2 disabled disabled 2108 128456664
Case 3 disabled enabled 861 126025861
Case 4 enabled enabled 1145 98748635

i am on version 22.05 with latest firmware ,

[22.05-RELEASE][pfsense]/root: ifconfig -vvm ix3
ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN
options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
ether 90:ec:77:29:72:dc
inet6 fe80::92ec:77ff:fe29:72dc%ix3 prefixlen 64 scopeid 0x8
inet 38.X.X.X netmask 0xffffffc0 broadcast X.X.X.255
inet X.X.X.220 netmask 0xffffffc0 broadcast X.X.X.255 vhid 12
inet X.X.X.221 netmask 0xffffffc0 broadcast X.X.X.255 vhid 13
inet X.X.X.230 netmask 0xffffffff broadcast X.X.X.230 vhid 15
inet X.X.X.252 netmask 0xffffffff broadcast X.X.X.252 vhid 16
inet X.X.X.247 netmask 0xffffffff broadcast X.X.X.247
carp: MASTER vhid 12 advbase 1 advskew 0
carp: MASTER vhid 13 advbase 1 advskew 0
carp: MASTER vhid 15 advbase 1 advskew 0
carp: MASTER vhid 16 advbase 1 advskew 0
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
media 10baseT/UTP
media 100baseTX
media 1000baseT
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[22.05-RELEASE][pfsense]/root: netstat -nI ix3
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
ix3 1500 <Link#8> 90:ec:77:29:72:dc 212495372 4046 0 208132359 0 0
ix3 - fe80::%ix3/64 fe80::92ec:77ff:f 0 - - 0 - -
ix3 - X.X.X.192 38.X.X.X 89092625 - - 1677680 - -
ix3 - X.X.X.192 X.X.X.220 0 - - 0 - -
ix3 - X.X.X.192 X.X.X.221 0 - - 0 - -
ix3 - X.X.X.230 X.X.X.230 0 - - 0 - -
ix3 - X.X.X.252 X.X.X.252 0 - - 0 - -
ix3 - X.X.X.247 X.X.X.247 0 - - 0 - -
[22.05-RELEASE][pfsense]/root: sysctl dev.ix.3 | grep err
dev.ix.3.mac_stats.checksum_errs: 4046
dev.ix.3.mac_stats.rec_len_errs: 0
dev.ix.3.mac_stats.byte_errs: 0
dev.ix.3.mac_stats.ill_errs: 0
dev.ix.3.mac_stats.crc_errs: 0
dev.ix.3.mac_stats.rx_errs: 4046
dev.ix.3.queue3.interrupt_rate: 31250
dev.ix.3.queue2.interrupt_rate: 31250
dev.ix.3.queue1.interrupt_rate: 31250
dev.ix.3.queue0.interrupt_rate: 31250
dev.ix.3.iflib.override_nrxds: 0
dev.ix.3.iflib.override_ntxds: 0
dev.ix.3.iflib.override_qs_enable: 0
dev.ix.3.iflib.override_nrxqs: 0
dev.ix.3.iflib.override_ntxqs: 0
[22.05-RELEASE][pfsense]/root: sysctl dev.ix.3 | grep fc
dev.ix.3.fc: 0
[22.05-RELEASE][pfsense]/root: sysctl hw.ix
hw.ix.enable_rss: 1
hw.ix.enable_fdir: 0
hw.ix.unsupported_sfp: 0
hw.ix.enable_msix: 1
hw.ix.advertise_speed: 0
hw.ix.flow_control: 0
hw.ix.max_interrupt_rate: 31250
[22.05-RELEASE][pfsense]/root:

[22.05-RELEASE][pfsense: /usr/bin/openssl engine -t -c
(devcrypto) /dev/crypto engine
[ available ]
(rdrand) Intel RDRAND engine
[RAND]
[ available ]
(dynamic) Dynamic engine loading support
[ unavailable ]

[22.05-RELEASE]pfsense: kldstat
Id Refs Address Size Name
1 30 0xffffffff80200000 3b03640 kernel
2 2 0xffffffff83d04000 9870 opensolaris.ko
3 1 0xffffffff83d0e000 39bde0 zfs.ko
4 3 0xffffffff84321000 50e0 gpiobus.ko
5 1 0xffffffff84327000 4a0 gpioled.ko
6 1 0xffffffff84328000 12c0 cordbuc.ko
7 1 0xffffffff8432a000 1010 cpuctl.ko
8 1 0xffffffff8432c000 87a0 aesni.ko
9 1 0xffffffff84335000 38a8 cryptodev.ko
10 1 0xffffffff84339000 bf8 coretemp.ko

this is my current openvpn settings that was migrated from SG-4860

dev ovpns1
disable-dco
verb 3
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
local X.X.X.194
tls-server
server 192.168.72.0 255.255.255.0
client-config-dir /var/etc/openvpn/server1/csc
username-as-common-name
plugin /usr/local/lib/openvpn/plugins/openvpn-plugin-auth-script.so /usr/local/sbin/ovpn_auth_verify_async user TG9jYWwgRGF0YWJhc2U= true server1 1194
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'X.X.X.194' 1"
lport 1194
management /var/etc/openvpn/server1/sock unix
max-clients 100
push "dhcp-option DOMAIN portfolioaid.local"
push "dhcp-option DNS 192.168.69.5"
push "dhcp-option DNS 192.168.69.8"
push "dhcp-option DNS 172.20.20.1"
push "dhcp-option DNS 185.228.168.112"
push "block-outside-dns"
push "register-dns"
push "redirect-gateway def1"
client-to-client
capath /var/etc/openvpn/server1/ca
cert /var/etc/openvpn/server1/cert
key /var/etc/openvpn/server1/key
dh /etc/dh-parameters.2048
tls-auth /var/etc/openvpn/server1/tls-auth 0
data-ciphers AES-256-CBC:AES-128-GCM
data-ciphers-fallback AES-256-CBC
allow-compression asym
persist-remote-ip
float
topology subnet
inactive 60
reneg-sec 0

on SG-4860 i was running on pfsense 2.4.5-p1 and it was great

i do have 1Gbps symetric on my wan interface, even for speed test when connected to vpn i can almost reach 120/40 Mbps

on the client side i tested 2 openvpn versions , 2.4.9 and 2.5.2 and same result.

this is my client config
dev tun
persist-tun
persist-key
data-ciphers AES-256-CBC:AES-128-GCM
data-ciphers-fallback AES-256-CBC
auth SHA256
tls-client
client
resolv-retry infinite
remote X.X.X.194 1194 udp4
nobind
verify-x509-name "X.X.X.194" name
auth-user-pass
pkcs12 pfsense-UDP4-1194-khodorb.p12
tls-auth pfsense-UDP4-1194-khodorb-tls.key 1
remote-cert-tls server
explicit-exit-notify
verb 4

i was checking states while connected to VPN and it shows established , i also checked the firewall for blocked traffic couldn't find any traffic being blocked for VPN, i don't know where to look, does anyone having problem with pfsense+ v 22.05 , do you guys recommand any stable version, any help would be highly appreciated .

[Gertjan]

@khodorb

These bit errors are invalid checksums, and they are incoming.
These means that between the upstream, sending device, and the receiving end, ix3, pfSense, something went electrically wrong.
Check fist : the upstream device.
The cable between the two devices.
The plugs on both sides.
You tried the ix2 ? I've one on my 4100, try that one.

[khodorb]

HI @gertjan
Actually i have 2 6100 devices, so i tested the second device on ix3 and got the same in/out errors in wan,
this was my setup

from my ISP i do have a fibre connection so when i was using the SG-4860 i was using a media converter and SFP adapter:

-TP Link fibre to Ethernet converter: MC220L-Gigabit Ethernet Media Converter
-TP Link SFP Module:TL-SM311LS(UN) VER3

so i tried using the same converter and connect that via ethenet on ix3 i also got the errors also i connecte the SFP directly to ix3 i am still having errors ,

I also asked my ISP for his configuration

and he provided this :

RP/0/RSP0/CPU0:nr11.b011027-3.yyz02#sh controllers Gi0/0/0/6 phy | i "x Power|Prod|Vend|avelength:"
Wed Aug 31 02:17:09.487 UTC
Vendor Name: OEM
Vendor OUI: 00.00.00
Vendor Part Number: SFP-GE-L-OEM (rev.: A )
Laser wavelength: 1310 nm (fraction: 0.00 nm)
Vendor Serial Number: FNS115001596
Tx Power: 0.27820 mW (-5.55643 dBm)
Rx Power: 0.17800 mW (-7.49580 dBm)
Product Id: SFP-GE-L
RP/0/RSP0/CPU0:nr11.b011027-3.yyz02#

MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
reliability 255/255, txload 1/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 1000Mb/s, LXFDX, link type is force-up
output flow control is off, input flow control is off
Carrier delay (up) is 10 msec
loopback not set,
Last link flapped 3d09h
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 3101000 bits/sec, 639 packets/sec
5 minute output rate 7139000 bits/sec, 966 packets/sec
7415383853 packets input, 5216629641057 bytes, 20 total input drops
0 drops for unrecognized upper-level protocol
Received 3827 broadcast packets, 28629876 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
7585701185 packets output, 5082547726685 bytes, 25384 total output drops
Output 67152672 broadcast packets, 46210 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
15 carrier transitions

when i looked online for the TP link SFP , i found this

i am not too sure what is wrong here.

[khodorb]

Hi Guys, Any thougts here

@stephenw10 , @jimp , @johnpoz

[marvosa]

@khodorb Admittedly, this is biased based on reputation, but the first thing I would do is replace anything with the TP-Link name on it.

[Alejo 0]

I have a similar setup with 2 x pfsense6100 using IPsec VPN but no TP Link and I have the same IN errors ("mac_stats.checksum_errs" output from sysctl dev.ix.3) on my ix3 interface. I wonder if this is related to a driver, cable or somewhere on the upstream area...

Sorry I had no answers for you :(

I will keep researching and will keep you posted if I find any solutions

[khodorb]

@alejo-0

found thid article

https://github.com/pfsense/FreeBSD-src/commit/5574b12aac2bdfe66ec1d9564f932eeec9ac213c

it is a drive that reports more errors , but unfortunately i am not able to know where to look in my setup

[Alejo 0]

@khodorb

That's a Github commit on the source code. From what I can tell, they added a piece of code to show these errors(the ones we are seeing now on our setups).

Since this piece of code wasn't there before, the errors weren't visible but now they are. In other words, we should have seen this errors before version 21.02 but we are only seeing them now.

I found the same link on the pfsense's redmine dating from 7 months ago, where Jim Pingle states the same.