OpenVPN connection breaks after around 15Mbps Authenticate/Decrypt packet error:

Turfrider · Mar 14, 2018, 3:52 PM

I have two issues
1. OpenVPN connection dying.
1a. This only happens if I download faster than about 15Mbps
1b. This still happens if something is downloading at 15Mbps and another client uses some traffic.
1c. This was not an issue on 2.3.x and I don't recall it been an issue when I first started testing 2.4 RC.
1d. Above is just stating what has happened so far, I'm really not sure about the root cause.
1e. This was running 2.3.x for almost a year without these issues.
1f. OpenVPN dying also can stop traffic routing through to the internet on other VLANs.
~~2. Intervlan routing times out after 20-30 seconds. This also never used to be an issue.~~
EDIT ^ - Rebooted the switch after months of uptime. Intervlan issue resolved. Will have to test openvpn at full speed when people disconnect.

No MTU changes on any of the systems, Ive been wondering if this is the issue.

Pfsense has most networks provided to it as VLAN's other than 10 and 70.
Pfsense creates a PPPOE connection

Fiber > Switch1 VLAN70 Access port
pfsense WAN > Switch VLAN70 Access port
pfsense LAN > Switch VLAN10(native),20,30,40,50,60 Trunk

Switch1 > Switch2 VLAN 10(native),20,30,40,50,60,65,70
Switch2 > UAP-Lite-AC VLAN 10(native),20,30,40,50 Trunk
Switch2 > Unraid VLAN 10(native),20,30,40,50,60,65,70

OpenVPN Issue
Change limited speed from 15Mbps to 40Mbps
Within about two or three mins of traffic at 40Mbps the OpenVPN log files fill with the logging shown below.

-OpenVPN connection dies, then reconnects. Once its reconnected no traffic passes through it, openvpn status should 40kb traffic.
-Clear internet access sometimes stops working, I feel there is a routing issue going on here.
-External access sometimes still works, e.g plex still working for people remotely. But not always and not something I often test.

Checking trace route to 8.8.8.8 on a network that does not use the VPN connection. (Resolver does have the interface listed as One possible interface for outgoing dns requests)
Switching to a network that uses public DNS only, and I still don't have internet access.

Strangely, I can still access my ISP's DNS servers and traceroute completes to them but nowhere else. I get the impression I have a routing issue that is killing my wan connection for all the networks somehow caused by this.

Adding mssfix 1400 reduces the speed I get from 40 to 25, but instead of the log file filling with the messages shown below. It's just one every so often until the same end result, loss of internet.

Shows I can't get to the internet anymore
1. 172.19.20.1 - Pfsense Gateway
2. 10.75.1.2 - First IP I see for internet traffic
3. ???

This is what it should look like when working normally.
1. 172.19.20.1
2. 10.75.1.2
3. 10.75.5.5
4. 10.55.201.198
5. 10.55.201.198
6. google1.lonap.net

This fills a couple of pages of log files before the connection dies.

Mar 14 15:23:32
openvpn
84879
Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #1568255 ] – see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Mar 14 15:23:32
openvpn
84879
PID_ERR large diff [65] [SSL-0] [0000000000000000000000000000000000000000________________________] 0:1568321 0:1568256 t=1521041012[0] r=[-4,64,15,139,1] sl=[25,64,64,528]
Mar 14 15:23:32
openvpn
84879
Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #1568256 ] – see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Mar 14 15:23:32
openvpn
84879
PID_ERR large diff [64] [SSL-0] [0000000000000000000000000000000000000000________________________] 0:1568321 0:1568257 t=1521041012[0] r=[-4,64,15,139,1] sl=[25,64,64,528]
Mar 14 15:23:32
openvpn
84879
Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #1568257 ] – see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

Turfrider · Mar 14, 2018, 3:55 PM

Any ideas or what other information should be shared?

Orignally I followed the dual wan pfsense setup guide, ended up having 3x VPN connections with 2x different providers. This was running good for around a year, i've not been able to figure out when this started happening but believe it was after the upgrade to 2.4.

Tried both GCM and CBC. Same results from what I can tell.

jimp · Mar 16, 2018, 2:28 PM

Those errors are almost always a link quality issue. Packets arriving out of order or duplicate copies of packets.

You can play with the replay window settings in OpenVPN but ultimately you probably need to look upstream for the source of the problem.

Turfrider · Mar 16, 2018, 3:11 PM

Jimp can you take a quick look at my other theard, basiclly the same issue but I noticed a change in routing table that effects my other vlans. i'm trying to understand what can cause the change in routing table. The "static" part is removed when openvpn dies, after it reconnects it's not replaced.

I'm giving up on UDP for the moment, but i made more comments about that in the other thread.

https://forum.pfsense.org/index.php?topic=145237.0

Before OpenVPN connection dies

Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP.static link#13 UHS lo0

After OpenVPN connection dies.

Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP link#13 UHS lo0