WAN interface loses .static when VPN fails at high traffic

Turfrider

This is the only difference I’ve found so far.

What can cause this interface related to my PPPOE connection to change status after a OpenVPN connection fails. All the do not apply route options are selected.
The .static is removed when openvpn fails, this causes all my other VLAN’s not routed through OpenVPN to also fail.

Before (from the routing table, full table show further down in my post)

Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP.static link#13 UHS lo0

After

Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP link#13 UHS lo0

I now know that I can stop my WAN interface and start it again to recover.

Below here is my actual post, before I noticed the change in .static shown above.
The last time I remember this been ok was on 2.3, I will be testing that version again soon I think.
I was running 2.4.2, now upgraded to 2.4.3 development branch, which didn’t change anything.

When pushing more than about 15MBs through the VPN connection, it brings stops all WAN traffic passing through the firewall. This includes other vlans that use pfsense as their default GW.
Speed limit it on the testing machine to 15MBs, I don’t see the errors in openvpn log and everything runs OK, increase that speed to between 18MB and 40MB/sec, everything breaks.

I’m on WIFI VLAN 20 > GW 172.19.20.1. (Same result with cable connection)
VPN traffic is on VLAN40 > GW 172.19.40.1

VPN traffic is generated on 172.19.40.10 for example

Both the system generating traffic and WIFI-AP are in a Layer2 switch, which is connected to another layer2 switch which is connected to pfsense. There is only one cable between the two switches, but pfsense has one cable for WAN and another for LANs/VLANS

Part 1
At this point the UI will show both VPN gateway and WAN gateway as Green. But I can’t ping anything through the WAN.

[2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 40 byte packets
1 10.75.1.2 (10.75.1.2) 0.923 ms 0.639 ms 0.600 ms
2 * * *

Note- Even more confusing sometimes, the only IP’s I can ping are my ISP’s DNS servers. In this case it didn’t work.
[2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 212.5.1.1
traceroute to 212.5.1.1 (212.50.1.1), 64 hops max, 40 byte packets
1 10.75.1.2 (10.75.1.2) 0.743 ms 0.650 ms 0.504 ms
2 * * *

MBP-5241:~ feck$ traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 52 byte packets
1 10.75.1.2 (10.75.1.2) 2.946 ms 1.812 ms 1.646 ms

Reboot pfsense

MBP-5241:~ mhardwick$ traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 52 byte packets
1 10.75.1.2 (10.75.1.2) 2.656 ms 1.609 ms 1.582 ms
2 10.75.5.5 (10.75.5.5) 3.888 ms 2.437 ms 2.034 ms
3 10.55.201.194 (10.55.201.194) 2.507 ms 2.802 ms 2.713 ms
4 10.55.201.194 (10.55.201.194) 2.230 ms 2.257 ms 2.070 ms
+many more etc..

[2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 40 byte packets
1 10.75.1.2 (10.75.1.2) 0.944 ms 0.675 ms 0.774 ms
2 10.75.5.5 (10.75.5.5) 0.949 ms 0.906 ms 0.987 ms
3 10.55.201.194 (10.55.201.194) 1.648 ms 1.731 ms 1.445 ms
4 10.55.201.194 (10.55.201.194) 1.345 ms 1.441 ms 1.162 ms
+many more etc..

Part 2
OpenVPN shows the following replay-window backtrack messages, then the connection dies and even if the log files show it recovers, at this point my VPN-LAN and None-VPN-LANs all lose internet access. Rebooting the pfsense box normally fixes.

Without mssfix 1400, the replay-window messages would be filling the log file when traffic is at max speed and dies within 2-3minutes. With mssfix 1400, the VPN connection lasted 9 mins, the frequency of these messages is greatly reduced but not stopped.

Before (this is while internet and vpn is working)

Internet:
Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP.static link#13 UHS lo0
10.4.0.0/16 10.4.0.1 UGS ovpnc1
10.4.0.1 link#15 UH ovpnc1
10.4.5.24 link#15 UHS lo0
10.75.1.2 link#13 UH pppoe0
localhost link#3 UH lo0
172.19.10.0/24 link#12 U re1.10
172.19.10.1 link#12 UHS lo0
172.19.20.0/24 link#7 U re1.20
172.19.20.1 link#7 UHS lo0
172.19.30.0/24 link#8 U re1.30
172.19.30.1 link#8 UHS lo0
172.19.40.0/24 link#9 U re1.40
172.19.40.1 link#9 UHS lo0
172.19.50.0/24 link#10 U re1.50
172.19.50.1 link#10 UHS lo0
172.19.60.0/24 link#11 U re1.60
172.19.60.1 link#11 UHS lo0
172.19.200.0/24 172.19.200.2 UGS ovpns3
172.19.200.1 link#14 UHS lo0
172.19.200.2 link#14 UH ovpns3
192.168.1.0/24 link#2 U re1
Raza link#2 UHS lo0

After (VPN has crashed similar to shown in the logging below, also lan traffic not routed through VPN also now fails)

Internet:
Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP link#13 UHS lo0
10.4.0.0/16 10.4.0.1 UGS ovpnc1
10.4.0.1 link#15 UH ovpnc1
10.4.5.24 link#15 UHS lo0
10.75.1.2 link#13 UH pppoe0
localhost link#3 UH lo0
172.19.10.0/24 link#12 U re1.10
172.19.10.1 link#12 UHS lo0
172.19.20.0/24 link#7 U re1.20
172.19.20.1 link#7 UHS lo0
172.19.30.0/24 link#8 U re1.30
172.19.30.1 link#8 UHS lo0
172.19.40.0/24 link#9 U re1.40
172.19.40.1 link#9 UHS lo0
172.19.50.0/24 link#10 U re1.50
172.19.50.1 link#10 UHS lo0
172.19.60.0/24 link#11 U re1.60
172.19.60.1 link#11 UHS lo0
172.19.200.0/24 172.19.200.2 UGS ovpns3
172.19.200.1 link#14 UHS lo0
172.19.200.2 link#14 UH ovpns3
192.168.1.0/24 link#2 U re1
Raza link#2 UHS lo0


Mar 15 13:05:06
openvpn
30838
PID_ERR replay-window backtrack occurred [12] [SSL-0] [000000000000_000000000000000000000000000000000000000000000000000] 0:2405392 0:2405380 t=1521119106[0] r=[0,64,15,12,1] sl=[48,64,64,528]
Mar 15 13:05:33
openvpn
30838
PID_ERR replay-window backtrack occurred [18] [SSL-0] [000000000000000000_000000000000000000000000000000000000000000000] 0:3158681 0:3158663 t=1521119133[0] r=[-2,64,15,18,1] sl=[39,64,64,528]
Mar 15 13:05:33
openvpn
30838
PID_ERR replay-window backtrack occurred [24] [SSL-0] [000000000000000000000000_000000000000000000000000000000000000000] 0:3165928 0:3165904 t=1521119133[0] r=[-2,64,15,24,1] sl=[24,64,64,528]
Mar 15 13:07:56
openvpn
30838
PID_ERR replay-window backtrack occurred [26] [SSL-0] [00000000000000000000000000_0000000000000000000000000000000000000] 0:6548134 0:6548108 t=1521119276[0] r=[0,64,15,26,1] sl=[26,64,64,528]
Mar 15 13:08:28
openvpn
30838
PID_ERR replay-window backtrack occurred [57] [SSL-0] [0000______________________________________________________000000] 0:7449452 0:7449395 t=1521119308[0] r=[-2,64,15,57,1] sl=[20,64,64,528]
Mar 15 13:09:28
openvpn
30838
[server] Inactivity timeout (--ping-restart), restarting
Mar 15 13:09:28
openvpn
30838
TCP/UDP: Closing socket
Mar 15 13:09:28
openvpn
30838
SIGUSR1[soft,ping-restart] received, process restarting
Mar 15 13:09:28
openvpn
30838
Restart pause, 5 second(s)
Mar 15 13:09:33
openvpn
30838
NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Mar 15 13:09:33
openvpn
30838
Re-using SSL/TLS context
Mar 15 13:09:33
openvpn
30838
Control Channel MTU parms [ L:1622 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Mar 15 13:09:33
openvpn
30838
Data Channel MTU parms [ L:1622 D:1400 EF:122 EB:406 ET:0 EL:3 ]
Mar 15 13:09:33
openvpn
30838
Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-client'
Mar 15 13:09:33
openvpn
30838
Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-server'
Mar 15 13:09:33
openvpn
30838
TCP/UDP: Preserving recently used remote address: [AF_INET]217.151.98.162:443
Mar 15 13:09:33
openvpn
30838
Socket Buffers: R=[42080->1048576] S=[57344->1048576]
Mar 15 13:09:33
openvpn
30838
UDPv4 link local: (not bound)
Mar 15 13:09:33
openvpn
30838
UDPv4 link remote: [AF_INET]217.151.98.162:443
Mar 15 13:09:33
openvpn
30838
TLS: Initial packet from [AF_INET]217.151.98.162:443 (via [AF_INET]PUBLIC.IP.ADDRESS.YUP), sid=a384c8dc 7bfcc125
Mar 15 13:09:33
openvpn
30838
VERIFY OK: depth=1, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=airvpn.org CA, emailAddress=info@airvpn.org
Mar 15 13:09:33
openvpn
30838
VERIFY KU OK
Mar 15 13:09:33
openvpn
30838
Validating certificate extended key usage
Mar 15 13:09:33
openvpn
30838
++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Mar 15 13:09:33
openvpn
30838
VERIFY EKU OK
Mar 15 13:09:33
openvpn
30838
VERIFY OK: depth=0, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=server, emailAddress=info@airvpn.org
Mar 15 13:09:33
openvpn
30838
Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
Mar 15 13:09:33
openvpn
30838
[server] Peer Connection Initiated with [AF_INET]217.151.98.162:443 (via [AF_INET]PUBLIC.IP.ADDRESS.YUP%)
Mar 15 13:09:34
openvpn
30838
SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
Mar 15 13:09:36
openvpn
30838
PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 10.4.0.1,comp-lzo no,route-gateway 10.4.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.4.5.24 255.255.0.0'
Mar 15 13:09:36
openvpn
30838
Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])
Mar 15 13:09:36
openvpn
30838
Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Mar 15 13:09:36
openvpn
30838
OPTIONS IMPORT: timers and/or timeouts modified
Mar 15 13:09:36
openvpn
30838
OPTIONS IMPORT: compression parms modified
Mar 15 13:09:36
openvpn
30838
OPTIONS IMPORT: --ifconfig/up options modified
Mar 15 13:09:36
openvpn
30838
OPTIONS IMPORT: route-related options modified
Mar 15 13:09:36
openvpn
30838
Data Channel MTU parms [ L:1558 D:1400 EF:58 EB:406 ET:0 EL:3 ]
Mar 15 13:09:36
openvpn
30838
Outgoing Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
Mar 15 13:09:36
openvpn
30838
Outgoing Data Channel: Using 160 bit message hash 'SHA1' for HMAC authentication
Mar 15 13:09:36
openvpn
30838
Incoming Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
Mar 15 13:09:36
openvpn
30838
Incoming Data Channel: Using 160 bit message hash 'SHA1' for HMAC authentication
Mar 15 13:09:36
openvpn
30838
Preserving previous TUN/TAP instance: ovpnc1
Mar 15 13:09:36
openvpn
30838
Initialization Sequence Completed

Turfrider

with mssfix 1400, 20MB/sec was stable. A few errors but no loss of connection.

22MB/sec gave a couple of errors but did not disconnect me

Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #10253565 ] – see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

24MB/sec started to spam errors and I lowered speed before it broke.

I guess it must just be latancy related when at high speeds over UDP, but my connection to the server and ping are solid outside of the tunnel from what I can tell.

Solved by… cheated really
Anyway, switched to TCP and reached 36MB/sec which isn't to far from my max without VPN.

The other issue with the routing table and the pppoe connection that shouldnt of been caused by openvpn failing shouldnt happen now as openvpn is stable.