WAN interface loses .static when VPN fails at high traffic
-
This is the only difference I’ve found so far.
What can cause this interface related to my PPPOE connection to change status after a OpenVPN connection fails. All the do not apply route options are selected.
The .static is removed when openvpn fails, this causes all my other VLAN’s not routed through OpenVPN to also fail.Before (from the routing table, full table show further down in my post)
Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP.static link#13 UHS lo0After
Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP link#13 UHS lo0I now know that I can stop my WAN interface and start it again to recover.
Below here is my actual post, before I noticed the change in .static shown above.
The last time I remember this been ok was on 2.3, I will be testing that version again soon I think.
I was running 2.4.2, now upgraded to 2.4.3 development branch, which didn’t change anything.When pushing more than about 15MBs through the VPN connection, it brings stops all WAN traffic passing through the firewall. This includes other vlans that use pfsense as their default GW.
Speed limit it on the testing machine to 15MBs, I don’t see the errors in openvpn log and everything runs OK, increase that speed to between 18MB and 40MB/sec, everything breaks.I’m on WIFI VLAN 20 > GW 172.19.20.1. (Same result with cable connection)
VPN traffic is on VLAN40 > GW 172.19.40.1VPN traffic is generated on 172.19.40.10 for example
Both the system generating traffic and WIFI-AP are in a Layer2 switch, which is connected to another layer2 switch which is connected to pfsense. There is only one cable between the two switches, but pfsense has one cable for WAN and another for LANs/VLANS
Part 1
At this point the UI will show both VPN gateway and WAN gateway as Green. But I can’t ping anything through the WAN.[2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 40 byte packets
1 10.75.1.2 (10.75.1.2) 0.923 ms 0.639 ms 0.600 ms
2 * * *Note- Even more confusing sometimes, the only IP’s I can ping are my ISP’s DNS servers. In this case it didn’t work.
[2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 212.5.1.1
traceroute to 212.5.1.1 (212.50.1.1), 64 hops max, 40 byte packets
1 10.75.1.2 (10.75.1.2) 0.743 ms 0.650 ms 0.504 ms
2 * * *MBP-5241:~ feck$ traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 52 byte packets
1 10.75.1.2 (10.75.1.2) 2.946 ms 1.812 ms 1.646 msReboot pfsense
MBP-5241:~ mhardwick$ traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 52 byte packets
1 10.75.1.2 (10.75.1.2) 2.656 ms 1.609 ms 1.582 ms
2 10.75.5.5 (10.75.5.5) 3.888 ms 2.437 ms 2.034 ms
3 10.55.201.194 (10.55.201.194) 2.507 ms 2.802 ms 2.713 ms
4 10.55.201.194 (10.55.201.194) 2.230 ms 2.257 ms 2.070 ms
+many more etc..[2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 4.4.4.4
traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 40 byte packets
1 10.75.1.2 (10.75.1.2) 0.944 ms 0.675 ms 0.774 ms
2 10.75.5.5 (10.75.5.5) 0.949 ms 0.906 ms 0.987 ms
3 10.55.201.194 (10.55.201.194) 1.648 ms 1.731 ms 1.445 ms
4 10.55.201.194 (10.55.201.194) 1.345 ms 1.441 ms 1.162 ms
+many more etc..Part 2
OpenVPN shows the following replay-window backtrack messages, then the connection dies and even if the log files show it recovers, at this point my VPN-LAN and None-VPN-LANs all lose internet access. Rebooting the pfsense box normally fixes.Without mssfix 1400, the replay-window messages would be filling the log file when traffic is at max speed and dies within 2-3minutes. With mssfix 1400, the VPN connection lasted 9 mins, the frequency of these messages is greatly reduced but not stopped.
Before (this is while internet and vpn is working)
Internet:
Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP.static link#13 UHS lo0
10.4.0.0/16 10.4.0.1 UGS ovpnc1
10.4.0.1 link#15 UH ovpnc1
10.4.5.24 link#15 UHS lo0
10.75.1.2 link#13 UH pppoe0
localhost link#3 UH lo0
172.19.10.0/24 link#12 U re1.10
172.19.10.1 link#12 UHS lo0
172.19.20.0/24 link#7 U re1.20
172.19.20.1 link#7 UHS lo0
172.19.30.0/24 link#8 U re1.30
172.19.30.1 link#8 UHS lo0
172.19.40.0/24 link#9 U re1.40
172.19.40.1 link#9 UHS lo0
172.19.50.0/24 link#10 U re1.50
172.19.50.1 link#10 UHS lo0
172.19.60.0/24 link#11 U re1.60
172.19.60.1 link#11 UHS lo0
172.19.200.0/24 172.19.200.2 UGS ovpns3
172.19.200.1 link#14 UHS lo0
172.19.200.2 link#14 UH ovpns3
192.168.1.0/24 link#2 U re1
Raza link#2 UHS lo0After (VPN has crashed similar to shown in the logging below, also lan traffic not routed through VPN also now fails)
Internet:
Destination Gateway Flags Netif Expire
default 10.75.1.2 UGS pppoe0
PUBLIC-IP link#13 UHS lo0
10.4.0.0/16 10.4.0.1 UGS ovpnc1
10.4.0.1 link#15 UH ovpnc1
10.4.5.24 link#15 UHS lo0
10.75.1.2 link#13 UH pppoe0
localhost link#3 UH lo0
172.19.10.0/24 link#12 U re1.10
172.19.10.1 link#12 UHS lo0
172.19.20.0/24 link#7 U re1.20
172.19.20.1 link#7 UHS lo0
172.19.30.0/24 link#8 U re1.30
172.19.30.1 link#8 UHS lo0
172.19.40.0/24 link#9 U re1.40
172.19.40.1 link#9 UHS lo0
172.19.50.0/24 link#10 U re1.50
172.19.50.1 link#10 UHS lo0
172.19.60.0/24 link#11 U re1.60
172.19.60.1 link#11 UHS lo0
172.19.200.0/24 172.19.200.2 UGS ovpns3
172.19.200.1 link#14 UHS lo0
172.19.200.2 link#14 UH ovpns3
192.168.1.0/24 link#2 U re1
Raza link#2 UHS lo0Mar 15 13:05:06 openvpn 30838 PID_ERR replay-window backtrack occurred [12] [SSL-0] [000000000000_000000000000000000000000000000000000000000000000000] 0:2405392 0:2405380 t=1521119106[0] r=[0,64,15,12,1] sl=[48,64,64,528] Mar 15 13:05:33 openvpn 30838 PID_ERR replay-window backtrack occurred [18] [SSL-0] [000000000000000000_000000000000000000000000000000000000000000000] 0:3158681 0:3158663 t=1521119133[0] r=[-2,64,15,18,1] sl=[39,64,64,528] Mar 15 13:05:33 openvpn 30838 PID_ERR replay-window backtrack occurred [24] [SSL-0] [000000000000000000000000_000000000000000000000000000000000000000] 0:3165928 0:3165904 t=1521119133[0] r=[-2,64,15,24,1] sl=[24,64,64,528] Mar 15 13:07:56 openvpn 30838 PID_ERR replay-window backtrack occurred [26] [SSL-0] [00000000000000000000000000_0000000000000000000000000000000000000] 0:6548134 0:6548108 t=1521119276[0] r=[0,64,15,26,1] sl=[26,64,64,528] Mar 15 13:08:28 openvpn 30838 PID_ERR replay-window backtrack occurred [57] [SSL-0] [0000______________________________________________________000000] 0:7449452 0:7449395 t=1521119308[0] r=[-2,64,15,57,1] sl=[20,64,64,528] Mar 15 13:09:28 openvpn 30838 [server] Inactivity timeout (--ping-restart), restarting Mar 15 13:09:28 openvpn 30838 TCP/UDP: Closing socket Mar 15 13:09:28 openvpn 30838 SIGUSR1[soft,ping-restart] received, process restarting Mar 15 13:09:28 openvpn 30838 Restart pause, 5 second(s) Mar 15 13:09:33 openvpn 30838 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Mar 15 13:09:33 openvpn 30838 Re-using SSL/TLS context Mar 15 13:09:33 openvpn 30838 Control Channel MTU parms [ L:1622 D:1184 EF:66 EB:0 ET:0 EL:3 ] Mar 15 13:09:33 openvpn 30838 Data Channel MTU parms [ L:1622 D:1400 EF:122 EB:406 ET:0 EL:3 ] Mar 15 13:09:33 openvpn 30838 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-client' Mar 15 13:09:33 openvpn 30838 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-server' Mar 15 13:09:33 openvpn 30838 TCP/UDP: Preserving recently used remote address: [AF_INET]217.151.98.162:443 Mar 15 13:09:33 openvpn 30838 Socket Buffers: R=[42080->1048576] S=[57344->1048576] Mar 15 13:09:33 openvpn 30838 UDPv4 link local: (not bound) Mar 15 13:09:33 openvpn 30838 UDPv4 link remote: [AF_INET]217.151.98.162:443 Mar 15 13:09:33 openvpn 30838 TLS: Initial packet from [AF_INET]217.151.98.162:443 (via [AF_INET]PUBLIC.IP.ADDRESS.YUP), sid=a384c8dc 7bfcc125 Mar 15 13:09:33 openvpn 30838 VERIFY OK: depth=1, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=airvpn.org CA, emailAddress=info@airvpn.org Mar 15 13:09:33 openvpn 30838 VERIFY KU OK Mar 15 13:09:33 openvpn 30838 Validating certificate extended key usage Mar 15 13:09:33 openvpn 30838 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication Mar 15 13:09:33 openvpn 30838 VERIFY EKU OK Mar 15 13:09:33 openvpn 30838 VERIFY OK: depth=0, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=server, emailAddress=info@airvpn.org Mar 15 13:09:33 openvpn 30838 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 4096 bit RSA Mar 15 13:09:33 openvpn 30838 [server] Peer Connection Initiated with [AF_INET]217.151.98.162:443 (via [AF_INET]PUBLIC.IP.ADDRESS.YUP%) Mar 15 13:09:34 openvpn 30838 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1) Mar 15 13:09:36 openvpn 30838 PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 10.4.0.1,comp-lzo no,route-gateway 10.4.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.4.5.24 255.255.0.0' Mar 15 13:09:36 openvpn 30838 Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS]) Mar 15 13:09:36 openvpn 30838 Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS]) Mar 15 13:09:36 openvpn 30838 OPTIONS IMPORT: timers and/or timeouts modified Mar 15 13:09:36 openvpn 30838 OPTIONS IMPORT: compression parms modified Mar 15 13:09:36 openvpn 30838 OPTIONS IMPORT: --ifconfig/up options modified Mar 15 13:09:36 openvpn 30838 OPTIONS IMPORT: route-related options modified Mar 15 13:09:36 openvpn 30838 Data Channel MTU parms [ L:1558 D:1400 EF:58 EB:406 ET:0 EL:3 ] Mar 15 13:09:36 openvpn 30838 Outgoing Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key Mar 15 13:09:36 openvpn 30838 Outgoing Data Channel: Using 160 bit message hash 'SHA1' for HMAC authentication Mar 15 13:09:36 openvpn 30838 Incoming Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key Mar 15 13:09:36 openvpn 30838 Incoming Data Channel: Using 160 bit message hash 'SHA1' for HMAC authentication Mar 15 13:09:36 openvpn 30838 Preserving previous TUN/TAP instance: ovpnc1 Mar 15 13:09:36 openvpn 30838 Initialization Sequence Completed
-
with mssfix 1400, 20MB/sec was stable. A few errors but no loss of connection.
22MB/sec gave a couple of errors but did not disconnect me
Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #10253565 ] – see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
24MB/sec started to spam errors and I lowered speed before it broke.
I guess it must just be latancy related when at high speeds over UDP, but my connection to the server and ping are solid outside of the tunnel from what I can tell.
Solved by… cheated really
Anyway, switched to TCP and reached 36MB/sec which isn't to far from my max without VPN.The other issue with the routing table and the pppoe connection that shouldnt of been caused by openvpn failing shouldnt happen now as openvpn is stable.