WAN interface loses .static when VPN fails at high traffic



  • This is the only difference I’ve found so far.

    What can cause this interface related to my PPPOE connection to change status after a OpenVPN connection fails. All the do not apply route options are selected.
    The .static is removed when openvpn fails, this causes all my other VLAN’s not routed through OpenVPN to also fail.

    Before (from the routing table, full table show further down in my post)

    Destination        Gateway            Flags    Netif Expire
    default            10.75.1.2          UGS      pppoe0
    PUBLIC-IP.static link#13            UHS        lo0

    After

    Destination        Gateway            Flags    Netif Expire
    default            10.75.1.2          UGS      pppoe0
    PUBLIC-IP        link#13            UHS        lo0

    I now know that I can stop my WAN interface and start it again to recover.

    Below here is my actual post, before I noticed the change in .static shown above.
    The last time I remember this been ok was on 2.3, I will be testing that version again soon I think.
    I was running 2.4.2, now upgraded to 2.4.3 development branch, which didn’t change anything.

    When pushing more than about 15MBs through the VPN connection, it brings stops all WAN traffic passing through the firewall. This includes other vlans that use pfsense as their default GW.
    Speed limit it on the testing machine to 15MBs, I don’t see the errors in openvpn log and everything runs OK, increase that speed to between 18MB and 40MB/sec, everything breaks.

    I’m on WIFI VLAN 20 > GW 172.19.20.1. (Same result with cable connection)
    VPN traffic is on VLAN40 > GW 172.19.40.1

    VPN traffic is generated on 172.19.40.10 for example

    Both the system generating traffic and WIFI-AP are in a Layer2 switch, which is connected to another layer2 switch which is connected to pfsense. There is only one cable between the two switches, but pfsense has one cable for WAN and another for LANs/VLANS

    Part 1
    At this point the UI will show both VPN gateway and WAN gateway as Green. But I can’t ping anything through the WAN.

    [2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 4.4.4.4
    traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 40 byte packets
    1  10.75.1.2 (10.75.1.2)  0.923 ms  0.639 ms  0.600 ms
    2  * * *

    Note- Even more confusing sometimes, the only IP’s I can ping are my ISP’s DNS servers. In this case it didn’t work.
    [2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 212.5.1.1
    traceroute to 212.5.1.1 (212.50.1.1), 64 hops max, 40 byte packets
    1  10.75.1.2 (10.75.1.2)  0.743 ms  0.650 ms  0.504 ms
    2  * * *

    MBP-5241:~ feck$ traceroute 4.4.4.4
    traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 52 byte packets
    1  10.75.1.2 (10.75.1.2)  2.946 ms  1.812 ms  1.646 ms

    Reboot pfsense

    MBP-5241:~ mhardwick$ traceroute 4.4.4.4
    traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 52 byte packets
    1  10.75.1.2 (10.75.1.2)  2.656 ms  1.609 ms  1.582 ms
    2  10.75.5.5 (10.75.5.5)  3.888 ms  2.437 ms  2.034 ms
    3  10.55.201.194 (10.55.201.194)  2.507 ms  2.802 ms  2.713 ms
    4  10.55.201.194 (10.55.201.194)  2.230 ms  2.257 ms  2.070 ms
    +many more etc..

    [2.4.3-DEVELOPMENT][admin@Raza.local.lan]/root: traceroute 4.4.4.4
    traceroute to 4.4.4.4 (4.4.4.4), 64 hops max, 40 byte packets
    1  10.75.1.2 (10.75.1.2)  0.944 ms  0.675 ms  0.774 ms
    2  10.75.5.5 (10.75.5.5)  0.949 ms  0.906 ms  0.987 ms
    3  10.55.201.194 (10.55.201.194)  1.648 ms  1.731 ms  1.445 ms
    4  10.55.201.194 (10.55.201.194)  1.345 ms  1.441 ms  1.162 ms
    +many more etc..

    Part 2
    OpenVPN shows the following replay-window backtrack messages, then the connection dies and even if the log files show it recovers, at this point my VPN-LAN and None-VPN-LANs all lose internet access. Rebooting the pfsense box normally fixes.

    Without mssfix 1400, the replay-window messages would be filling the log file when traffic is at max speed and dies within 2-3minutes. With mssfix 1400, the VPN connection lasted 9 mins, the frequency of these messages is greatly reduced but not stopped.

    Before (this is while internet and vpn is working)

    Internet:
    Destination        Gateway            Flags    Netif Expire
    default            10.75.1.2          UGS      pppoe0
    PUBLIC-IP.static link#13            UHS        lo0
    10.4.0.0/16        10.4.0.1          UGS      ovpnc1
    10.4.0.1          link#15            UH      ovpnc1
    10.4.5.24          link#15            UHS        lo0
    10.75.1.2          link#13            UH      pppoe0
    localhost          link#3            UH          lo0
    172.19.10.0/24    link#12            U        re1.10
    172.19.10.1        link#12            UHS        lo0
    172.19.20.0/24    link#7            U        re1.20
    172.19.20.1        link#7            UHS        lo0
    172.19.30.0/24    link#8            U        re1.30
    172.19.30.1        link#8            UHS        lo0
    172.19.40.0/24    link#9            U        re1.40
    172.19.40.1        link#9            UHS        lo0
    172.19.50.0/24    link#10            U        re1.50
    172.19.50.1        link#10            UHS        lo0
    172.19.60.0/24    link#11            U        re1.60
    172.19.60.1        link#11            UHS        lo0
    172.19.200.0/24    172.19.200.2      UGS      ovpns3
    172.19.200.1      link#14            UHS        lo0
    172.19.200.2      link#14            UH      ovpns3
    192.168.1.0/24    link#2            U          re1
    Raza              link#2            UHS        lo0

    After (VPN has crashed similar to shown in the logging below, also lan traffic not routed through VPN also now fails)

    Internet:
    Destination        Gateway            Flags    Netif Expire
    default            10.75.1.2          UGS      pppoe0
    PUBLIC-IP        link#13            UHS        lo0
    10.4.0.0/16        10.4.0.1          UGS      ovpnc1
    10.4.0.1          link#15            UH      ovpnc1
    10.4.5.24          link#15            UHS        lo0
    10.75.1.2          link#13            UH      pppoe0
    localhost          link#3            UH          lo0
    172.19.10.0/24    link#12            U        re1.10
    172.19.10.1        link#12            UHS        lo0
    172.19.20.0/24    link#7            U        re1.20
    172.19.20.1        link#7            UHS        lo0
    172.19.30.0/24    link#8            U        re1.30
    172.19.30.1        link#8            UHS        lo0
    172.19.40.0/24    link#9            U        re1.40
    172.19.40.1        link#9            UHS        lo0
    172.19.50.0/24    link#10            U        re1.50
    172.19.50.1        link#10            UHS        lo0
    172.19.60.0/24    link#11            U        re1.60
    172.19.60.1        link#11            UHS        lo0
    172.19.200.0/24    172.19.200.2      UGS      ovpns3
    172.19.200.1      link#14            UHS        lo0
    172.19.200.2      link#14            UH      ovpns3
    192.168.1.0/24    link#2            U          re1
    Raza              link#2            UHS        lo0

    
    Mar 15 13:05:06
    openvpn
    30838
    PID_ERR replay-window backtrack occurred [12] [SSL-0] [000000000000_000000000000000000000000000000000000000000000000000] 0:2405392 0:2405380 t=1521119106[0] r=[0,64,15,12,1] sl=[48,64,64,528]
    Mar 15 13:05:33
    openvpn
    30838
    PID_ERR replay-window backtrack occurred [18] [SSL-0] [000000000000000000_000000000000000000000000000000000000000000000] 0:3158681 0:3158663 t=1521119133[0] r=[-2,64,15,18,1] sl=[39,64,64,528]
    Mar 15 13:05:33
    openvpn
    30838
    PID_ERR replay-window backtrack occurred [24] [SSL-0] [000000000000000000000000_000000000000000000000000000000000000000] 0:3165928 0:3165904 t=1521119133[0] r=[-2,64,15,24,1] sl=[24,64,64,528]
    Mar 15 13:07:56
    openvpn
    30838
    PID_ERR replay-window backtrack occurred [26] [SSL-0] [00000000000000000000000000_0000000000000000000000000000000000000] 0:6548134 0:6548108 t=1521119276[0] r=[0,64,15,26,1] sl=[26,64,64,528]
    Mar 15 13:08:28
    openvpn
    30838
    PID_ERR replay-window backtrack occurred [57] [SSL-0] [0000______________________________________________________000000] 0:7449452 0:7449395 t=1521119308[0] r=[-2,64,15,57,1] sl=[20,64,64,528]
    Mar 15 13:09:28
    openvpn
    30838
    [server] Inactivity timeout (--ping-restart), restarting
    Mar 15 13:09:28
    openvpn
    30838
    TCP/UDP: Closing socket
    Mar 15 13:09:28
    openvpn
    30838
    SIGUSR1[soft,ping-restart] received, process restarting
    Mar 15 13:09:28
    openvpn
    30838
    Restart pause, 5 second(s)
    Mar 15 13:09:33
    openvpn
    30838
    NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
    Mar 15 13:09:33
    openvpn
    30838
    Re-using SSL/TLS context
    Mar 15 13:09:33
    openvpn
    30838
    Control Channel MTU parms [ L:1622 D:1184 EF:66 EB:0 ET:0 EL:3 ]
    Mar 15 13:09:33
    openvpn
    30838
    Data Channel MTU parms [ L:1622 D:1400 EF:122 EB:406 ET:0 EL:3 ]
    Mar 15 13:09:33
    openvpn
    30838
    Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 1,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-client'
    Mar 15 13:09:33
    openvpn
    30838
    Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,keydir 0,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-server'
    Mar 15 13:09:33
    openvpn
    30838
    TCP/UDP: Preserving recently used remote address: [AF_INET]217.151.98.162:443
    Mar 15 13:09:33
    openvpn
    30838
    Socket Buffers: R=[42080->1048576] S=[57344->1048576]
    Mar 15 13:09:33
    openvpn
    30838
    UDPv4 link local: (not bound)
    Mar 15 13:09:33
    openvpn
    30838
    UDPv4 link remote: [AF_INET]217.151.98.162:443
    Mar 15 13:09:33
    openvpn
    30838
    TLS: Initial packet from [AF_INET]217.151.98.162:443 (via [AF_INET]PUBLIC.IP.ADDRESS.YUP), sid=a384c8dc 7bfcc125
    Mar 15 13:09:33
    openvpn
    30838
    VERIFY OK: depth=1, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=airvpn.org CA, emailAddress=info@airvpn.org
    Mar 15 13:09:33
    openvpn
    30838
    VERIFY KU OK
    Mar 15 13:09:33
    openvpn
    30838
    Validating certificate extended key usage
    Mar 15 13:09:33
    openvpn
    30838
    ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
    Mar 15 13:09:33
    openvpn
    30838
    VERIFY EKU OK
    Mar 15 13:09:33
    openvpn
    30838
    VERIFY OK: depth=0, C=IT, ST=IT, L=Perugia, O=airvpn.org, CN=server, emailAddress=info@airvpn.org
    Mar 15 13:09:33
    openvpn
    30838
    Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
    Mar 15 13:09:33
    openvpn
    30838
    [server] Peer Connection Initiated with [AF_INET]217.151.98.162:443 (via [AF_INET]PUBLIC.IP.ADDRESS.YUP%)
    Mar 15 13:09:34
    openvpn
    30838
    SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
    Mar 15 13:09:36
    openvpn
    30838
    PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 10.4.0.1,comp-lzo no,route-gateway 10.4.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.4.5.24 255.255.0.0'
    Mar 15 13:09:36
    openvpn
    30838
    Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])
    Mar 15 13:09:36
    openvpn
    30838
    Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
    Mar 15 13:09:36
    openvpn
    30838
    OPTIONS IMPORT: timers and/or timeouts modified
    Mar 15 13:09:36
    openvpn
    30838
    OPTIONS IMPORT: compression parms modified
    Mar 15 13:09:36
    openvpn
    30838
    OPTIONS IMPORT: --ifconfig/up options modified
    Mar 15 13:09:36
    openvpn
    30838
    OPTIONS IMPORT: route-related options modified
    Mar 15 13:09:36
    openvpn
    30838
    Data Channel MTU parms [ L:1558 D:1400 EF:58 EB:406 ET:0 EL:3 ]
    Mar 15 13:09:36
    openvpn
    30838
    Outgoing Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
    Mar 15 13:09:36
    openvpn
    30838
    Outgoing Data Channel: Using 160 bit message hash 'SHA1' for HMAC authentication
    Mar 15 13:09:36
    openvpn
    30838
    Incoming Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
    Mar 15 13:09:36
    openvpn
    30838
    Incoming Data Channel: Using 160 bit message hash 'SHA1' for HMAC authentication
    Mar 15 13:09:36
    openvpn
    30838
    Preserving previous TUN/TAP instance: ovpnc1
    Mar 15 13:09:36
    openvpn
    30838
    Initialization Sequence Completed
    
    


  • with mssfix 1400, 20MB/sec was stable. A few errors but no loss of connection.

    22MB/sec gave a couple of errors but did not disconnect me

    Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #10253565 ] – see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

    24MB/sec started to spam errors and I lowered speed before it broke.

    I guess it must just be latancy related when at high speeds over UDP, but my connection to the server and ping are solid outside of the tunnel from what I can tell.

    Solved by… cheated really
    Anyway, switched to TCP and reached 36MB/sec which isn't to far from my max without VPN.

    The other issue with the routing table and the pppoe connection that shouldnt of been caused by openvpn failing shouldnt happen now as openvpn is stable.


Log in to reply