@AhnHEL:
You could also try disabling "State Killing On Gateway Failure" in System/Advanced/Miscellaneous within the GUI. This should keep your VPN up when Apinger reports a Loss.
Actually I found that this isn't a state clearing issue.
When delay occurs:
Mar 24 12:14:36 apinger: alarm canceled: AWAN(x.x.x.x) *** AWANdelay ***
Mar 24 12:14:08 apinger: ALARM: AWAN(x.x.x.x) *** AWANdelay ***
The firewall is RESTARTING services altogether!
Mar 24 12:14:52 php-fpm[15338]: /rc.start_packages: Restarting/Starting all packages.
Mar 24 12:14:51 check_reload_status: Starting packages
Mar 24 12:14:51 php-fpm[15338]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 10.1.1.1 - Restarting packages.
Mar 24 12:14:51 check_reload_status: Reloading filter
Mar 24 12:14:51 php-fpm[15338]: /rc.newwanip: rc.newwanip: on (IP address: 10.1.1.1) (interface: []) (real interface: ovpns4).
Mar 24 12:14:51 php-fpm[15338]: /rc.newwanip: rc.newwanip: Info: starting on ovpns4.
Mar 24 12:14:50 check_reload_status: rc.newwanip starting ovpns4
Mar 24 12:14:50 kernel: ovpns4: link state changed to UP
Mar 24 12:14:47 check_reload_status: Reloading filter
Mar 24 12:14:47 kernel: ovpns4: link state changed to DOWN
Mar 24 12:14:47 php-fpm[65238]: /rc.openvpn: OpenVPN: Resync server4 Remote Access VPN
Mar 24 12:14:47 php-fpm[65238]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use AWAN.
Mar 24 12:14:46 check_reload_status: Reloading filter
Mar 24 12:14:46 check_reload_status: Restarting OpenVPN tunnels/interfaces
Mar 24 12:14:46 check_reload_status: Restarting ipsec tunnels
Mar 24 12:14:46 check_reload_status: updating dyndns AWAN
Mar 24 12:14:33 nrpe[3162]: There's already an NRPE server running (PID 88503). Bailing out…
Mar 24 12:14:33 nrpe[3162]: Starting up daemon
Mar 24 12:14:31 php-fpm[25418]: /rc.filter_configure_sync: MONITOR: AWAN has high latency, omitting from routing group WAN1toWAN2
Mar 24 12:14:30 nrpe[73937]: There's already an NRPE server running (PID 88503). Bailing out…
Mar 24 12:14:30 nrpe[73937]: Starting up daemon
Mar 24 12:14:29 php-fpm[56975]: /rc.start_packages: [filer] filer_xmlrpc_sync.php is starting.
Mar 24 12:14:29 php-fpm[56975]: /rc.start_packages: [filer] filer_xmlrpc_sync.php is starting.
Mar 24 12:14:28 php-fpm[56975]: /rc.start_packages: Restarting/Starting all packages.
Mar 24 12:14:27 check_reload_status: Starting packages
Mar 24 12:14:27 php-fpm[39957]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> 10.1.1.1 - Restarting packages.
Mar 24 12:14:27 check_reload_status: Reloading filter
Mar 24 12:14:27 php-fpm[39957]: /rc.newwanip: rc.newwanip: on (IP address: 10.1.1.1) (interface: []) (real interface: ovpns4).
Mar 24 12:14:27 php-fpm[39957]: /rc.newwanip: rc.newwanip: Info: starting on ovpns4.
Mar 24 12:14:26 check_reload_status: rc.newwanip starting ovpns4
Mar 24 12:14:26 kernel: ovpns4: link state changed to UP
Mar 24 12:14:21 php-fpm[34202]: /rc.filter_configure_sync: MONITOR: AWAN has high latency, omitting from routing group WAN1toWAN2
Mar 24 12:14:20 php-fpm[25418]: /rc.openvpn: MONITOR: AWAN has high latency, omitting from routing group WAN1toWAN2
Mar 24 12:14:20 check_reload_status: Reloading filter
Mar 24 12:14:20 kernel: ovpns4: link state changed to DOWN
Mar 24 12:14:20 php-fpm[25418]: /rc.openvpn: MONITOR: AWAN has high latency, omitting from routing group WAN1toWAN2
Mar 24 12:14:20 php-fpm[25418]: /rc.openvpn: MONITOR: AWAN has high latency, omitting from routing group WAN1toWAN2
Mar 24 12:14:20 php-fpm[25418]: /rc.openvpn: OpenVPN: Resync server4 Remote Access VPN
Mar 24 12:14:20 php-fpm[25418]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use AWAN.
Mar 24 12:14:20 php-fpm[25418]: /rc.dyndns.update: MONITOR: AWAN has high latency, omitting from routing group WAN1toWAN2
Mar 24 12:14:19 check_reload_status: Reloading filter
Mar 24 12:14:19 check_reload_status: Restarting OpenVPN tunnels/interfaces
Mar 24 12:14:19 check_reload_status: Restarting ipsec tunnels
Mar 24 12:14:19 check_reload_status: updating dyndns AWAN
I know I need to educate myself on the traffic shaper in PFSense, however, it seems to me that services shouldn't be restarted just because apinger detects delay or removes a gateway from a group. And I do currently have state killing disabled.