Problem with Automatic Filter Reload when OpenVPN is in a Gateway Group

AP

Hello,

I currently have a Gateway Group consisting of two gateways in failover mode:

OpenVPN (tier 1)
OPT1 (tier 2)

My two physical uplinks are on WAN and OPT1. The OpenVPN client uses WAN as its uplink. The WAN link is my primary uplink but is a public network, so I tunnel traffic going out this interface through an OpenVPN server. OPT1 is over an LTE network and is bandwidth constrained.

I have set up rules so that all LAN traffic is sent through the GW Group, along with proper NAT rules. Overall its working well.

This is fairly new set up and I ran into an interesting problem:

In a normal scenario, both the OpenVPN and OPT1 Gateways are active and LAN traffic is routed through OpenVPN (as expected)
If my OpenVPN server goes down, the OpenVPN gateway also goes down and LAN traffic is shifted to OPT1 (as expected)
Here's the weird part: When the OpenVPN server comes back online, the OpenVPN Gateway comes back on as well. However, LAN traffic will continue to route through OPT1
If I manually initiate a Filter Reload the problem is resolved and LAN traffic correctly returns to be routed through OpenVPN

I dug into the System Log and noticed that an automatic filter reload occurs if a physical gateway goes up or down (WAN or OPT1), and also occurs when the OpenVPN link goes down, but a reload does not occur if the OpenVPN link comes back up. Please see the images below.

Digging into the firewall rules (using command "pfctl -sr -v"), I followed how this rule changes in each of the scenarios above (the number corresponding to the scenario above):

pass in quick on mvneta0.4091 route-to (ovpnc1 172.19.17.2) inet from 172.19.26.0/24 to any flags S/SA keep state label "USER_RULE: LAN to any rule"
pass in quick on mvneta0.4091 route-to (mvneta0.4092 172.19.25.1) inet from 172.19.26.0/24 to any flags S/SA keep state label "USER_RULE: LAN to any rule"
pass in quick on mvneta0.4091 route-to (mvneta0.4092 172.19.25.1) inet from 172.19.26.0/24 to any flags S/SA keep state label "USER_RULE: LAN to any rule"
pass in quick on mvneta0.4091 route-to (ovpnc1 172.19.17.2) inet from 172.19.26.0/24 to any flags S/SA keep state label "USER_RULE: LAN to any rule"

Similarly, looking at /tmp/rules.debug in the various scenarios:

GWGateway_Failover = " route-to { ( ovpnc1 172.19.17.2 ) } "
GWGateway_Failover = " route-to { ( mvneta0.4092 172.19.25.1 ) } "
GWGateway_Failover = " route-to { ( mvneta0.4092 172.19.25.1 ) } "
GWGateway_Failover = " route-to { ( ovpnc1 172.19.17.2 ) } "

As you can likely assume, my LAN subnet is 172.19.26.0/24, gateway IP for OPT1 is 172.19.25.1 and OpenVPN is 172.19.17.2.

So it looks like pfSense not initiating a filter reloading when the OpenVPN gateway comes back online.

Has anyone run into a similar issue? Is there a way I can configure pfSense to auto reload filters when OpenVPN comes back online?

Note that I have also tried enabling State Killing in the advanced menu, which didn't help.

Thanks a lot,
AP

System Log when a physical link goes down and up (OPT1). Notice the "Reloading filter" at 10:37:23, when the link returns online:

System Log when OpenVPN link goes down and up. Notice the lack of "Reloading filter" once the link comes back up:

AP

Just as an update, I was able to create a pretty simple work-around for this issue.

I added the following directive to the end of this file: /usr/local/sbin/ovpn-linkup (which runs once the OpenVPN client link comes online)

(/bin/sleep 120 && /etc/rc.filter_configure)&

This basically waits 2 mins after the OpenVPN link comes up (enough time for the newwanip stuff to finish) and forces a filter reload.

Funny enough, the last line of the file is already a command to reload the filters, but I guess there's something weird going on with it.

Marking as resolved.

darcey

I encountered a similar problem. A policy rule referencing the vpn gateway was not getting created at start up.
I first tried specifying an IP rather than hostname for vpn server in the client config (in case lack of DNS delayed VPN coming up). That did not help. I then tried the fix above and thatseems to have worked.
Since pfSense does not create policy rules if the gateway is not up, it's strange very few users seem to have experienced the issue. I imagined many would have policy routing rules that depend on a VPN gateway.
I prefer to avoid modifying system scripts. Is there another approach to fixing this?
Thanks.

Jon8RFC

Sorry to bump this. It seems like a bug, or at least some type of "it works for enough people, so it's probably fine" scenario.

The workaround suggested above resolves it--thank you! I had been manually reloading the filter after finally figuring out, through trial and error, that a filter reload was a reliable fix for this problem with PFSense.

Looking at the system logs, it seems like the filter reload happens, but too soon in the process:

Feb 7 15:54:00 check_reload_status 377 Syncing firewall
Feb 7 15:54:00 php-fpm 339 OpenVPN PID written: 69810
Feb 7 15:54:00 check_reload_status 377 Reloading filter
Feb 7 15:54:01 kernel ovpnc1: link state changed to UP
Feb 7 15:54:01 check_reload_status 377 rc.newwanip starting ovpnc1
Feb 7 15:54:02 php-fpm 72794 /rc.newwanip: rc.newwanip: Info: starting on ovpnc1.
Feb 7 15:54:02 php-fpm 72794 /rc.newwanip: rc.newwanip: on (IP address: 192.168.12.4) (interface: OPT1_VPN[opt1]) (real interface: ovpnc1).
Feb 7 15:54:09 php-fpm 72794 /rc.newwanip: Gateway, none 'available' for inet6, use the first one configured. ''
Feb 7 15:54:09 php-fpm 72794 /rc.newwanip: IP Address has changed, killing states on former IP Address 192.168.12.3.
Feb 7 15:54:12 dhcpleases 52624 Could not deliver signal HUP to process 47506: No such process.
Feb 7 15:54:13 php-fpm 72794 /rc.newwanip: Creating rrd update script
Feb 7 15:54:15 php-fpm 72794 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.12.3 -> 192.168.12.4 - Restarting packages.
Feb 7 15:54:15 check_reload_status 377 Starting packages
Feb 7 15:54:16 php-fpm 339 /rc.start_packages: Restarting/Starting all packages.
Feb 7 15:54:16 check_reload_status 377 Syncing firewall

It seems to work properly after the link state changed to DOWN, however:

Feb 7 15:34:40 check_reload_status 377 Syncing firewall
Feb 7 15:34:40 php-fpm 338 OpenVPN terminate old pid: 93674
Feb 7 15:34:40 kernel ovpnc1: link state changed to DOWN
Feb 7 15:34:40 check_reload_status 377 Reloading filter

But, changing

/usr/bin/touch /tmp/${1}up
# reload filter
/usr/local/sbin/pfSctl -c "interface newip ${1}"
exit 0

to

/usr/bin/touch /tmp/${1}up
# reload filter
/usr/local/sbin/pfSctl -c "interface newip ${1}"
(/bin/sleep 20 && /etc/rc.filter_configure)&
exit 0

Fixes my issue:

Feb 7 15:58:18 check_reload_status 377 Syncing firewall
Feb 7 15:58:18 php-fpm 339 OpenVPN PID written: 45730
Feb 7 15:58:18 check_reload_status 377 Reloading filter
Feb 7 15:58:19 kernel ovpnc1: link state changed to UP
Feb 7 15:58:19 check_reload_status 377 rc.newwanip starting ovpnc1
Feb 7 15:58:20 php-fpm 85757 /rc.newwanip: rc.newwanip: Info: starting on ovpnc1.
Feb 7 15:58:20 php-fpm 85757 /rc.newwanip: rc.newwanip: on (IP address: 192.168.12.2) (interface: OPT1_VPN[opt1]) (real interface: ovpnc1).
Feb 7 15:58:27 php-fpm 85757 /rc.newwanip: Gateway, none 'available' for inet6, use the first one configured. ''
Feb 7 15:58:27 php-fpm 85757 /rc.newwanip: IP Address has changed, killing states on former IP Address 192.168.12.5.
Feb 7 15:58:29 dhcpleases 20659 Could not deliver signal HUP to process 99067: No such process.
Feb 7 15:58:31 php-fpm 85757 /rc.newwanip: Creating rrd update script
Feb 7 15:58:33 php-fpm 85757 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.12.5 -> 192.168.12.2 - Restarting packages.
Feb 7 15:58:33 check_reload_status 377 Starting packages
Feb 7 15:58:34 php-fpm 339 /rc.start_packages: Restarting/Starting all packages.
Feb 7 15:58:34 check_reload_status 377 Syncing firewall
Feb 7 15:58:40 check_reload_status 377 Reloading filter

Even if I set the file like this, it seems to reliably resolve the issue, but there's a dpinger error in the logs:

/usr/bin/touch /tmp/${1}up
# reload filter
/usr/local/sbin/pfSctl -c "interface newip ${1}"
/etc/rc.filter_configure
exit 0

Feb 7 16:03:51 check_reload_status 377 Syncing firewall
Feb 7 16:03:51 php-fpm 338 OpenVPN PID written: 76411
Feb 7 16:03:51 check_reload_status 377 Reloading filter
Feb 7 16:03:52 kernel ovpnc1: link state changed to UP
Feb 7 16:03:52 check_reload_status 377 rc.newwanip starting ovpnc1
Feb 7 16:03:53 check_reload_status 377 Reloading filter
Feb 7 16:03:53 php-fpm 339 /rc.newwanip: rc.newwanip: Info: starting on ovpnc1.
Feb 7 16:03:53 php-fpm 339 /rc.newwanip: rc.newwanip: on (IP address: 192.168.12.3) (interface: OPT1_VPN[opt1]) (real interface: ovpnc1).
Feb 7 16:04:00 php-fpm 85757 /rc.filter_configure_sync: dpinger: No dpinger session running for gateway OPT1_VPN_VPNV4
Feb 7 16:04:00 php-fpm 339 /rc.newwanip: Gateway, none 'available' for inet6, use the first one configured. ''
Feb 7 16:04:00 php-fpm 339 /rc.newwanip: IP Address has changed, killing states on former IP Address 192.168.12.2.
Feb 7 16:04:03 dhcpleases 75280 Could not deliver signal HUP to process 42317: No such process.
Feb 7 16:04:05 php-fpm 339 /rc.newwanip: Creating rrd update script
Feb 7 16:04:07 php-fpm 339 /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.12.2 -> 192.168.12.3 - Restarting packages.
Feb 7 16:04:07 check_reload_status 377 Starting packages
Feb 7 16:04:08 php-fpm 338 /rc.start_packages: Restarting/Starting all packages.
Feb 7 16:04:08 check_reload_status 377 Syncing firewall

If I reload at the beginning of that file, it happens before "rc.newwanip starting ovpnc1", and it doesn't resolve the issue.

It seems like a buggy order of operations with regard to when PFSense does the filter reload. I wonder if this is the case in other areas where the filter reloads at the wrong time.

@jimp Should I/we open a bug report, or am I too uninformed on the issue at hand? Please let me know how I can help get this corrected for 2.6.0 if it is a bug.

darcey

@jon8rfc
It happened to me quite often. Following boot, policy based route was absent, a filter reload once the vpn came up successfully created it. I used to check pfctl -vv -sr | grep route.

Sorry, but I cannot remember when it ceased to become an issue. Possibly following the upgrade to 2.5. But I haven't encountered the issue for a long while now and I am not using any modified scripts.

jazzl0ver

still happening with 2.7.0.
we have a couple of openvpn client sessions established from the pfsense. sometimes the tunnels restart but the traffic stops passing (actually it's directed to the uplink interface, not to the ovpnc). filter reloading fixes the issue.