Routing issue with Multi-Path VPN after upgrading to pfSense 2.8.0
-
Hi there,
I recently upgraded my pfSense 2.7.2 to 2.8.0.
All seems fine, except for some problems with multipath BGP routing over VPN using OpenVPN site-to-site.The problem seems to be due to the change in Firewall State Policy mode that occurred with the release change (and also documented in the release notes).
However, with the same setup, the problem only occurs on OpenVPN tunnels, not on IPSec (probably because of the option "Disable state-policy override for IPsec rules" ...? )I explain my setup and the issue in detail.
Config
Site A - Configuration
Dual WAN (2x LTE connections)
LAN: 172.20.0.0/25
Tunnel 1 (WAN 1): 172.19.101.2/30
Tunnel 2 (WAN 2): 172.19.102.2/30
FRR BGP with multipath enabled
OpenVPN Client roleSite A - Routing Table:
Destination Gateway Flags Netif Expire
default 192.168.8.1 UGS ix1
172.19.101.0/30 link#13 U ovpnc1
172.19.102.0/30 link#14 U ovpnc2
172.20.0.0/25 link#1 U ix0
172.30.0.0/25 172.19.102.1 UG1 ovpnc2
172.30.0.0/25 172.19.101.1 UG1 ovpnc1
...Site B - Configuration
Single WAN (FTTH)
LAN: 172.30.0.0/25
Tunnel 1: 172.19.101.1/30
Tunnel 2: 172.19.102.1/30
BGP with multipath enabled
OpenVPN Server roleSite B - Routing Table:
Destination Gateway Flags Netif Expire
0.0.0.0 192.168.1.254 UGS igb4
172.19.101.0/30 link#15 U ovpns2
172.19.102.0/30 link#16 U ovpns4
172.20.0.0/25 172.19.101.2 UG1 ovpns2
172.20.0.0/25 172.19.102.2 UG1 ovpns4
...Problem Description
After upgrading from pfSense 2.7.2 to 2.8.0, inter-site connectivity became extremely unstable (almost always broken).
The issue manifests as:- SSH connections hang after initial handshake - Connections appear to establish but then freeze
- Firewall logs show a lot of blocked packets (marked with the arrow and "direction is out") on the site that is receiving the connection
- State table shows inconsistent states across interfaces
Root Cause
The issue appears to be caused by asymmetric routing where:
- Outbound traffic uses one VPN tunnel (selected by BGP)
- Return traffic uses a different tunnel (also selected by BGP independently)
- pfSense 2.8.0's stricter state tracking sees this as invalid traffic
Validation
When disabling either VPN tunnel (leaving only one active), the problem completely disappears, confirming this is an asymmetric routing issue.
Attempted Solutions
- Verified firewall rules - All necessary allow rules are in place
- Checked VPN tunnel stability - Both tunnels are stable (confirmed also with iperf3)
- Analyzed routing tables - BGP multipath is working as expected
- Changing the State Policy to "Floating" nor State type to Sloopy, on a per-rule basis, seems to have no effect
Workaround found
Changing the Firewall State Policy (System > Advanced > Firewall & NAT) from "Interface Bound States" to "Floating States" completely resolves the issue.
Additional Findings
OpenVPN vs IPSec VTI
This issue appears to be specific to OpenVPN tunnels.
I have another site-to-site connection with identical BGP multipath configuration using IPSec VTI, and it works perfectly without any issues in pfSense 2.8.0.Could it be that only traffic on IPSec tunnels is managed differently?
In particular, as indicated at the beginning, I noticed that there is the option "Disable state-policy override for IPsec rules" (When Firewall State Policy is set to Interface Bound States, unchecking this option allows IPsec rules to automatically use floating states where needed. This option is ignored when IPsec Filter Mode is set to assigned interfaces.).
Obviously in my case it is not selected (default option)Questo potrebbe spiegare la differenza di comportamento tra IPSec ed OpenVPN.
-
I add a consideration.
Currently the workaround of changing the global Firewall State Policy has worked, but can it have negative impacts on other things?
It would be nice to have an overall solution to the problem on OpenVPN, maybe with an option like it was done for IPSec.
Bye,
Edo