Asymmetric routing with multi WAN and OpenVPN
-
I have 3 ISP multi WAN setup as failover in a gateway group.
I also have a OpenVPN client to VPN provider.
On LAN rules I have policy based routing active for some FQDN routed over that OpenVPN tunnel.
Everything works fine. But now I want the OpenVPN tunnel established via WAN2 (Tier 2). Connection is up, gateway is up. If I try to reach via ICMP a FQDN listed in the alias of the policy based rule I get an answer. If I try to reach it via TCP (HTTPS) the gateway gets massive packetloss and I can't get the webpage to open.
I think thats a asymmetric routing issue but I can't figure out where the problem is. I also did try the hints from Netgate docs and setup a floating rule outbound for TCP and that OpenVPN gateway set.
Any hints?
-
I believe we are experiencing a similar issue with routing OpenVPN. Does it work if your default gateway is WAN2?
We were able to make it function but I don't know what the trigger was in our environment. One day it worked on our secondaryWAN with the OpenVPN server set to use the secondaryWAN as the default. Then overnight we saw several drops over a few hours and OpenVPN stopped routing through the secondaryWAN and would only route through the primaryWAN. Even though the OpenVPN server is set to use the secondaryWAN interface. Unfortunately the users' OpenVPN config only connects through the secondaryWAN and so to get them up and running we switched the default gateway to the secondaryWAN. We are seeing this on 22.05-RELEASE after upgrading from 2.4.5-p1.
-
@jc2it
In my case I am wondering if it is related to relayd deprecation.
OR perhaps a regression?
https://redmine.pfsense.org/issues/11436 -
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
We are seeing this on 22.05-RELEASE after upgrading from 2.4.5-p1.
Was it working correctly on 2.4.5-p1 ?
-
@michmoor
yes, for years. We upgraded to take advantage of renewing certificates and have had problems when the default connection drops for maintenance or an outage.I noticed this line in the OpenVPN log today:
ROUTE_GATEWAY xxx.xxx.xxx.xxx/255.255.255.xxx IFACE=igb1 HWADDR=yy:yy:yy:yy:yy:yyWhere xxx.xxx.xxx.xxx is the Default Gateway Public address which is not the interface the OpenVPN server is tied to.
I believe this was just after a reboot, but I did a bunch of testing and didn't record that detail.
I am trying to find it in the config through SSH but haven't located it yet.
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
I believe we are experiencing a similar issue with routing OpenVPN. Does it work if your default gateway is WAN2?
Yes! If my Tier 1 WAN goes down and switches to Tier 2 WAN everything is fine again.
-
@mrsunfire
Did you state special MTU settings in the OpenVPN client config?@jc2it
Ensure that the rule, which is passing the OpenVPN connection is defined on the interface tab.
There must no floating pass rule and no one on an interface group be applied to the incoming OpenVPN packets. -
@viragomann said:
Ensure that the rule, which is passing the OpenVPN connection is defined on the interface tab.
It is.
@viragomann said:
There must no floating pass rule
There is none.
@viragomann said:
no one on an interface group be applied to the incoming OpenVPN packets
I am not exactly sure how this would be accomplished but I have no Interface Groups.
-
@jc2it @viragomann @mrsunfire @mrsunfire
Where is the OpenVPN command that generates this entry in the OpenVPN log?
ROUTE_GATEWAY yyy.zzz.xxx.xxx/255.255.255.qqq IFACE=igb1 HWADDR=a1:a2:a3:a4:a5:a6 -
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
Ensure that the rule, which is passing the OpenVPN connection is defined on the interface tab.
It is.
What is the advanced option setting in this rule?
It's also required that there is a gateway assigned to the concerned interface in the interface settings if the IP configuration is manually stated.
-
@viragomann said:
What is the advanced option setting in this rule?
It's also required that there is a gateway assigned to the concerned interface in the interface settings if the IP configuration is manually stated.It is the same Gateway that is assigned to the interface that the rule applies to and is assigned to the OpenVPN Server Config.
Despite that the OpenVPN log recorded the ROUTE_GATEWAY command pointing the the Default Gateway.
-
@jc2it
It makes no sense to state a gateway in a rule for passing incoming connections.The gateway option is meant for policy routing traffic a specific gateway, normally used for upstream connections.
-
@viragomann said:
It makes no sense to state a gateway in a rule for passing incoming connections.
Hmmm, Yes, this was probably set in relation to reading the Multi-WAN section of the implementation guide. I think that refers to LAN rules however. Regardless by stating the specific gateway on a rule associated with an interface it is merely redundant. Or stating what should be the default. Logically, it should not cause the default gateway to be used instead.
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
Regardless by stating the specific gateway on a rule associated with an interface it is merely redundant.
What do you think is redundant here?
Stating a gateway in a pass rule forces any matching packet to this gateway. So this is quite nonsense for incoming packets to me.
And I'm pretty surprised that your VPN server even works with that.Logically, it should not cause the default gateway to be used instead.
I cannot imagine, what it effects exactly, but it want not have any advantage for sure. So I would set the gateway to "none" again.
-
@viragomann said:
What do you think is redundant here?
The default behavior is to have a NAT "reply to" associated with incoming rules. This should use the gateway that is associated with the interface the packet arrived in at. Explicitly specifying the gateway should remove any "confusion" as to which gateway to use.
@viragomann said:
I would set the gateway to "none" again.
There is no "none" option. There are only a list of gateways, one of which is the default gateway. Which is where we don't want the packets "reply-to" in our Multi-WAN environment.Just so you know when we were experiencing the issue yesterday. changing this policy-based rule setting to "default" had zero effect, so I changed it back to what made logical sense given the issue we were experiencing.
-
After writing my last reply I recalled something I saw in the firewall System log yesterday but didn't find any answers to. Perhaps this is related to the issue where the OpenVPN packets are leaving from the default gateway instead of the gateway they arrived from.
Dec 8 14:38:25 php-fpm 50688 /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500@mrsunfire Do you have this message in your "Status/System Logs/System/General"
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
The default behavior is to have a NAT "reply to" associated with incoming rules. This should use the gateway that is associated with the interface the packet arrived in at. Explicitly specifying the gateway should remove any "confusion" as to which gateway to use.
The rule won't affect any respond packets at all, but incoming packets only. Responses are treated automatically by pfSense.
And direct any incoming packets back the the gateway might not be what you want. -
@viragomann said:
And direct any incoming packets back the the gateway might not be what you want.
You make a good point. However, I definitely don't want them leaving via the default gateway or any other gateway in the list, which is the other option. Perhaps recreating the rule without the option, is the way to change it. Setting it to "default" didn't work yesterday.
-
@jc2it
The requirements for passing responses back out to the interface, where the requests were coming in are- a gateway must be assigned to the interface. Verify Status > Interfaces
- the gateway state must be "online". Verify Status > Gateways
- the pass rule, which allows the incoming access must be defined on the interface, because only such rules add the reply-to tag to the connection.
Consider that floating and interface group rules are probed first.
If all these are given it should work.
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
After writing my last reply I recalled something I saw in the firewall System log yesterday but didn't find any answers to. Perhaps this is related to the issue where the OpenVPN packets are leaving from the default gateway instead of the gateway they arrived from.
Dec 8 14:38:25 php-fpm 50688 /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500@mrsunfire Do you have this message in your "Status/System Logs/System/General"
I found the source of this message. Back on November 4th we had changed a NAT rule from Pure NAT to NAT+Proxy because FTP stopped working correctly. That issue is probably caused by the same thing that causes the issue with the OpenVPN routing problem. After changing it back to PureNAT I no longer get the message when the filter reloads.