Asymmetric routing with multi WAN and OpenVPN
-
@michmoor
yes, for years. We upgraded to take advantage of renewing certificates and have had problems when the default connection drops for maintenance or an outage.I noticed this line in the OpenVPN log today:
ROUTE_GATEWAY xxx.xxx.xxx.xxx/255.255.255.xxx IFACE=igb1 HWADDR=yy:yy:yy:yy:yy:yyWhere xxx.xxx.xxx.xxx is the Default Gateway Public address which is not the interface the OpenVPN server is tied to.
I believe this was just after a reboot, but I did a bunch of testing and didn't record that detail.
I am trying to find it in the config through SSH but haven't located it yet.
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
I believe we are experiencing a similar issue with routing OpenVPN. Does it work if your default gateway is WAN2?
Yes! If my Tier 1 WAN goes down and switches to Tier 2 WAN everything is fine again.
-
@mrsunfire
Did you state special MTU settings in the OpenVPN client config?@jc2it
Ensure that the rule, which is passing the OpenVPN connection is defined on the interface tab.
There must no floating pass rule and no one on an interface group be applied to the incoming OpenVPN packets. -
@viragomann said:
Ensure that the rule, which is passing the OpenVPN connection is defined on the interface tab.
It is.
@viragomann said:
There must no floating pass rule
There is none.
@viragomann said:
no one on an interface group be applied to the incoming OpenVPN packets
I am not exactly sure how this would be accomplished but I have no Interface Groups.
-
@jc2it @viragomann @mrsunfire @mrsunfire
Where is the OpenVPN command that generates this entry in the OpenVPN log?
ROUTE_GATEWAY yyy.zzz.xxx.xxx/255.255.255.qqq IFACE=igb1 HWADDR=a1:a2:a3:a4:a5:a6 -
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
Ensure that the rule, which is passing the OpenVPN connection is defined on the interface tab.
It is.
What is the advanced option setting in this rule?
It's also required that there is a gateway assigned to the concerned interface in the interface settings if the IP configuration is manually stated.
-
@viragomann said:
What is the advanced option setting in this rule?
It's also required that there is a gateway assigned to the concerned interface in the interface settings if the IP configuration is manually stated.It is the same Gateway that is assigned to the interface that the rule applies to and is assigned to the OpenVPN Server Config.
Despite that the OpenVPN log recorded the ROUTE_GATEWAY command pointing the the Default Gateway.
-
@jc2it
It makes no sense to state a gateway in a rule for passing incoming connections.The gateway option is meant for policy routing traffic a specific gateway, normally used for upstream connections.
-
@viragomann said:
It makes no sense to state a gateway in a rule for passing incoming connections.
Hmmm, Yes, this was probably set in relation to reading the Multi-WAN section of the implementation guide. I think that refers to LAN rules however. Regardless by stating the specific gateway on a rule associated with an interface it is merely redundant. Or stating what should be the default. Logically, it should not cause the default gateway to be used instead.
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
Regardless by stating the specific gateway on a rule associated with an interface it is merely redundant.
What do you think is redundant here?
Stating a gateway in a pass rule forces any matching packet to this gateway. So this is quite nonsense for incoming packets to me.
And I'm pretty surprised that your VPN server even works with that.Logically, it should not cause the default gateway to be used instead.
I cannot imagine, what it effects exactly, but it want not have any advantage for sure. So I would set the gateway to "none" again.
-
@viragomann said:
What do you think is redundant here?
The default behavior is to have a NAT "reply to" associated with incoming rules. This should use the gateway that is associated with the interface the packet arrived in at. Explicitly specifying the gateway should remove any "confusion" as to which gateway to use.
@viragomann said:
I would set the gateway to "none" again.
There is no "none" option. There are only a list of gateways, one of which is the default gateway. Which is where we don't want the packets "reply-to" in our Multi-WAN environment.Just so you know when we were experiencing the issue yesterday. changing this policy-based rule setting to "default" had zero effect, so I changed it back to what made logical sense given the issue we were experiencing.
-
After writing my last reply I recalled something I saw in the firewall System log yesterday but didn't find any answers to. Perhaps this is related to the issue where the OpenVPN packets are leaving from the default gateway instead of the gateway they arrived from.
Dec 8 14:38:25 php-fpm 50688 /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500@mrsunfire Do you have this message in your "Status/System Logs/System/General"
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
The default behavior is to have a NAT "reply to" associated with incoming rules. This should use the gateway that is associated with the interface the packet arrived in at. Explicitly specifying the gateway should remove any "confusion" as to which gateway to use.
The rule won't affect any respond packets at all, but incoming packets only. Responses are treated automatically by pfSense.
And direct any incoming packets back the the gateway might not be what you want. -
@viragomann said:
And direct any incoming packets back the the gateway might not be what you want.
You make a good point. However, I definitely don't want them leaving via the default gateway or any other gateway in the list, which is the other option. Perhaps recreating the rule without the option, is the way to change it. Setting it to "default" didn't work yesterday.
-
@jc2it
The requirements for passing responses back out to the interface, where the requests were coming in are- a gateway must be assigned to the interface. Verify Status > Interfaces
- the gateway state must be "online". Verify Status > Gateways
- the pass rule, which allows the incoming access must be defined on the interface, because only such rules add the reply-to tag to the connection.
Consider that floating and interface group rules are probed first.
If all these are given it should work.
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
After writing my last reply I recalled something I saw in the firewall System log yesterday but didn't find any answers to. Perhaps this is related to the issue where the OpenVPN packets are leaving from the default gateway instead of the gateway they arrived from.
Dec 8 14:38:25 php-fpm 50688 /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500@mrsunfire Do you have this message in your "Status/System Logs/System/General"
I found the source of this message. Back on November 4th we had changed a NAT rule from Pure NAT to NAT+Proxy because FTP stopped working correctly. That issue is probably caused by the same thing that causes the issue with the OpenVPN routing problem. After changing it back to PureNAT I no longer get the message when the filter reloads.
-
@viragomann said in Asymmetric routing with multi WAN and OpenVPN:
@mrsunfire
Did you state special MTU settings in the OpenVPN client config?Not really. MTU 1500 (default) works fine for that connection (DOCSIS).
-
We have verified those are all functioning and were functioning in our system yesterday. However overnight, early morning on the 8th, we had several severe drops to the WAN that is configured as the system default gateway. This resulted in a change to the firewall that "moved"/ "added" a ROUTE_GATEWAY entry to the OpenVPN log. It runs at service restart time and sets the OpenVPN service to the default gateway. The fix was to change the system default gateway to the other WAN and then adjust outbound LAN rules for the services using the previous WAN.
We did this to restore service after testing/troubleshooting for several hours yesterday morning.
Where can I find the configuration entry causing the ROUTE_GATEWAY command to run when OpenVPN restarts?
-
@jc2it said in Asymmetric routing with multi WAN and OpenVPN:
Dec 8 14:38:25 php-fpm 50688 /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500@mrsunfire Do you have this message in your "Status/System Logs/System/General"
No.