Weird asymetric routing issue [solved]
-
Hi,
-
Is there a gateway set on your LAN interface ?
-
Can you reproduce that issue with a different "Test source machine" ?
-
Could you post your Routing table ?
-
Is there any other NAT rules except the ones on the "Outbound" tab you've published and the Port forward to 10.25.0.110 ?
Captain
-
-
@Myster_fr
I wrote two simple lines. Instead of responsing to that you answer a lot of other stuff.@Myster_fr said in Weird asymetric routing issue:
Thanks Viragoman,
This is all already done.
There's an interface defined, the correct rules are in place.Let me summarize again :
- traffic generated by the internal machine goes over VPN as expected
- when traffic is generated from outside to the VPN IP address, they correctly hit the internal machine, BUT the replies from the internal machine go out through the WAN interface AND show the VPN interface source address, which is completely wrong.
If the replies had gone through WAN and show WAN IP, I'd agree that something's wrong with my rules, or my NAT.
But they're going out with the VPN IP address, on the WAN interface, which doesn't make sense.
It would mean that pfSense makes a routing decision AFTER NAT has been applied, and does not respect connection state to route the traffic back over the VPN_SE interface (this is all TCP).An again, traffic generated by the internal machine goes out through the VPN interface.
Only replies show this weird behavior.That is the expected behavior in a multi-WAN setup if you don't obey what I wrote above.
So please post the rules from the OpenVPN tab.
-
@viragoman, I answered your three point in the first two lines of my answer, but let me start over
1/ the ovpnc3 is already created as "VPN_SE" interface
2/ the rules allowing incoming traffic through my VPN IP address are set on the "VPN_SE" interface, and visible in the first screenshot of my "too long answer"
. It's the first screenshot for that matter. Rule content below.
3/ The floating rules are visible in the second screenshot : only one rule blocking tagged traffic going out through the WAN interface. Rule content belowIt seems like I'm unable to express my issue correctly.
When the traffic is initiated by the internal machine, all is routed as expected through the VPN_SE interface.
It's only replies to externally initiated traffic that get routed weirdly.RULE CONTENT - VPN_SE interface/port forwarding
RULE CONTENT - LAN - Packet tagging
RULE CONTENT - Floating rule
(In this floating rule, "WAN" interface is the only one selected in the list)
@CaptainPistache :
1/ there's no gateway on the LAN interface : pfSense is the gateway for LAN
2/ Issue is reproductible from any source machine (also tried port testing services like portchecker.co
3/ Routing table looks normal (sorry, I don't see the point of sharing it at the moment). Routing should not be involved in this situation. What is expected: traffic generated by the internal machine is tagged "NO_WAN_EGRESS" and the FW rule tells to use the "VPN_SE_VPN4" gateway, bypassing all system rules. Which again, works fine when the traffic is generated by the internal machine.
4/ There are inbound NAT rules to establish port forwarding from VPN_SE interface to the internal machine. Works fine as the tcpdump clearly show incoming connection attempts reach the internal machine. -
@Myster_fr said in Weird asymetric routing issue:
It seems like I'm unable to express my issue correctly.
When the traffic is initiated by the internal machine, all is routed as expected through the VPN_SE interface.
It's only replies to externally initiated traffic that get routed weirdly.Again: That is exactly what I expect.
You're still not responding to my question. So there is nothing I can do for you. Sorry.
-
That may be a dumb question but just out of curiosity, why do you bother with more complex firewall rules instead of just a static route to redirect all traffic from your internal machine through your OpenVPN interface ?
-
@viragomann Ok sir, sorry, then I must not have understood your question correctly.
What do you need from me that I missed providing ?
-
@CaptainPistache here's a bit of an explanation :
You are missing some basic concepts in routing.
A route only targets a destination, not a source.So a simple route isn't sufficient, as I don't have specific external hosts I need to restrict traffic to.
I want one machine to always go through the VPN, and if the VPN is down, then this machine doesn't have any external connectivity at all.
I don't ever want this machine to use WAN and directly connect to internet, under any circumstance.In order to do so, you need to use FW rules and set a specific gateway (my VPN connection gateway) for traffic emitted by my LAN machine.
In order to prevent this machine's traffic to use any other gateway (namely WAN) you need to push the button a bit further :
1/ Tag the traffic
2/ Create a floating rule blocking that tag on the WAN interface.There are plenty of howtos to achieve this, and the setup is quite straightforward.
I even have it perfectly working on a different setup.
I did side by side comparisons between the rules, tags, etc of the working one, and this one, without finding what's wrong.I guess something other than a rule takes precedence at some point, and I'm trying to get some inputs from other users ;).
In the end, I might consider starting over from a blank configuration...
-
@viragomann said in Weird asymetric routing issue:
So please post the rules from the OpenVPN tab.
-
@viragomann This ?
FYI, the pfSense box is also an OpenVPN server, and thought this tab refers to the locally running pfSense server not the client part.
-
@Myster_fr said in Weird asymetric routing issue:
@viragomann This ?
FYI, the pfSense box is also an OpenVPN server, and thought this tab refers to the locally running pfSense server not the client part.
Exactly.
And now read again, what I wrote in my first post:@viragomann said in Weird asymetric routing issue:
Ensure that there is no rule on the OpenVPN or on floating tab matching the permitted traffic.
That's why it isn't working.
So to solve, either edit the OpenVPN rule so that it isn't applied to the desired traffic from the client instance or also assign interfaces to the other OpenVPN instances and move the rule to these tabs and remove it from OpenVPN.
-
Holly Molly ! Thanks @viragomann !!
I've disabled the rule in the OpenVPN tab, and it started working, must now check if it had side effects, but at least it's a step forward.
Sorry for having misread your guidance, when you were talking about "OpenVPN tab" I understood "the tab of the VPN interface", instead of the actual OpenVPN tab.
Now, if I may ask for a few more minutes of your time, could you elaborate on this OpenVPN tab concept ?
What does it cover/not cover ?
When does it supersede (or not) the other OpenVPN-related interfaces rules ?I was under the wrong impression that setting an "allow any/any" would simply always allow all traffic over the tunnels, how come in this particular setup it did the opposite ?
-
@Myster_fr said in Weird asymetric routing issue:
@CaptainPistache here's a bit of an explanation :
You are missing some basic concepts in routing.
A route only targets a destination, not a source.So a simple route isn't sufficient, as I don't have specific external hosts I need to restrict traffic to.
I want one machine to always go through the VPN, and if the VPN is down, then this machine doesn't have any external connectivity at all.
I don't ever want this machine to use WAN and directly connect to internet, under any circumstance.In order to do so, you need to use FW rules and set a specific gateway (my VPN connection gateway) for traffic emitted by my LAN machine.
In order to prevent this machine's traffic to use any other gateway (namely WAN) you need to push the button a bit further :
1/ Tag the traffic
2/ Create a floating rule blocking that tag on the WAN interface.There are plenty of howtos to achieve this, and the setup is quite straightforward.
I even have it perfectly working on a different setup.
I did side by side comparisons between the rules, tags, etc of the working one, and this one, without finding what's wrong.I guess something other than a rule takes precedence at some point, and I'm trying to get some inputs from other users ;).
In the end, I might consider starting over from a blank configuration...
I knew that was a dumb question ;)
But thanks to you, I've learned something new today.@Myster_fr said in Weird asymetric routing issue:
Now, if I may ask for a few more minutes of your time, could you elaborate on this OpenVPN tab concept ?
What does it cover/not cover ?
When does it supersede (or not) the other OpenVPN-related interfaces rules ?I was under the wrong impression that setting an "allow any/any" would simply always allow all traffic over the tunnels, how come in this particular setup it did the opposite ?
I think the answer to your question is here.
Specifically :
"Rules added here are processed after the OpenVPN tab rules, which are checked first. In order to match the rules on an assigned VPN tab, the traffic must not match any rules on the OpenVPN tab. Remove any “Allow All” style rules from the OpenVPN tab and craft more specific rules instead."
-
Haha, thanks @CaptainPistache ! and there are no dumb questions !
That's usually the good thing with forums, we always learn from each other !
-
@Myster_fr said in Weird asymetric routing issue:
Now, if I may ask for a few more minutes of your time, could you elaborate on this OpenVPN tab concept ?
What does it cover/not cover ?
When does it supersede (or not) the other OpenVPN-related interfaces rules ?The OpenVPN tab is an implicit interface group containing all OpenVPN instances (servers and clients) running on pfSense.
Hence rules you define here are applied to all OpenVPN instances.However, what you need to get the responses back to the correct OpenVPN gateway is the reply-to function of pfSense. That is a tag pfSense puts on packets which are entering an interface which have a gateway assigned to (Multi-WAN. The OpenVPN is as well a kind of WAN here.) The tag is assigned by the firewall rule which allows the particular traffic on the respective interface tab.
But that cannot work on interface groups or floating rules, because these rules may be cover multiple interfaces and so there is no distinct gateway for the reply-to tag.Therefor I pointed that to consider in my first post.
-
@viragomann Awesome, thanks again !