Weird asymetric routing issue [solved]

Myster_fr · Nov 3, 2020, 1:52 PM

Hello,

Here's my setup:
pfsense box, a single WAN connetion from my ISP.
pfsense box is also an OpenVPN client for private browsing.
pfsense v2.4.5-RELEASE-p1

My VPN provider assigns me a public IP to my client, not a private IP that gets NATed later on.

+----------------------+    |                     |        |                                |  +-------------------+
|                      |    |                     +--------+                                |  |                   |
|  Test source machine |    |                     | TUNNEL |   opvnc3 - 100.100.100.198     |  |  internal machine |
|  (245.245.245.130)   +----+      INTERNET       +--------+                               +---+  (10.25.0.110)    |
|                      |    |                     |        |                                |  |                   |
|                      |    |                     +--------+   WAN - 134.156.134.156        |  +-------------------+
+----------------------+    |                     |        |                                |
                            +---------------------+        +--------------------------------+

I followed the various how-to to setup multi-wan, and made some configuration to have one of my machines exclusively use the OpenVPN tunnel (and never uses direct WAN).
To do so I did the following :

defined an outbound NAT statement to NAT everything going out through the VPN interface
set a FW rule on my LAN interface to tell traffic from that specific machine should be tagged with "NO_WAN_EGRESS" tag, and use my OpenVPN client interface's gateway
set a floating FW rule preventing packets tagged with "NO_WAN_EGRESS" to use the WAN interface

What works:
Now, as expected, traffic from this machine (understand : initiated by this machine) can only go out through the VPN interface, goal achieved.
Succesfully tested by browsing some websites, confirmed by checking public IP visible to those websites.

What doesn't work :
I also configured some port forward on the VPN interface, to be forwarded to that very same machine.
When I try to remotely connect through that forwarded port, I see packets reach my internal machine (tcpdump).
I also see my machine answer back.

However, on the pfsense box, I see the following (tcpdump in SSH shell):
1/ I see incoming packets on the ovpnc3 interface (my VPN client interface)
2/ I do not see any outgoing reply packets on the ovpnc3 interface
3/ however, I see the outgoing reply packets on my WAN interface, using the ovpnc3 interface public IP address. WTH ??

I can't figure out what triggers this behavior.
I've checked my routing, my gateway settings, I can't explain that

The fact the outgoing packets show the ovpnc3 IP address as source IP seems to show my packet tagging rules work.
But what makes pfSense actually send out those reply packets through the WAN interface instead ?
What is ever weirder, is that traffic generated by my LAN machine itself uses the ovpnc3 interface without a problem.
Only reply traffic is routed weirdly.

Any clues what could generate this ?
I have another pfSense elsewhere, with a similar setup, and don't have this issue at all.

Here are some captures (IP addresses obfuscated).

Interfaces & IP addresses (fake)

ovpnc3 - 100.100.100.198 : pfsense's OpenVPN client interface public IP
245.245.245.130 : external machine i'm testing from
10.25.0.110 : internal machine, behind pfSense
em0.2 is my WAN interface on pfSense box

INTERNAL machine : shows both inbound and outbound packets are received and sent.

✓ internal_machine ~ $ tcpdump -i ens18 host 245.245.245.130
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), capture size 262144 bytes
12:03:50.465892 IP 245.245.245.130.45603 > 10.25.0.110.62570: Flags [S], seq 759656872, win 1024, options [mss 1288], length 0
12:03:50.465914 IP 10.25.0.110.62570 > 245.245.245.130.45603: Flags [S.], seq 1162659305, ack 759656873, win 64240, options [mss 1460], length 0
12:03:51.467556 IP 245.245.245.130.45604 > 10.25.0.110.62570: Flags [S], seq 759591337, win 1024, options [mss 1288], length 0
12:03:51.467586 IP 10.25.0.110.62570 > 245.245.245.130.45604: Flags [S.], seq 2506716640, ack 759591338, win 64240, options [mss 1460], length 0

pfSense box, capture on OpenVPN client interface (ovpnc3).
Shows incoming packets fine, but there's no trace of the replies

[2.4.5-RELEASE][admin@pfSense.universe.local]/root: tcpdump -i ovpnc3 host 245.245.245.130
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ovpnc3, link-type NULL (BSD loopback), capture size 262144 bytes
12:06:21.594826 IP 245.245.245.130.61091 > 100.100.100.198.62570: Flags [S], seq 234691584, win 1024, options [mss 1288], length 0
12:06:22.596879 IP 245.245.245.130.61092 > 100.100.100.198.62570: Flags [S], seq 234626049, win 1024, options [mss 1288], length 0

pfSense box, capture on the WAN interface.
Doesn't show the inbound packets (expected), but shows the replies being sent over the WAN interface, with the opvnc's IP address as a source ??

[2.4.5-RELEASE][admin@pfSense.universe.local]/root: tcpdump -i em0.2 host 245.245.245.130 and port 62570
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0.2, link-type EN10MB (Ethernet), capture size 262144 bytes
12:08:18.615407 IP 100.100.100.198.62570 > 245.245.245.130.54824: Flags [S.], seq 200831507, ack 2417907882, win 64240, options [mss 1460], length 0
12:08:19.616152 IP 100.100.100.198.62570 > 245.245.245.130.54824: Flags [S.], seq 200831507, ack 2417907882, win 64240, options [mss 1460], length 0
12:08:19.616434 IP 100.100.100.198.62570 > 245.245.245.130.54825: Flags [S.], seq 1604522520, ack 2417973417, win 64240, options [mss 1460], length 0

viragomann · Nov 1, 2020, 8:10 PM

@Myster_fr
Assign an interface to ovpnc3 and enable it. You will get a new firewall rule tab for this interface. Move all firewall rules allowing incoming traffic from this VPN to this new tab. Ensure that there is no rule on the OpenVPN or on floating tab matching the permitted traffic.

Myster_fr · Nov 2, 2020, 9:24 AM

Thanks Viragoman,

This is all already done.
There's an interface defined, the correct rules are in place.

Let me summarize again :

traffic generated by the internal machine goes over VPN as expected
when traffic is generated from outside to the VPN IP address, they correctly hit the internal machine, BUT the replies from the internal machine go out through the WAN interface AND show the VPN interface source address, which is completely wrong.

If the replies had gone through WAN and show WAN IP, I'd agree that something's wrong with my rules, or my NAT.
But they're going out with the VPN IP address, on the WAN interface, which doesn't make sense.
It would mean that pfSense makes a routing decision AFTER NAT has been applied, and does not respect connection state to route the traffic back over the VPN_SE interface (this is all TCP).

An again, traffic generated by the internal machine goes out through the VPN interface.
Only replies show this weird behavior.

Here are some screenshots :
VPN_SE is my ovpnc3 interface.

Here's my floating rule :

FW rule on the LAN interface

Finally, my NAT statements:

CaptainPistache · Nov 2, 2020, 11:24 AM

Hi,

Is there a gateway set on your LAN interface ?
Can you reproduce that issue with a different "Test source machine" ?
Could you post your Routing table ?
Is there any other NAT rules except the ones on the "Outbound" tab you've published and the Port forward to 10.25.0.110 ?

Captain

viragomann · Nov 2, 2020, 12:07 PM

@Myster_fr
I wrote two simple lines. Instead of responsing to that you answer a lot of other stuff.

@Myster_fr said in Weird asymetric routing issue:

Thanks Viragoman,

This is all already done.
There's an interface defined, the correct rules are in place.

Let me summarize again :

traffic generated by the internal machine goes over VPN as expected

when traffic is generated from outside to the VPN IP address, they correctly hit the internal machine, BUT the replies from the internal machine go out through the WAN interface AND show the VPN interface source address, which is completely wrong.

If the replies had gone through WAN and show WAN IP, I'd agree that something's wrong with my rules, or my NAT.
But they're going out with the VPN IP address, on the WAN interface, which doesn't make sense.
It would mean that pfSense makes a routing decision AFTER NAT has been applied, and does not respect connection state to route the traffic back over the VPN_SE interface (this is all TCP).

An again, traffic generated by the internal machine goes out through the VPN interface.
Only replies show this weird behavior.

That is the expected behavior in a multi-WAN setup if you don't obey what I wrote above.

So please post the rules from the OpenVPN tab.

Myster_fr · Nov 2, 2020, 2:30 PM

@viragoman, I answered your three point in the first two lines of my answer, but let me start over
1/ the ovpnc3 is already created as "VPN_SE" interface
2/ the rules allowing incoming traffic through my VPN IP address are set on the "VPN_SE" interface, and visible in the first screenshot of my "too long answer" . It's the first screenshot for that matter. Rule content below.
3/ The floating rules are visible in the second screenshot : only one rule blocking tagged traffic going out through the WAN interface. Rule content below

It seems like I'm unable to express my issue correctly.
When the traffic is initiated by the internal machine, all is routed as expected through the VPN_SE interface.
It's only replies to externally initiated traffic that get routed weirdly.

RULE CONTENT - VPN_SE interface/port forwarding

RULE CONTENT - LAN - Packet tagging

RULE CONTENT - Floating rule
(In this floating rule, "WAN" interface is the only one selected in the list)

@CaptainPistache :
1/ there's no gateway on the LAN interface : pfSense is the gateway for LAN
2/ Issue is reproductible from any source machine (also tried port testing services like portchecker.co
3/ Routing table looks normal (sorry, I don't see the point of sharing it at the moment). Routing should not be involved in this situation. What is expected: traffic generated by the internal machine is tagged "NO_WAN_EGRESS" and the FW rule tells to use the "VPN_SE_VPN4" gateway, bypassing all system rules. Which again, works fine when the traffic is generated by the internal machine.
4/ There are inbound NAT rules to establish port forwarding from VPN_SE interface to the internal machine. Works fine as the tcpdump clearly show incoming connection attempts reach the internal machine.

viragomann · Nov 2, 2020, 2:49 PM

@Myster_fr said in Weird asymetric routing issue:

It seems like I'm unable to express my issue correctly.
When the traffic is initiated by the internal machine, all is routed as expected through the VPN_SE interface.
It's only replies to externally initiated traffic that get routed weirdly.

Again: That is exactly what I expect.

You're still not responding to my question. So there is nothing I can do for you. Sorry.

CaptainPistache · Nov 2, 2020, 3:00 PM

@Myster_fr

That may be a dumb question but just out of curiosity, why do you bother with more complex firewall rules instead of just a static route to redirect all traffic from your internal machine through your OpenVPN interface ?

Myster_fr · Nov 2, 2020, 3:13 PM

@viragomann Ok sir, sorry, then I must not have understood your question correctly.

What do you need from me that I missed providing ?

Myster_fr · Nov 2, 2020, 3:21 PM

@CaptainPistache here's a bit of an explanation :

You are missing some basic concepts in routing.
A route only targets a destination, not a source.

So a simple route isn't sufficient, as I don't have specific external hosts I need to restrict traffic to.
I want one machine to always go through the VPN, and if the VPN is down, then this machine doesn't have any external connectivity at all.
I don't ever want this machine to use WAN and directly connect to internet, under any circumstance.

In order to do so, you need to use FW rules and set a specific gateway (my VPN connection gateway) for traffic emitted by my LAN machine.

In order to prevent this machine's traffic to use any other gateway (namely WAN) you need to push the button a bit further :
1/ Tag the traffic
2/ Create a floating rule blocking that tag on the WAN interface.

There are plenty of howtos to achieve this, and the setup is quite straightforward.
I even have it perfectly working on a different setup.
I did side by side comparisons between the rules, tags, etc of the working one, and this one, without finding what's wrong.

I guess something other than a rule takes precedence at some point, and I'm trying to get some inputs from other users ;).

In the end, I might consider starting over from a blank configuration...

viragomann · Nov 2, 2020, 3:23 PM

@Myster_fr

@viragomann said in Weird asymetric routing issue:

So please post the rules from the OpenVPN tab.

Myster_fr · Nov 2, 2020, 3:26 PM

@viragomann This ?

FYI, the pfSense box is also an OpenVPN server, and thought this tab refers to the locally running pfSense server not the client part.

viragomann · Nov 2, 2020, 3:35 PM

@Myster_fr said in Weird asymetric routing issue:

@viragomann This ?

FYI, the pfSense box is also an OpenVPN server, and thought this tab refers to the locally running pfSense server not the client part.

Exactly.
And now read again, what I wrote in my first post:

@viragomann said in Weird asymetric routing issue:

Ensure that there is no rule on the OpenVPN or on floating tab matching the permitted traffic.

That's why it isn't working.

So to solve, either edit the OpenVPN rule so that it isn't applied to the desired traffic from the client instance or also assign interfaces to the other OpenVPN instances and move the rule to these tabs and remove it from OpenVPN.

Myster_fr · Nov 2, 2020, 3:46 PM

Holly Molly ! Thanks @viragomann !!

I've disabled the rule in the OpenVPN tab, and it started working, must now check if it had side effects, but at least it's a step forward.

Sorry for having misread your guidance, when you were talking about "OpenVPN tab" I understood "the tab of the VPN interface", instead of the actual OpenVPN tab.

Now, if I may ask for a few more minutes of your time, could you elaborate on this OpenVPN tab concept ?
What does it cover/not cover ?
When does it supersede (or not) the other OpenVPN-related interfaces rules ?

I was under the wrong impression that setting an "allow any/any" would simply always allow all traffic over the tunnels, how come in this particular setup it did the opposite ?

CaptainPistache · Nov 2, 2020, 4:04 PM

@Myster_fr said in Weird asymetric routing issue:

@CaptainPistache here's a bit of an explanation :

You are missing some basic concepts in routing.
A route only targets a destination, not a source.

So a simple route isn't sufficient, as I don't have specific external hosts I need to restrict traffic to.
I want one machine to always go through the VPN, and if the VPN is down, then this machine doesn't have any external connectivity at all.
I don't ever want this machine to use WAN and directly connect to internet, under any circumstance.

In order to do so, you need to use FW rules and set a specific gateway (my VPN connection gateway) for traffic emitted by my LAN machine.

In order to prevent this machine's traffic to use any other gateway (namely WAN) you need to push the button a bit further :
1/ Tag the traffic
2/ Create a floating rule blocking that tag on the WAN interface.

There are plenty of howtos to achieve this, and the setup is quite straightforward.
I even have it perfectly working on a different setup.
I did side by side comparisons between the rules, tags, etc of the working one, and this one, without finding what's wrong.

I guess something other than a rule takes precedence at some point, and I'm trying to get some inputs from other users ;).

In the end, I might consider starting over from a blank configuration...

I knew that was a dumb question ;)
But thanks to you, I've learned something new today.

@Myster_fr said in Weird asymetric routing issue:

Now, if I may ask for a few more minutes of your time, could you elaborate on this OpenVPN tab concept ?
What does it cover/not cover ?
When does it supersede (or not) the other OpenVPN-related interfaces rules ?

I was under the wrong impression that setting an "allow any/any" would simply always allow all traffic over the tunnels, how come in this particular setup it did the opposite ?

I think the answer to your question is here.

Specifically :

"Rules added here are processed after the OpenVPN tab rules, which are checked first. In order to match the rules on an assigned VPN tab, the traffic must not match any rules on the OpenVPN tab. Remove any “Allow All” style rules from the OpenVPN tab and craft more specific rules instead."

Myster_fr · Nov 2, 2020, 4:07 PM

Haha, thanks @CaptainPistache ! and there are no dumb questions !

That's usually the good thing with forums, we always learn from each other !

viragomann · Nov 2, 2020, 4:32 PM

@Myster_fr said in Weird asymetric routing issue:

Now, if I may ask for a few more minutes of your time, could you elaborate on this OpenVPN tab concept ?
What does it cover/not cover ?
When does it supersede (or not) the other OpenVPN-related interfaces rules ?

The OpenVPN tab is an implicit interface group containing all OpenVPN instances (servers and clients) running on pfSense.
Hence rules you define here are applied to all OpenVPN instances.

However, what you need to get the responses back to the correct OpenVPN gateway is the reply-to function of pfSense. That is a tag pfSense puts on packets which are entering an interface which have a gateway assigned to (Multi-WAN. The OpenVPN is as well a kind of WAN here.) The tag is assigned by the firewall rule which allows the particular traffic on the respective interface tab.
But that cannot work on interface groups or floating rules, because these rules may be cover multiple interfaces and so there is no distinct gateway for the reply-to tag.

Therefor I pointed that to consider in my first post.

Myster_fr · Nov 2, 2020, 4:25 PM

@viragomann Awesome, thanks again !