DNAT and tunnel interface: packet not forwarded to the expected interface

vlurk

pfSense 2.8.1 has the following interfaces:

igc0 : wan (DHCP)
igc1 : lan (static)
tun_wg0 : VPN (static), wireguard setup with VPN provider (not site-to-site!)

I have two gateways defined in system routing:

For the WAN (igc0), acquired automatically from DHCP client.
For the VPN (tun_wg0), the IP address of the tun_wg0 interface. 10.145.191.77/32

My setup is that I default to the WAN gateway. But, I have rules that attach the VPN gateway (dynamic routing) for some local (lan) hosts.
It works really well for outbound traffic, but now I am trying to also perform port forwarding for traffic received from the tunnel.

I have a port forwarding NAT rule attached on the VPN (tun_wg0) interface, that translate the destination IP with IP that sits on the LAN.
The issue: once translated, I would expect the traffic to go out on igc1 but it is actually sent to igc0. And I have no idea why.
Packet capture shows the following:

capture on tun_wg0 shows the packet with the public src IP I am testing with, destination IP is the tun_wg0 ip. (pre-NAT)
capture on igc1 (lan) where I expect the packet shows nothing
capture on igc0 (wan) shows the packet correctly translated. The destination IP is an IP that belongs to the lan (igc1) subnet.

From pfSense, I can ping the local host just fine: the LAN interface is in the routing TABLE, obviously. And I have some other port forwarding rule on the WAN interface, toward LAN hosts... It is working. I only experience this routing issue with the NAT rule attached to the VPN (tunnel) interface.

I tried so many things since yesterday... It would be hard to resume everything but here are the most relavant thing (IMHO)s.

To get the reply-to compiled in the firewall rule, I added a upstream gateway IP to the tunnel interface, same IP as the interface. I can confirm it is working with pfctl -sr;
-the rules compiled start with "pass in quick on tun_wg0 reply-to (tun_wg0 10.145.191.77)"
I also tried policy routing. I created a gateway attached to the LAN interface, tried both the destination host or the pfsense local address, and I attached to the rule.
it was indeed compiled correctly (route-to igc1, relevant IP), but the packets would still end up on igc0
Masquerading. I have seen it suggested, but it didn't work. The issue is that it is a port forwarding so an inbound rule gets hit first. I translate the destination IP. The packet gets routed straight to igc0 so even if I attached an outbound NAT rule there to change the src IP, it would never reach the host. And of course, attaching the outbound NAT rule to the LAN interface has no effect.

Nothing looks wrong in the route table;

0.0.0.0	        10.11.16.241	UGS	16	1500	igc0	
10.11.16.241	link#1		UHS	15	1500	igc0	
10.145.191.77	link#6		UH	18	16384	lo0	
74.15.*.*/29	link#1		U	7	1500	igc0	
74.15.*.*	link#6		UHS	14	16384	lo0	
127.0.0.1	link#6		UH	3	16384	lo0	
192.168.10.0/24	link#2		U	9	1500	igc1	
192.168.10.254	link#6		UHS	10	16384	lo0

Note: I have redacted the last two bytes of my public IP for security purpose, these two lines don't show up like that on my system.

I don't understand why the packet ends up on igc0/link1 with a destination IP for 192.168.10.0/24. With no understanding of why it's happening, it is really hard for me to come up with a solution. Bypassing the route-to policy is mind-blowing to me.

I spent almost 3.5 hours with various AI suggesting all sort of things that wouldn't work. And worse part is... I got all of this working previously on a PPPoE setup. But going back to the previous configuration isn't an option.

Thanks in advance to anyone who even consider replying to this wall of text. :)

vlurk

One last thing... I have a workaround that works with TCP: HAProxy.
Frontend on tun_wg0 interface, backend is the server sitting on my LAN (obviously).
No need for DNAT using HAProxy.

Luckily, TCP is sufficient for now but I might have UDP use cases coming soon also so this workaround is not a complete solution. But it works.

vlurk

For what it's worth, I fixed it.
It's possibly a combination of different things... After all, I am now using a different interface to host the service: I relocated the service on a different VLAN, amongst the then thousand things I have tried.

But I currently use DNAT (haproxy stopped), and have active sessions rolling (data in/out, visible with pfState) for the service exposed with DNAT on the tunnel interface. Behaving as expected.
Yet, one thing I could not expect, and still don't understand exactly why.... I can't test my setup at all from my LAN, at all.
Meaning that if I do a nmap -p <service port> <public IP for VPN endpoint that is not hosted by me or my ISP>, it fails with the same behavior I documented originally.

I don't know how pfSense is capable of matching the packet that was translated, ingress from the VPN tunnel, to then route it "back" to the WAN... But it does. That packet has the src-IP from my Internet provider, that is true. But it was received on the tunnel interface, with the tunnel interface local IP in destination and yet... It will perform DNAT (successfully) and route the packet to the WAN interface (incorrectly), completely bypassing the route table or even a <route-to> policy that should forward the packet to another local interface.

There is mechanism that takes precedence on pf <route-to>, and the routing table, that I don't understand. Probably with how outbound NAT is handled (optimized) in the kernel?

If someone knows, please share. Otherwise, I'll probably try a few rounds of AI questioning that will likely not yield a very good answer, as LLMs don't seem to be really proficient with pfSense and FreeBSD15 yet.

Cheers, and a happy Saint-Jean-Baptiste day to all Quebecers.

stephenw10

Testing from your own LAN to the public IP end of the tunnel would result in the source address being your own WAN. Unless the tunnel operator is source NATing the traffic to hide it replies would go back directly.

The gateway on the tunnel for reply-to should be the remote end of the tunnel not the local tun IP.

giuliafw70

I would separate two tests here: one from a real outside host, and one from your own LAN. The LAN test is a bad signal for this case because the source is effectively your WAN-side address path, so asymmetric return can make it look like pfSense chose the wrong egress after NAT.

Also, using the local /32 tunnel address as the gateway is suspicious. For reply-to to be useful it needs a real next hop on that interface; pointing it at itself can compile but still not mean what you want. If the VPN provider does not expose a usable peer/next-hop address, I would rely more on captures from an external test client than on the LAN hairpin test.

vlurk

@giuliafw70 and @stephenw10

The VPN provider does not expose a usable peer/next-hop address, unfortunately.
Creating a gateway on the tun interface, using its local IP, is how I got the setup to work, the capability to do policy routing by adding the tunnel as a gateway to firewall rules also.
I don't think I have any other choice due to the provider setup.

Not relying on local tests at all is my biggest take away from my story at this point: testing from the local lan introduced some additional mechanisms that skewed my test results. I just don't understand exactly what mechanisms, and why. It does work from an external client perspective, but I can't (yet) explain why it does not locally. Why the routing table is not enforced.

The packet I receive from the tunnel interface does have my WAN IP in source indeed.
The VPN provider translates the destination IP before forwarding it to me with my tunnel interface IP, then I translate it again for a local address.
It effectively means as the packet arrives from the VPN provider, the destination IP is never associated with associated with the WAN interface, before or after DNAT.
I can't comprehend why it would be forwarded to the WAN interface in such scenario, especially after I compiled a rule with route-to forcing the packets matched by the allow route to be routed on the LAN interface. The routing table and route-to directives should always be enforced, but it most definitely isn't in my case. This is basically what I still fail to understand at this point.

MalagaFirewall8

For the local test, the confusing bit is probably the source, not the translated destination. After rdr, the packet is going to a LAN host, but the source is still your WAN-side address/path, so any return path or state lookup can look like normal WAN traffic rather than a clean “came from tunnel, go back tunnel” case. I would inspect the state that gets created during the bad LAN test with pfctl -ss -vv | grep -A2 <port-or-lan-host> and compare it to a real external test. If the interfaces/state differ, that explains why the route table feels bypassed: pf is following the state/reply-to decision made when the flow was created, not doing a fresh route lookup for every packet.

stephenw10

@giuliafw70 and @MalagaFirewall8 you look like AI agents. Convince me you're not!

But, yes, test from a real external address first. I wouldn't ever expect a route looped connection from the LAN to work as expected unless something external is masquerading.