Routing forwarded packets back through OpenVPN connection

dr1

Spent most of yesterday trying to solve this, but I dont have enough BSD experience to figure it out it seems. I am replacing a Linux router I had setup for over a decade now probably. I have got all the functionality replaced and working except one thing. I've got 3 networks basically: WAN, LAN, & OpenVPN creates a 3rd.

I'm running OpenVPN server on pfSense, the incoming connection is running on a VM on the internet, its sole purpose basically is provide me with a static IP. It runs OpenVPN and connects to me, port forwards a selection of ports, and provides NAT for the returning packets. On my Linux firewall this required me to add a route, and mark new connections originating from the vpn.

echo "20 vpn" >> /etc/iproute2/rt_tables
ip rule add fwmark 0x12b lookup vpn
ip route add table vpn default via 10.3.244.2 dev VPN00
iptables -t mangle -A PREROUTING -i ETH00 -m connmark --mark 0x12b -j MARK --set-xmark 0x12b/0xffffffff
iptables -t nat -I PREROUTING 1 -i VPN00 -m state --state NEW -j CONNMARK --set-xmark 0x12b/0xffffffff

I'm trying to do the same here on pfSense but, I dont even know my way around the command line, and I've seemed to exhaust what I can figure out how to do with the GUI. From the pfSense machine, via tcpdump, I can confirm Im receiving the (Internet) -> (Internal Lan) connections, that are being forwarded through. And theres an attempt at (LAN) -> (Internet) response, and I assume that is incorrectly going out the WAN.

Two things I learned experimenting: I could create a Floating rule to say, match ALL traffic originating from a server on my LAN, and set it to gateway out of an IP on the VPN, but it wouldnt switch gateways, as far as I could tell. The firewall rule was matching however. The 2nd thing was attemping to use tag/tagged, and I'm not sure this can achieve the same as my above example using this method, the important part of the above example is the CONNMARK which is keeping track of the connection. So I assume perhaps a different approach needs to be taken, but Im not sure what that is yet.

Any suggestions that could put me on the right path are much appreciated. Also I really wasnt sure what category to place this question in, and also didnt want to limit exposure too much.

viragomann

@dr1 said in Routing forwarded packets back through OpenVPN connection:

I'm running OpenVPN server on pfSense, the incoming connection is running on a VM on the internet, its sole purpose basically is provide me with a static IP. It runs OpenVPN and connects to me, port forwards a selection of ports, and provides NAT for the returning packets.

You're running the OpenVPN server behind a dynamic public IP?

I'd rather expect that you run the server behind a static IP and you connect to the server and the server forwards traffic to you.

Can you explain your setup more clearly, please?

dr1

Yes that is a little backwards in practical sense. The VM is connecting to my hostname that is updated with DynDNS service (freedns). I mainly chose that direction so I could have a nice UI for server config, certificate management, etc. The VM is bare minimum on resources, and this is the only thing it does. All I had to do to switch over was to send it a new key and certs. Both ends of the connection are always up unless something has gone wrong.

viragomann

@dr1
So you need a site-to-site VPN.
Set the OpenVPN tunnel mask to /30.

Further this requires that there is a firewall rule passing the incoming traffic on the distinct VPN interface.

So if you didn't that already, assign an interface to the OpenVPN server instance (e.g. ovpns1) and enalbe it.
Then add a pass rule to this interface to allow the desired incoming traffic or move over the rule from the OpenVPN tab.

Ensure that there is no pass rule on the OpenVPN tab an no floating rule matching the incoming traffic.

Note that "OpenVPN" is an interface group, which includes all OpenVPN instances. Rules on interface groups and floating quick rules have priority over rules on the respective interface tabs.

dr1

So you need a site-to-site VPN.
Set the OpenVPN tunnel mask to /30.

Ok do I need to do this for some behind the scenes magic to take place? Does it care that its a /24 instead? Ive left as it for time being so I dont introduce more errors into the mix.

Further this requires that there is a firewall rule passing the incoming traffic on the distinct VPN interface.

So if you didn't that already, assign an interface to the OpenVPN server instance (e.g. ovpns1) and enalbe it.
Then add a pass rule to this interface to allow the desired incoming traffic or move over the rule from the OpenVPN tab.

Ensure that there is no pass rule on the OpenVPN tab an no floating rule matching the incoming traffic.

Note that "OpenVPN" is an interface group, which includes all OpenVPN instances. Rules on interface groups and floating quick rules have priority over rules on the respective interface tabs.

Ok that helps understand whats going on a bit more. I had quite a mess at some point trying different things. I now have 3 enabled interfaces, WAN, LAN, & OPT2 (vpn). Ive clearned the OpenVPN firewall rules, and added a pass everything to OPT2. I had been messing with the gateways so now im back to only 1 gateway (WAN). Ive deleted the floating rules I had.

As it stands it appears the same as before, but I assume having assigned that interface I have more control now. Everyone can talk amongst the LAN/VPN but the problem is still the same as before. The traffic that gets forwarded in seems to take the wrong path back out.

This is my view of it passing through the pfSense machine.

: tcpdump -n 'tcp port 993'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:48:33.721418 IP 107.122.192.162.20182 > 10.3.2.30.993: Flags [S], seq 3133438093, win 65535, options [mss 1255,sackOK,TS val 3122661809 ecr 0,nop,wscale 9], length 0
10:48:33.721474 IP 107.122.192.162.20183 > 10.3.2.30.993: Flags [S], seq 4233414992, win 65535, options [mss 1255,sackOK,TS val 3122661814 ecr 0,nop,wscale 9], length 0
10:48:33.721950 IP 10.3.2.30.993 > 107.122.192.162.20182: Flags [S.], seq 4082019908, ack 3133438094, win 65160, options [mss 1460,sackOK,TS val 2477894394 ecr 3122661809,nop,wscale 7], length 0
10:48:33.721971 IP 10.3.2.30.993 > 107.122.192.162.20183: Flags [S.], seq 926252875, ack 4233414993, win 65160, options [mss 1460,sackOK,TS val 2477894394 ecr 3122661814,nop,wscale 7], length 0

dr1

Better picture of the problem perhaps:

listening on ovpns1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:58:50.965079 IP 107.122.192.162.20180 > 10.3.2.30.993: Flags [S], seq 3551375273, win 65535, options [mss 1255,sackOK,TS val 3123278917 ecr 0,nop,wscale 9], length 0

listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:58:42.655367 IP 10.3.2.30.993 > 107.122.192.162.20180: Flags [S.], seq 4138910922, ack 3551375274, win 65160, options [mss 1460,sackOK,TS val 2478503328 ecr 3123263710,nop,wscale 7], length 0

viragomann

@dr1 said in Routing forwarded packets back through OpenVPN connection:

Ok do I need to do this for some behind the scenes magic to take place? Does it care that its a /24 instead?

I'm in doubt that it works with this. Since the gateway is not clear to pfSense possibly the reply-to tag cannot be set properly.

Ive clearned the OpenVPN firewall rules, and added a pass everything to OPT2

Just enable logging in this rule, initiate an access and check the firewall log then.

If the rule is applied I'd suspect that it's on the tunnel subnet.

dr1

Rules

Log

No return packet on that interface.

Whats the method for forcing packets out a specific interface? I've seen the route-to flag to be used with pf, I dont know how to implement that in UI. I could probably solve this by force just from the port numbers, at least in the interim.

viragomann

@dr1
Again, it needs the reply-to tag, and this needs to determine the gateway of the interface to work.
So check on Status > Interface if the virtual IP of the client is listed as gateway at OPT2.
If you only see the OpenVPN server IP there, what I assume, it won't work.

dr1

@viragomann

You were right my friend, thank you thank you thank you. I was nearing giving up on this project lol. Was about to plug the broken router back in for now.

My OpenVPN config was 'tap' mode, which mainly was because of the previous system requiring me to use it. I didnt think it would cause to much problems but behind the scenes what needed to happen was not happening. I had no gateway interface being configured (thats why i had added one a couple times for testing).

I've set it to a /30 and 'tun' mode and it seems to be working without further input now.