[SOLVED] Route pfsense itself over VPN.

nimrod

I created the alias with tailscale cidr ip. Used it as a source in a firewall rule on tailscale group, and the traffic is still going through my ISP instead of VPN. Here is the rule.

Tried reloading firewall rules and restarting tailscale service with no result. Tried using ip address of my phone instead of complete cidr block. Same thing.

The interesting part about this is that the DNS resolution is going through VPN which is what i want, but the rest of the traffic is still going through default gateway.

stephenw10

Hmm, no states shown on it so it's not matching and passing traffic for some reason.

Is it actually arriving with that source?

Is some other rule passing it?

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Hmm, no states shown on it so it's not matching and passing traffic for some reason.

Is it actually arriving with that source?

Yes. Here is the screenshot from tailscale dashboard.

These ip addresses are also matching what the tailscale in pfsense is showing.

Is some other rule passing it?

There are no other rules anywhere, except tailscale group, where im routing 100.64.0.0/10 subnet.

stephenw10

Hmm, and the rule still shows no states or traffic on it?

Do you have a floating rule allowing that traffic perhaps?

You can check the states to see what rule opened it at the CLI: pfctl -vvss
That will show all the states though which could a lot!

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Hmm, and the rule still shows no states or traffic on it?

Correct.

Do you have a floating rule allowing that traffic perhaps?

No.

You can check the states to see what rule opened it at the CLI: pfctl -vvss
That will show all the states though which could a lot!

pfctl -vvss is showing tons of states on all interfaces and vlans, so i refined my search with pfctl -vvss |grep 100.133 and then pfctl -vvss |grep 100.111 and then pfctl -vvss |grep 100.64

None of these searches returned anything even if i execute them while the phone is connected to tailscale and receiving/sending traffic. If i use known non tailscale ip address, its shown clearly and then i can find it in the list along with the firewall id. But tailscale ips return nothing.

stephenw10

Try sending some traffic from a client to some unique IP and check the states created by that. You could be seeing it post NAT.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Try sending some traffic from a client to some unique IP and check the states created by that. You could be seeing it post NAT.

This is getting very strange now.

First i initiated continuous rtsp traffic from my PC to local camera and then i run pfctl -vvss to check the states and i could clearly see this:

all tcp 20.20.1.1:19791 (10.1.1.100:57738) -> 20.20.1.236:8083 ESTABLISHED:ESTABLISHED [222663977 + 2147156224] wscale 7 [2738703565 + 4293657088] wscale 7 age 00:01:20, expires in 23:59:46, 25:55 pkts, 2212:68118 bytes, rule 122, allow-opts, log id: f013346700000000 creatorid: cd852d8c origif: igb5.25

10.1.1.100 - pc
20.20..1.1 - pfsense camera interface
20.20.1.236 - camera

Then i did the same from my phone via tailscale (tailscale is configured to broadcast 20.20.1.0/24 subnet) and this is what i see when i run pfctl -vvss again:

all tcp 20.20.1.1:1996 -> 20.20.1.236 :8083 ESTABLISHED:ESTABLISHED [2330702467 + 2147156224] wscale 7 [599643748 + 65792] wscale 7 age 00:01:10, expires in 23:59:52, 26:22 pkts, 3121:14535 bytes, rule 122, allow-opts, log id: 0a14346700000000 creatorid: cd852d8c origif: igb5.25

There is no tailscale or my phone ip anywhere in the logs when access to camera is initiated via tailscale.

stephenw10

Hmm, that's odd. Let me do some digging here. What pfSense version is this?

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Hmm, that's odd. Let me do some digging here. What pfSense version is this?

Its 2.7.2 CE with all the patches applied using system patches pacakage.

stephenw10

Mmm, OK, clearly been too ling since I played with Tailscale. That's the expected behaviour.

See: https://www.youtube.com/watch?v=Fg_jIPVcioY&t=1240s

Which makes it very inconvenient as there's no way to policy route the traffic as far as I know. The only way to route that traffic would be to set the default route to the VPN. You could then policy route traffic other traffic via the WAN but pfSense itself would always use the VPN.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Mmm, OK, clearly been too ling since I played with Tailscale. That's the expected behaviour.

You have been helpful Stephen no matter the outcome. Much appreciated.

See: https://www.youtube.com/watch?v=Fg_jIPVcioY&t=1240s

This makes sense now.

Which makes it very inconvenient as there's no way to policy route the traffic as far as I know. The only way to route that traffic would be to set the default route to the VPN. You could then policy route traffic other traffic via the WAN but pfSense itself would always use the VPN.

Thats how im using it currently, but the issue is that other openvpn tunnels that i use are failing to connect when pfsense is rebooted, and then i need to manually restart all openvpn services including the one selected as the default wan. Once thats done, everything is working perfectly, including the tailscale traffic.

stephenw10

You could set static routes for those VPN endpoints via the WAN gateway.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

You could set static routes for those VPN endpoints via the WAN gateway.

Im sorry, im talking about openvpn clients in pfsense. They fail to connect to mullvad vpn server when rebooted. Can a static route help with that, and if so, how?

Not really sure what should i set in Destination network.

stephenw10

Put in the IP address of the VPN server the client connects to. pfSense uses the system routing table to determine how to route outgoing connections. Adding a static route to the VPN server will route it via the WAN.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Put in the IP address of the VPN server the client connects to. pfSense uses the system routing table to determine how to route outgoing connections. Adding a static route to the VPN server will route it via the WAN.

I have 3 openvpn clients in pfsense. Should i create 3 static routes ? One for each vpn server ?

stephenw10

Yes, add static routes to anything that needs to use the WAN gateway.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Yes, add static routes to anything that needs to use the WAN gateway.

One more question, what should i set as a interface in openvpn client

Or it doesnt matter and this is getting overridden by static mapping ?

I also have one wireguard client connected. I assume i need to create a static mapping for it as well ?

stephenw10

If you set the interface there it should source the connection from that IP address and route-to should force it via any gateway on that interface. But you seem to be seeing that not happen so adding a static route should correct that. At least as a test. If it still fails to connect then that's not the issue.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

If you set the interface there it should source the connection from that IP address and route-to should force it via any gateway on that interface. But you seem to be seeing that not happen so adding a static route should correct that. At least as a test. If it still fails to connect then that's not the issue.

I created static routes for all openvpn clients that connect using WAN interface, rebooted, and pfsense got stuck at

Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM

for 5-6 minutes and then continued to boot and stuck again at

igb5.35: link state changed to UP igb5.100: link state changed to UP igb5.200: link state changed to UP igb5.55: link state changed to UP igb5.65: link state changed to UP igb5.25: link state changed to UP igb5.15: link state changed to UP igb5.45: link state changed to UP

I waited for 20 minutes but nothing happened.

Pressing Alt+Ctrl+Del starts the shutdown sequence and system reboots and its stuck again in the same way i described above. Web gui was inaccessible as well as SSH. I could not access command prompt either, so i had to reinstall pfsense again restore my config which, luckily, i have saved today.

I never had crash like this before.

stephenw10

That's not a crash it's what you see if you're not looking at the primary console. You should see just before the mountroot something like:
Nov 8 13:08:47 kernel Dual Console: Serial Primary, Video Secondary

You can force the console to boot with the other one as primary at boot:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/boot-issues.html#booting-with-an-alternate-console

The fact it's not booting to the console menu implies it's stuck at something else that only showing on the primary console.