[SOLVED] Route pfsense itself over VPN.

stephenw10

Hmm, and the rule still shows no states or traffic on it?

Do you have a floating rule allowing that traffic perhaps?

You can check the states to see what rule opened it at the CLI: pfctl -vvss
That will show all the states though which could a lot!

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Hmm, and the rule still shows no states or traffic on it?

Correct.

Do you have a floating rule allowing that traffic perhaps?

No.

You can check the states to see what rule opened it at the CLI: pfctl -vvss
That will show all the states though which could a lot!

pfctl -vvss is showing tons of states on all interfaces and vlans, so i refined my search with pfctl -vvss |grep 100.133 and then pfctl -vvss |grep 100.111 and then pfctl -vvss |grep 100.64

None of these searches returned anything even if i execute them while the phone is connected to tailscale and receiving/sending traffic. If i use known non tailscale ip address, its shown clearly and then i can find it in the list along with the firewall id. But tailscale ips return nothing.

stephenw10

Try sending some traffic from a client to some unique IP and check the states created by that. You could be seeing it post NAT.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Try sending some traffic from a client to some unique IP and check the states created by that. You could be seeing it post NAT.

This is getting very strange now.

First i initiated continuous rtsp traffic from my PC to local camera and then i run pfctl -vvss to check the states and i could clearly see this:

all tcp 20.20.1.1:19791 (10.1.1.100:57738) -> 20.20.1.236:8083 ESTABLISHED:ESTABLISHED [222663977 + 2147156224] wscale 7 [2738703565 + 4293657088] wscale 7 age 00:01:20, expires in 23:59:46, 25:55 pkts, 2212:68118 bytes, rule 122, allow-opts, log id: f013346700000000 creatorid: cd852d8c origif: igb5.25

10.1.1.100 - pc
20.20..1.1 - pfsense camera interface
20.20.1.236 - camera

Then i did the same from my phone via tailscale (tailscale is configured to broadcast 20.20.1.0/24 subnet) and this is what i see when i run pfctl -vvss again:

all tcp 20.20.1.1:1996 -> 20.20.1.236 :8083 ESTABLISHED:ESTABLISHED [2330702467 + 2147156224] wscale 7 [599643748 + 65792] wscale 7 age 00:01:10, expires in 23:59:52, 26:22 pkts, 3121:14535 bytes, rule 122, allow-opts, log id: 0a14346700000000 creatorid: cd852d8c origif: igb5.25

There is no tailscale or my phone ip anywhere in the logs when access to camera is initiated via tailscale.

stephenw10

Hmm, that's odd. Let me do some digging here. What pfSense version is this?

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Hmm, that's odd. Let me do some digging here. What pfSense version is this?

Its 2.7.2 CE with all the patches applied using system patches pacakage.

stephenw10

Mmm, OK, clearly been too ling since I played with Tailscale. That's the expected behaviour.

See: https://www.youtube.com/watch?v=Fg_jIPVcioY&t=1240s

Which makes it very inconvenient as there's no way to policy route the traffic as far as I know. The only way to route that traffic would be to set the default route to the VPN. You could then policy route traffic other traffic via the WAN but pfSense itself would always use the VPN.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Mmm, OK, clearly been too ling since I played with Tailscale. That's the expected behaviour.

You have been helpful Stephen no matter the outcome. Much appreciated.

See: https://www.youtube.com/watch?v=Fg_jIPVcioY&t=1240s

This makes sense now.

Which makes it very inconvenient as there's no way to policy route the traffic as far as I know. The only way to route that traffic would be to set the default route to the VPN. You could then policy route traffic other traffic via the WAN but pfSense itself would always use the VPN.

Thats how im using it currently, but the issue is that other openvpn tunnels that i use are failing to connect when pfsense is rebooted, and then i need to manually restart all openvpn services including the one selected as the default wan. Once thats done, everything is working perfectly, including the tailscale traffic.

stephenw10

You could set static routes for those VPN endpoints via the WAN gateway.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

You could set static routes for those VPN endpoints via the WAN gateway.

Im sorry, im talking about openvpn clients in pfsense. They fail to connect to mullvad vpn server when rebooted. Can a static route help with that, and if so, how?

Not really sure what should i set in Destination network.

stephenw10

Put in the IP address of the VPN server the client connects to. pfSense uses the system routing table to determine how to route outgoing connections. Adding a static route to the VPN server will route it via the WAN.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Put in the IP address of the VPN server the client connects to. pfSense uses the system routing table to determine how to route outgoing connections. Adding a static route to the VPN server will route it via the WAN.

I have 3 openvpn clients in pfsense. Should i create 3 static routes ? One for each vpn server ?

stephenw10

Yes, add static routes to anything that needs to use the WAN gateway.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Yes, add static routes to anything that needs to use the WAN gateway.

One more question, what should i set as a interface in openvpn client

Or it doesnt matter and this is getting overridden by static mapping ?

I also have one wireguard client connected. I assume i need to create a static mapping for it as well ?

stephenw10

If you set the interface there it should source the connection from that IP address and route-to should force it via any gateway on that interface. But you seem to be seeing that not happen so adding a static route should correct that. At least as a test. If it still fails to connect then that's not the issue.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

If you set the interface there it should source the connection from that IP address and route-to should force it via any gateway on that interface. But you seem to be seeing that not happen so adding a static route should correct that. At least as a test. If it still fails to connect then that's not the issue.

I created static routes for all openvpn clients that connect using WAN interface, rebooted, and pfsense got stuck at

Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM

for 5-6 minutes and then continued to boot and stuck again at

igb5.35: link state changed to UP igb5.100: link state changed to UP igb5.200: link state changed to UP igb5.55: link state changed to UP igb5.65: link state changed to UP igb5.25: link state changed to UP igb5.15: link state changed to UP igb5.45: link state changed to UP

I waited for 20 minutes but nothing happened.

Pressing Alt+Ctrl+Del starts the shutdown sequence and system reboots and its stuck again in the same way i described above. Web gui was inaccessible as well as SSH. I could not access command prompt either, so i had to reinstall pfsense again restore my config which, luckily, i have saved today.

I never had crash like this before.

stephenw10

That's not a crash it's what you see if you're not looking at the primary console. You should see just before the mountroot something like:
Nov 8 13:08:47 kernel Dual Console: Serial Primary, Video Secondary

You can force the console to boot with the other one as primary at boot:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/boot-issues.html#booting-with-an-alternate-console

The fact it's not booting to the console menu implies it's stuck at something else that only showing on the primary console.

nimrod

@stephenw10

I forgot to mention, im not using serial console. I have a monitor and keyboard connected directly to pfsense box. The output i posted in previous post is from the primary console. I just copied dmesg output from fresh installation that works in order to show you where previous installation was stuck stuck.

Hope it makes sense now.

stephenw10

Hmm, well it sure seems like it's set to serial as primary console from what you're seeing. But as I say you should be able to see which is set at the vga console.

nimrod

@stephenw10 said in Route pfsense itself over VPN.:

Hmm, well it sure seems like it's set to serial as primary console from what you're seeing. But as I say you should be able to see which is set at the vga console.

Success !!!

I made the mistake when creating a static route by typing ip address of openvpn server and then selecting the subnet. Selecting the subnet was a mistake that caused pfsense to be stuck at boot. I created new static route by typing the openvpn server address, chose the appropriate WAN and saved the setting. After reboot, pfsense booted just fine. I created additional static routes for the remaining vpn clients and everything just worked. Rebooted once again, no issues. Then i selected openvpn client as a default gateway and that was it.

All tailscale clients are now going through vpn, and all vpn clients connect without any issues after reboot.

Thank you very much Stephen. All this would not be possible without your help. Im marking this thread as resolved.

Cheers.