So I managed to achive what I wanted via additional DNS server using dnsmasq. The example setup looks like this:
Isolated DNS server running DNSMASQ: 192.168.10.2
LAN: 192.168.1.0/24
WG0: 10.10.0.50
VPN DNS server: 10.10.0.2
I created two aliases:
vpn_isolation - with networks for each machine that will be forced to use VPN - network aliases can include single hosts with netmask /32 and it's less problematic than to remove 255 entries from an ip alias that expanded whole /24 network :D
isolated_dns - this alias only contains 192.168.10.2 - this will make our life way easier if we decide to move the dnsmasq to different subnet
First we create port forwarding rule:
Firewall -> NAT -> Port Forward
Interface: LAN
Protocol: TCP/UDP
Source: Address or Alias: vpn_isolation
Destination Port Range: DNS / DNS
Redirect Target IP: isolated_dns
Redirect Target Port: DNS
Filter Rule Association: Add associated filter rule
NAT Reflection: disable
Description: Force DNS to VPN
Next we need same rule for interface on which the dnsmasq works so we can pass all the traffic to VPN from dnsmasq.
Then we need to create a policy routing rule that will match ips/networks from vpn_isolation alias on the LAN interface:
Firewall -> Rules -> LAN
Source: ip address or alias: vpn_isolation
Destination: either * or exclude private networks to allow routing to internal subnets
Gateway: WG0_GATEWAY
Finally we need to spawn a linux box or container with IP 192.168.10.2 that runs dnsmasq. Below is example dnsmasq config:
no-resolv
no-poll
# we tell dnsmasq to use VPN
server=10.10.0.2
# then we tell dnsmasq to use 192.168.1.1 to resolve *.lan and *.myinternaldomain.omgyay
# (or any other domain or suffix we need)
server=/lan/192.168.1.1
server=/myinternaldomain.omgyay/192.168.1.1
# this is important otherwise dnsmasq won't reply to queries from different network
listen-address=192.168.10.2,127.0.0.1
We can test the setup from a machine with IP included in vps_isolation alias:
use https://dnsleaktest.com/ - it should show single DNS or at least DNS different that the one pfsense's DNS Responder/Forwarder uses
more imporant - we need to check if we don't leak original WAN subnet via ECS - just issue curl -SL https://test.nextdns.io and resulting JSON
should not include "ecs" key with your WAN subnet - this was the biggest problem for me when using DOT from DNS Resolver - if you own a ripe you basically dox yourself this way.
Both port forwarding and policy routing firewall rules have to be added to every interface we want to use vpn isolation and they need to be above any other policy routing rules that might redirect traffic elsewhere and go through clearnet ofc.
With this setup when you want to enable/disable vpn for any host or network behind pfsense all you need to do is edit the vpn_isolation alias and you're done.
CAVEAT: make sure the dnsmasq dns server is on it's own subnet. this makes things easier. I was able to get this working with same subnet for dnsmasq and vpn_isolation, but you have to create an additional port forwarding rule above the one that intercepts DNS traffic that matches traffic from dnsmasq and has "Disable redirection for traffic matching this rule" checked. This will allow dnsmasq to talk to pfsense :)
