The router from my ISP is setup in 'modem only' mode, it does not perform any routing or wi-fi functions, its only connection is t othe WAN port of my pfSense unit.
I run Unifi switches and access points all of which sit behind the pfSense unit.
I am guessing that since I only have the one WAN IP, once the VPN tunnel is opened from the pfSense firewall, the VPN IP is now perceived by all clients to be their external IP, whereas previously when I ran a VPN on an individual device, the VPN IP only applied to that single device.
In effect running a PIA VPN tunnel from the pfsense firewall can only act as a 'whole house' VPN, regardless of what firewall rules I may use.
I have also noticed a severe drop in bandwidth when using the PIA OpenVPN tunnel on the pfSense firewall.
All tests were performed from my iMac desktop:
Test Case down/up
No VPN 386/20.8
pfSense + London 152/19.8
pfSense + Southampton 205/19.4
VPN app + London 303/19.5
VPN app + Southampton 293/19.6
The PIA app based firewall is using wireguard, although until recently it was using OpenVPN, the results using the app are usually within 50-60Mb/s of the figures with no VPN (they are a bit down today), but never as bad as those shown for OpenVPN on pfSense.
Looks like I may be sticking with local VPNs for now.
For comparison, I ran a speedtest from my media server using a wireguard based PIA tunnel to the same London server and recorded speeds of 317/19.6 with the VPN tunnel and 322/21.1 without. The media server is connected to the same switch as my iMac, both with 1m cables.