I have a number of physical networks and VLANs all working happily with vswitches in pfSense running inside ESXi.
Maybe it would help us to help you if you told us the IPs for each of the networks. I'm having trouble understanding why you have physical port + VLAN for the same networks eg LAN & Wifi. Your most recent description of what you want to achieve is different to your original post. The more detail you give us, the easier it will be for us to understand what you are trying to do and then help you.