Make a test environment on the virtual machine and check if you are able to connect. It is a waste of time to guess and look for the reason where it may lie, for example in the enabled DDoS functions on the switch.
Alternatively, you can paste the ipsec logs onto pfsense here.