I figured it out, I forgot some settings elsewhere

OK then the MAC address should be spoofed. The MAC address on the LAGG should also be the spoofed MAC. That is exactly what would be expected.

The copy jobs will be between nas to vsphere and external. Probably it will be smb3, i did not decide yet. After removing the whole lagg config on pfsense and switch it works! I can work with that but i'm still interested why it did not work with lag...