As always is the case, I resolved this minutes after posting.
It's quite an obscure setting but I needed to also enable Security --> Forged Transmits in the vSwitch.

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-7DC6486F-5400-44DF-8A62-6273798A2F80.html