@Gertjan:
From what I understood, all your LAN stuff is hooked up to a switch - and the pFsense box is also on this switch.
That's correct.
Cable modem <-> pfSense box <-> switch <-> computers, nas devices etc.
I have now tested switching back the Netgear router/firewall to replace pfSense, and amazingly, all the problems described in the first post disappeared. Just to verify the causality of this, I switched back the pfSense box, and none of the problems reappered.
Conclusion: no idea what was causing this, probably I'll never find out. What matters is that everything works.