I did reboot pfsensene VM and made sure from ip a in the proxmox host that there was no IP set for the interfaces I was bridging and it magically worked.

Actually, now it's even better. In the proxmox interface I managed to set an IP for the interface that is being bridged, so that when pfsense VM is down I can still reach the proxmox on its fixed ip on the LAN side.