So I deleted all the HAProxy settings, uninstalled the package, rebootet my pfsense box, installed the HAProxy 1.8.30 package (haproxy package 0.61_3) and reconfigured all backends and frontends again ... now none of the backends were working.
I repeated the process with the very same result. For a backend to work I had to enable the health check which shouldn't be necessary I guess.
So I repeated the whole process a third time and for some reason it works now as expected. All backends work without any issues so far. I hope this is still the case when I add further backends in the future 🤔