I've been isolating and troubleshooting some more and it seems like it's an IPv6 issue after all.
Fully disabling IPv6 resolves all issues.
So, now I need to find out which pfSense settings need adjustments. There are some tutorials for my ISP (XS4ALL) but none of those resolve the issue.
I'm going to dive into a config export of the ISP's router and see if I can discover something funky. Because that one works.