Solved. This turned out to be an ISP side routing issue. Like in https://forum.pfsense.org/index.php?topic=104583.0 , I was trying to advertise the /48 to the ISP gateway through SA, which is apparently not how it's supposed to work.
Testing with netcat6, I listened at a remote server for UDP packets and sent one from the LAN. It went through. So in fact LAN->Internet was working, but replies (Internet->LAN) never came through.
This was resolved by the ISP setting up a separate link network {linkprefix}::/64, and then routing {prefix}::/48 <-> {linkprefix}::1/64 (ISP GW) <-> {linkprefix}::2/64 (Pfsense) <-> LAN. (No route daemon running on pfsense, only static configs at ISP side.)
Unfortunately, I never found out why/how Internet<->Pfsense traffic worked before (regardless of Pfsense box's address), if there were no routes set up at ISP's before. Maybe their gateway added my Pfsense box as a single host to their routing tables through IPv6 Neighbor discovery or something.