OK, I finally found some time to dig into this. Writing this followup in case anyone runs into a similar issue in the future.
Some investigation with tcpdump showed that the remote host was seeing checksum errors on the TCP packets (yet ICMP was working fine).
Disabling hardware checksum offloading in pfSense resolved my issue (System -> Advanced -> Networking -> Hardware Checksum Offloading).
pfSense is running under XCPng 8.1.0 on an unknown motherboard using onboard Intel 82576 Gigabit NIC.
Not sure if the checksum error is creeping in due to XCP or NIC drivers, but in my case the network traffic is quite low, so I'll run with the hardware checksum disabled.