The fact that the transfer fails at almost exactly the same point and coincides with a 100% CPU spike and gateway latency increase makes me suspect the pfSense box itself rather than the VLANs or SMB connection.
I’d check the pfSense CPU usage per process during the transfer rather than just the overall CPU graph. Since you’re also running pfBlocker, DNS, DHCP and WireGuard, one of those services could be contributing to the spike. It may also be worth temporarily disabling WireGuard and pfBlocker completely and testing a large transfer between two VLANs.
I’d also try an iperf3 test between the VLANs. If iperf3 reproduces the CPU spike/drop, you can eliminate SMB as the cause and focus on packet processing, firewall rules, NIC drivers/offloading, or the hardware itself.
If the issue consistently appears at the same transfer point, I’d definitely investigate what process is consuming CPU at that exact moment.