[SOLVED]
Thank you all for all the tips and recommendations. I have now solved the issue by doing the following:
As pfSense was crashing when I plugged a cable into one of the ports, and the ports were all in auto (IB) mode as I discovered, I figured it might be an issue related to the NIC changing its configuration from IB to ETH, and FreeBSD trying to load the driver or something (I'm not sure).
However, I saw this amazing video on changing the mode of the NIC permanently with the MFT tool. As the configuration gets saved to the NIC itself, you can use any other Linux or FreeBSD installation, use MFT to change the values, and then reinstall pfSense. After that, the NIC is correctly in the ETH mode, and the driver loads correctly without issues. (I see no issues displayed in dmesg as far as I can tell.)
Video: https://www.youtube.com/watch?v=D1qN7Qg3bSg
Now, I just need to make sure the cards run stable and reliably, but from what I see now, I am confident.
Thanks again to everyone who helped.