Thank you @NOCling and @keyser for your insight.
I went back and reviewed step by step 2 guides from the Netgate documentation:
For Remote VPN connections with Certificate, I followed this guide: https://docs.netgate.com/pfsense/en/latest/recipes/ipsec-mobile-ikev2-eap-tls.html And for Site to Site VPN connections with Certificate, I followed this other guide: https://docs.netgate.com/pfsense/en/latest/recipes/ipsec-s2s-tls.htmlMy issue was I got creative and changed some settings the first time. After knowing @keyser and @NOCling were able to make it work, I followed those 2 the guides, and everything worked fine!
Thank you both!