Have you been looking at the trafficflow using tcpdump on site B to see if requests reaches the server and what happens when the server responds ?
Syntax in shell: tcpdump -i LANIF -n host externalclient
Where LANIF should be replaced with whatever interface on pfSense your server is connected to and externalclient replaced by the IP of the client on the internet trying to reach the server.
If you don´t see any responses from server here, then try to change LANIF to what corresponds to your WAN interface and try again.
If so you might have a case of what is called asymetric routing, ie. client on the internet surfs to your public IP on site A, traffic flows over to site B though IPSec and eventually reaches server on site B. The quirk is that server on site B cannot find the client IP in any routingtable except default route and that points out through WAN interface of site B.
In that case you´ll have to rewrite the sourceaddress at site A.