I'd also investigate a MTU mismatch etc... Here's my (potentially flawed) logic:
Server on Side A has larger MTU than Server on Side B. (I assume you copy server to server)
Initializing the transfer from Site (B) I can copy FROM a file server on Server (A) with roughly 20MB/sec which is great.
I assume the server on Side B requests a small packet size... (Maybe Path MTU Discovery)
Initializing the transfer from Site (B) I can copy TO a file server on Server (A) with roughly 20MB/sec which is great.
The server on Side B sends data packets that are smaller than Server A maximum accept size.
Initializing the transfer from Server (A) I can copy FROM a file server on Site (B) with roughly 20MB/sec which is great.
The server on Side B will only send small packets (or packets that are smaller than what Server A can receive)
...but Initializing the transfer from Server (A) I can copy TO a file server on Site (B) with only roughly 8MB/sec
Server A doesn't know that Server B can only receive small packets. The Firewall (VPN endpoint) on Side B now has do extra work breaking up large packets into smaller ones - which Server B can accept.
So my guess would be fragmentation etc...
MTU can be set on Host interfaces, too ... You could try reducing the MTU Size on Server A network interface.
Also have a look at the pfsense option (Remove DF bit)
https://www.reddit.com/r/sysadmin/comments/2mt3jc/reducing_mtu_value_to_fix_slow_cifssmb_over_vpn/