This is working now.
I set up a static routes at both ends and also configured gateway grouping between VPN and point to point link. I believe some changes in pfsense like advanced gateway settings for rules and routes don't get active until a reboot is done. This caused me a lot of a hardship earlier.
Still there are some small glitches:
1. I have fibre as default link in the branch firewall, so firewall (only this) has no Internet connection (so updates for firmware, bogons, etc are not automatic).
2. Sometimes even though the link comes back up, the firewall doesn't switch back to fibre link after using the backup VPN.