If I block the ip with a fw rule, then it will be blocked. I need the ip to connect to the remote side, but just not over the vpn.
The traffic will only be blocked on the VPN interface. If the traffic was passing in over the WAN or another interface, you could pass the traffic. If it's a routing issue, that could be complicated as a tunnel will trump a local route, but that would be an unusual situation.