Well I figured out the problem, but I can't come up with a way to fix it (for me) yet. Let's say your client network (the client to the CARPed firewalls) is 10.20.30.0/24. The server network is 10.40.50.0/24, firewall A is 10.40.50.1 and firewall B is 10.40.50.2.
If the client tries to connect to 10.40.50.1 it works fine of course. If the client tries to connect to 10.40.50.2 it goes out on the LAN from 10.40.50.1 correctly, the problem here is actually the reply from 10.40.50.2, because it has no route to 10.20.30.0/24. You can solve this by adding a static route on firewall B (10.40.50.2) on the LAN for 10.20.30.0/24 with the gateway set to 10.40.50.1. This only works if firewall A is the VPN server and firewall B is not (if firewall A is down, there is no VPN connection).
In my situation, I have the OpenVPN server configuration duplicated on both firewalls, and I have it listening on the CARP WAN IP. The client connects to the CARP IP so that if one firewall goes down, it will reconnect to the other one automatically as soon it picks up the CARP IP. That part of it works fine, but I can never connect to the server I'm not connected to.
I can't add a static route because both have routes for 10.20.30.0 already even if the tunnel is not up and as far as I can tell there's no way I can change this behavior, or otherwise allow for automatically changing the route.