OpenVPN Site-to-Site: OpenVPN connected, but pfSense doesn't "see" connection
I have two pfSense systems, both running 2.3.4-RELEASE-p1 under VMware 5.5. System A has one WAN interface, and is running two OpenVPN instances for Site-to-Site connections on port 1194 and 1195 (Shared Key). System B has two WANs, each of which connects to one OpenVPN instance on System A. The tunnel networks are both /30's. I'm using a Gateway Group on System B to assign traffic to the group via firewall rules (policy based routing). "Dummy" interfaces and NAT rules have been setup, of course.
The system can be described as a "Multi-WAN, Load-Balancing OpenVPN Site-to-Site network", and it's been working flawlessly for two years now. At some point I've updated System A from 2.3.4 to 2.3.4p1, and System B to the same version some time later, without making any config changes (there have been prior pfSense updates as well, which didn't create any issues). There have been changes to the internet lines on both sides at various points during the time the system was working, though I think that's inconsequential in respect to my problem.
The problem: yesterday I've noticed that "Status/OpenVPN" on System B shows one of the connections (WAN1) as "down", even though the corresponding connection on Sytem A shows "up" (with the correct "Remote Host" IP adress). The other connection (WAN2) is working fine. The log on System A shows "Peer Connection Initiated with [AF_INET]xx.xxx.xxx.xx:xxxxx" entries roughly every minute, the log on system B first "UDPv4 link remote: [AF_INET]xx.xxx.xxx.xx:xxxx" (with the correct IP adress for System A), and then after some time "Inactivity timeout (–ping-restart), restarting". While this is going on, System A shows the connection as "up" (even "Connected Since" doesn't change), while System B insists on the connection being "down".
I've double- and triple-checked configs on both sides, restarted all machines and infrastructure hardware involved, checked logs, etc.. I still can't see where the problem is. My guess is that I'm missing something to do with the new OpenVPN version on 2.3.4p1, or I'm encountering a pfSense bug that's triggered by my specific setup.
I'm very grateful for any help or comments, especially from other users runnning OpenVPN in a Multi-WAN/Load Balancing setup on 2.3.4p1.
I just saw another system with a down/up openvpn earlier today.
The problem there was one-way traffic.
Traffic could flow from the side that showed down to the site that showed up but traffic could not flow from the site that showed up to the site that showed down.
The tunnel was partially up. Pings sent across from the "down" side would go out the tunnel, be received and replied to by the other side, but would never arrive.
It was a CARP VIP on the down side that the ISP was losing the MAC address for. They would accept traffic from that address but couldn't deliver traffic to it.