Which VPN service are you using? Almost all mainstream providers offer a split tunneling feature that allows you to choose which data to send through the VPN and which not. I use PureVPN but many others like ExpressVPN offer the same with their apps.
It is unusual, but it's the standard Comcast setup when you have a business account with static public IPs. For residential, or lower-tier business accounts, you get a dynamic public IP. I'm talking about v4, but they are now providing a static v6 block with the v4, and a residential user gets a dynamic /60.
Closing this. Thanks for pointing me into the direction of testing the Ping on the CARP VIP. That ended up being the issue. Turns out somehow ISP took back one of our 3 IPs, we got them to put it back on our account and now we are back to normal. Can ping off that CARP VIP as well as port forwarding works now using the CARP VIP as Destination Address.