@viragomann I think therefore the more secure option is as you suggest to setup OpenVPN server on the pfSense box and see if I can use the mini router as a OpenVPN "client".

@JKnott It is unusual, but it's the standard Comcast setup when you have a business account with static public IPs. For residential, or lower-tier business accounts, you get a dynamic public IP. I'm talking about v4, but they are now providing a static v6 block with the v4, and a residential user gets a dynamic /60.

@Gertjan shodan.io is a service that scans the internet for known exposure and for vulnerabilities i remember you are french, so I link you here a video in French on the subject https://youtu.be/SxjmOFBtsvk

Closing this. Thanks for pointing me into the direction of testing the Ping on the CARP VIP. That ended up being the issue. Turns out somehow ISP took back one of our 3 IPs, we got them to put it back on our account and now we are back to normal. Can ping off that CARP VIP as well as port forwarding works now using the CARP VIP as Destination Address. Thanks again @Derelict