That's part of why it's still just a patch and not in the code. It helps some situations, but not others. That said, you can't really have it both ways. It can either restart the packages or not restart the packages.
The reason it has to restart is typically what you have seen -- FRR needs to restart to latch back onto an interface which has changed (or probably deleted and recreated).
IPsec may fare better because of the changes I made in https://redmine.pfsense.org/issues/9668 which cycles the interface in FRR live without restarting the package. I'm not sure if a similar change for OpenVPN would be viable.