Just to close the loop on this - I re-enabled pf with any/any inbound and outbound with NAT disabled and have not found any resulting issues.
From a performance perspective, I saw about a 50% performance hit in throughput. Luckily, I'm running this instance as a VM so by adding a second core to this instance, I'm back to near wire speed with pf running.
ESXi 7 on AMD Ryzen 5 3600 CPU if anyone is interested.
Thanks for the replies on this.