For anyone else who has this idea, I think it's a bigger pain than is warranted. I did more reading on libtorrent. The ports in the normal option menu of torrent clients are listen ports. When a connection is made and you seed a torrent, libtorrent uses dynamic outbound ports. You can set these as static, often using obscure options, but the libtorrent devs suggest not doing so as it can cause issues with establishing connections.
So instead of doing all that and possibly having connection issues I will just be containerizing the torrent client on my server, using macvlan to give it a dedicated IP on my LAN, then routing that IP over the VPN interface using PBR.
As for the other torrent clients on random computers on my LAN, it's probably best we stop using those and just use my server's client. Or we can use the VPN client on those computers.