Latency is only affected because of bufferbloat. You could try to limit everyone's bandwidth, but fixing the bufferbloat can get you the 80/20 with little effort, fewer edge cases, and reduced complexity.
The currently simplest way is to enable FairQ as the shaper on LAN and WAN interfaces, configure the default queue on the interfaces to have Codel enabled, and to set the bandwidth to some value less than what real bandwidth you have.
In the near future, scheduled for 2.4.4, fq_Codel should be superior and easier to setup.
This is just an alternative that you may want to try.