@areynot Are you using queues under your limiters? I believe that you want one or more upload queues with the source mask set to /32 and one or more download queues with the destination mask set to /32. My understanding - and it may be flawed - is that doing so will result in each host on your LAN being assigned one upload queue and one download queue. So consider the simplest case where you make one download queue for your download limiter and one upload queue for your upload limiter, and then just assign every host on your LAN to use these queues. If you have N hosts on your network, then the most active queues you should have at any given time is N download queues and N upload queues, and your download and upload bandwidth should be shared roughly evenly across them.
At a very basic level, though, limiters (pipes) establish a cap on your bandwidth, but queues assigned to those limiters determine how that bandwidth is shared among multiple hosts.