You don't need 3 boxes. Unless you find something that's specifically a "bandwidth splitter" to use in one of those places. One pfsense install could replace all 3 of those boxes in the diagram, with some caveats.
You can do policy routing based on protocol from L4 information, but some protocols are going to be difficult to route based on that because they use so many ports, like Bittorrent.
"large file transfers" don't look any different at L4 than small file transfers with the same protocol, so that may be a problem.
"anything else that would 'clog' a regular single-ip connection" - basically impossible to detect at L4, at least as far as policy routing is concerned.
Some of what you're describing would work fine on pfsense, other things would be difficult to do well with any device, though some pricey enterprise class routing/load balancing equipment may be able to tackle it all.