2WANs + 1 ISP + 2 NICs + 2 NAT VMs + 1 pfSense VM == ???

[asdlkf]

Proposal:

Linux + VMWare Server for linux

                           x.x.10.1                          x.x.1.100
        WAN_IP_1  eth0 Box1 eth1                             lan client
                  |            |                                |
          VMWare  |      VMWare|  x.x.10.2     x.x.1.1       physical x.x.1.101
 Modem----switch--|       switch-----eth0 Box3 eth1----------switch---Lan Client
                  |            |  x.x.20.2                      |
                  |            |                                |
        WAN_IP_2  eth0 Box2 eth1                             lan client
                           x.x.20.1                          x.x.1.102

Basically:
Box 1 is a smoothwall NAT box (or any other basic nat divice with a dhcp client for the wan connection)
Box 2 is a smoothwall NAT box (or any other basic nat divice with a dhcp client for the wan connection)
Box 3 is some sort of bandwidth splitter/router/?????/pfsense///??? im not sure

I would like to either round-robin between box1 and box2 or ideally have layer 4 routing

If pfSense can do layer 4 routing, i would like to be able to have the following basic rules:
SSH connections: box 1
DNS and all other 'quick-burst' connections on box 1 (http, https)
VoiP and other streaming user-experience-affected protocols on box 1
Latancy sensative applications (FPS games (all multiplayer games really)) on box 1

Large file transfers on box 2
bit torrent on box 2
anything else that would 'clog' a regular single-ip connection: box 2

is this possible? can anyone point me in the right direction?

(i currently have the dual nat setup functional, but i have to configure my clients with one gateway at a time so i basically end up with a bit torrent computer and a gaming computer which really sucks)

[cmb]

You don't need 3 boxes. Unless you find something that's specifically a "bandwidth splitter" to use in one of those places. One pfsense install could replace all 3 of those boxes in the diagram, with some caveats.

You can do policy routing based on protocol from L4 information, but some protocols are going to be difficult to route based on that because they use so many ports, like Bittorrent.

"large file transfers" don't look any different at L4 than small file transfers with the same protocol, so that may be a problem.

"anything else that would 'clog' a regular single-ip connection" - basically impossible to detect at L4, at least as far as policy routing is concerned.

Some of what you're describing would work fine on pfsense, other things would be difficult to do well with any device, though some pricey enterprise class routing/load balancing equipment may be able to tackle it all.