@BlueKobold:
Where the ISP-provided cheap modem in that case is faster, can reach near the gigabit speeds..
In normal a modem is not doing SPI, NAT and passing firewall rules! So for sure the ordanary modem
must be even faster, this is a must be!
Not true, the modem is doing NAT in that case. And it has some basic SPI functionality as well. Keep in mind we're talking about single-stream throughput, the easiest case. I don't know how well those would stand up across much larger numbers of simultaneous connections.
@BlueKobold:
….where a 4860 tops out at 700ish Mbps on PPPoE in that circumstance because it's stuck to one core.
With SPI and NAT you will loose even something around 3% - 5% of the whole throughput, depending on your hardware for sure. But often with other hardware I really think this would be not the end of the line, with a
Xeon E3-1286v3, muchECC RAM and an intel server network adapter it would be also able to archive more throughput, but also holding the level of security!
Sure, in that case all you need is a CPU that has faster cores, so a single core can handle a higher traffic rate. I have no doubt a new Xeon would easily max out a 1 Gbps link in the PPPoE scenario (at least with large-ish packets, not at purely 64 byte frames). But that's also an unreasonably expensive firewall/router box for home and SMB uses.
It's not true in general that you'll lose any throughput from SPI or NAT, as long as your system's adequately fast for your connection speed. We're talking microseconds of processing time from arrival of a packet on the LAN NIC to it exiting the WAN NIC, as long as there is CPU capacity to spare. That's such a tiny portion of your latency to any Internet destination it has no measurable impact. It's far less than just the jitter to close Internet destinations on high quality connectivity.