Gigabit performance issues
I'm new to pfsense, and was hoping to run it in a VM as a router to replace my older dd wrt setups now that I got a new fiber internet connection. My connection is symmetric gigabit from ATT with a 5268AC and a /29 of static IPs (which I'm also new to, but a main reason I wanted to switch to pfsense)
I've setup pfsense with virtualbox, and tested my internet speeds, and going through pfsense, my upload and download speeds seem constricted to the mid to low 100 Mbps. I can't seem to figure out why, its not memory bound, its not CPU bound during my tests. I'm running on a core i7 3GHz, and I've tried 2 seperate computers with a ubuntu host and a windows 10 host. I've tested with the I've searched google and read tons of stuff on here about these issues, but from what I gather, my performance shouldn't be this bad and I can't figure out any explanation or what to do. Switching a computer from the pfsense box directly to the RG immediately gives it back the 900+ Mbps upload/download on speedtest, so it doesn't seem to be any ATT flakiness.
For the numbers… using speedtest connected directly to the RG gives me 900+ Mbps upload/download. Using speedtest-cli on the pfsense box seems to get me in the 300 Mbps range, losing 60%+ of my bandwidth, but roughly double the speeds of what it delivers to the clients, which gets usually around 150 Mbps. Using iperf between pfsense and the clients gets me around 650 Mbps (which I still feel should be better, no?). During none of these tests does the pfsense CPU seem to go above about 70% on any core.
What I've done so far that has led to small, minimal improvements is switch to the paravirtualized drivers from the Intel ones. My onboard NICs are Realtek. Increasing the CPU/memory allocation to pfsense has led to no improvements. Where am I going wrong? Would switching to another virtualization platform help? i'm just so confused, as it doesn't seem I'm doing any encryption and it just seems network speed is bottlenecking, which others seem to do without problems. I'd like to avoid extra hardware running 24/7 if I can, which is why I'm trying to virtualize it. I do have a spare core i7 lying around, is performance gonna be much better if I install pfsense directly on that or am I running into something else?
I'm not trying to use any advanced packages, no VPN, and have reset pfsense to the default configuration to try to figure out this issue. I've tried the ATT RG with both factory default settings and DMZ+ and cascaded router so far, all with no change at all in performance on the boxes behind pfsense. I was thinking about trying the gateway bypass methods, but I have a feeling thats not going to help me and just complicate things till I can figure this out. I've tried disabling all hardware offloading options as well, which seems to be a popular suggestion, but no difference there at all.
I decided to try installing pfsense on Hyper V on Win 10 Pro and I got a HUGE boost in speed. I seem to get around 600 Mbps download and 900 Mbps upload. My understanding is the loss in upload speed is in the normal range, but the download still seems like abnormal loss (but far better then the 90% loss before), as soon as I take it off pfsense it is always 900 Mbps+. I just can't find what is bottlenecking. Is there anything for me to check in that range? I also think its really weird that now I have upload working pretty normally, and download is experiencing a 300 Mbps loss.
I'd also prefer to use Ubuntu as the host system, if anyone has any ideas on why Virtualbox performance is sooo atrocious on reasonable hardware?