Gigabit PPPoE and Intel Drivers


  • Netgate Administrator

    What did it do? Got you up to Gigabit line rate over PPPoE?

    What speed were you seeing before/

    Steve



  • I see very little difference with the net.isr.dispatch change. Ever since the spectre/meltdown bios update I'm barely cracking 650 with my c2758.

    Anyone know if denverton is more capable for pppoe or do I really need to go into core series CPU?

    I'm really looking for low tdp (preferably fanless) and ipmi and quad nic. I've found its nearly impossible to guarantee finding a non counterfeit Intel nic aftermarket without paying more than the CPU/motherboard for it :)



  • Looking at the benchmarks it doesn't look like denverton is any faster than avaton. More power efficient but that's it. So denverton likely won't fare much better.



  • @dopey
    Did you restart firewall after change applied?
    Do you have the same result on your em card?



  • Oh duh!! I didn't switch back to the on igb NIC after making the change. I'll try that when I get a chance.



  • @w0w
    But at least it looks you have some performance drop on em card also after some changes? Is it spectre/meltdown patch?



  • Yeah, the spectre/meltdown update coincided with a pretty big drop in performance with the em driver.



  • Did a few more tests.
    With em driver
    net.isr.dispatch=deferred
    700-800mbps

    net.isr.dispatch=direct
    675-715
    most of the tests seem around 700 give or take a few

    With igb
    net.isr.dispatch=direct
    500-600mbps

    net.isr.dispatch=deferred
    650-700

    So net.isr.dispatch in both cases made a difference, but still shy of the 920 or so I should be pulling.


  • Netgate Administrator

    You can disable the Kernel PTI workaround for Meltdown in System > Advanced > Misc. You almost certainly don't need it anyway unless you are running virtual.

    The IBRS workaround for Spectre may not be active anyway but you can disable that too with the loader tunable:
    hw.ibrs_disable=1

    https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities

    Steve



  • @stephenw10 said in Gigabit PPPoE and Intel Drivers:

    You can disable the Kernel PTI workaround for Meltdown in System > Advanced > Misc. You almost certainly don't need it anyway unless you are running virtual.

    That's not correct. You need to mitigate meltdown unless you are 100% confident that there is no need for privilege separation on a system. (E.g., if you have no reason to run a web service as something other than root, or run pre-auth ssh code as an unprivileged user, etc.) If you use privilege separation as a mitigation for other vulnerabilities (e.g., bug in web script, bug in ssh, etc.) then you need meltdown mitigation in order for the privilege separation to actually be meaningful. Other speculative execution bugs like L1TF-VMM (CVE-2018-3646) are specific to virtual machines.



  • @vamike that would only really apply if there's any ability to execute malicious code within the privilege separated processes right? If the router is locked down so only trusted individuals can to access it and there are no available vulnerablities (big IF I know) there's should be no way someone can take advantage of the vulnerablities.

    I know there was some grumblings of a remote spectre like exposure but I don't know if that applies to routers.



  • @dopey said in Gigabit PPPoE and Intel Drivers:

    @vamike that would only really apply if there's any ability to execute malicious code within the privilege separated processes right? If the router is locked down so only trusted individuals can to access it and there are no available vulnerablities (big IF I know) there's should be no way someone can take advantage of the vulnerablities.

    Sure. Like any other mitigation, it's a risk based decision. OTOH, if you can be sure that you can lock things down and never have a vulnerability, why are you running a firewall at all?


  • Netgate Administrator

    Mmm, interesting. Some stuff I had not considered there.

    Anyway you can test it and see if it improves performance by any useful amount. If not leave it enabled.

    Steve



  • I'd expect the spectre mitigations to be more costly than meltdown, and arguably less relevant.



  • @vamike looking at the processes running on my router, unbound and dhcpd are the only two things not running as root. So given that it seems that avoiding meltdown/spectre on a native bare-metal install is fine. Anything that can take advantage of meltdown or spectre would likely simply take advantage of being root.



  • Kernel PTI disabled and net.isr.dispatch=deferred

    https://www.speedtest.net/result/7815707411

    A little bit better than I was getting before the meltdown patch with dispatch=direct

    Not too shabby.



  • @w0w said in Gigabit PPPoE and Intel Drivers:

    There are some updates in FreeBSD bug report — https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856
    Can someone test this possible solution suggested?
    In terminal do

    sysctl net.isr.dispatch=deferred
    

    Try some gigabit tests, like dslreports or whatever. Check for your speeds and report it here, please.

    Thanks, that did it for me, allowing me to almost double my Rx on my realtek Nics on my Zotac box.. So glad, getting ~ 750/750 now.. which is good enough for now.



  • @doboy
    If you read back a few posts you'll see some of my experiences. I actually had to disable the meltdown fixes as well to get back to close to what I was getting before.



  • Hello,
    I have been hesitant for months not to build my own pfsense router because of this bug but i'm posting now results and breaking the ice.

    my test rig:

    Intel 2 Duo CPU E4500 @ 2.20GHz
    4GB Ram

    I have bought by mistake a quad nic with igb driver ( HP NC365T 1Gb PCI-E 593720-001 ). I was surprised of getting full speeds:

    alt text
    alt text
    alt text

    I'm getting 50% CPU usage on 1gbit download, 99% with suricata on WAN.

    So for everyone rest assure it works, it load balances over all cores.

    I also tested with some cheapo Realtek 1gbit RTL8111F and getting almost the same results.

    I have to mention i have made the tunning that netgate recommends here: https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html

    kern.ipc.nmbclusters 1000000
    net.isr.dispatch deferred
    net.isr.maxthreads 2
    net.isr.numthreads 2
    hw.pci.enable_msix 0
    hw.pci.enable_msi 0


  • Netgate Administrator

    50% CPU usage on that dual core CPU is probably 100% on one core if you;re checking on the dashboard.
    Suricata can use the other core bringing it up to 100% total. You would have to check at the command line to see the CPU usage breakdown: top -aSH.

    That is more that I would have thought but the single thread rating of the E4500 is significantly higher than, say, the J1900 that has been seen to be limited to ~500Mbps PPPoE. Though those Celerons seem particularly effected by this for some reason.

    Steve


Log in to reply