Gigabit PPPoE and Intel Drivers



  • Greetings,

    I am considering upgrades to support my new CenturyLink Gigabit Fiber service, which I understand in my area is still using PPPoE.

    After reading some threads at the pfsense boards, SNB boards, etc., it would seem that Intel gigabit cards using the igb driver cannot perform full 900MBit+ Wan-to-Lan speeds due to the bug noted at the link below - UNLESS using a desktop-class processor which can overcome the limitation:

    https://redmine.pfsense.org/issues/4821

    With that being said, I have also found a few less clear pieces of information posted around that indicate similar setups that use older Intel Gigabit cards (e.g. Intel 82574L) which use the em driver instead of igb driver do not seem to exhibit these issues, and can achieve 900Mbit+ Wan-to-Lan. See these links for a couple examples:

    https://kdemaria.wordpress.com/2015/04/22/how-to-configure-pfsense-2-2-2-for-centurylink-gigabit-seattle-edition/ (check comments)

    http://www.dslreports.com/forum/r30270839-Prism-TV-HOWTO-Use-pfSense-with-CenturyLink-FTTH-and-Prism-TV-in-Seattle

    I would ideally like to bypass using the CenturyLink provided router and use a pfSense box directly, but I want to make sure that if I am going to spec something out, it will work.

    Here are some example hardware comparisons I was trying to make before making this post:

    I was looking at this originally:

    http://www.mitxpc.com/proddetail.php?prod=EKJNF9HGM350

    Which seemed it would be fine - modern low power processor and quad-gigabit Intel LAN. But this Intel LAN controller uses the igb driver and this setup will likely not be sufficient - as shown in this thread:

    https://forum.pfsense.org/index.php?topic=104282.0

    An alternative I found:

    http://www.mitxpc.com/proddetail.php?prod=JBC200F9N-E4IN-B

    This has the same processor but uses a daughterboard to add 4x Intel 82574L ports. I am having a hard time deciding if JUST the switch from the igb to em driver will be 'enough' or if this processor will not suffice for PPPoE gigabit regardless of driver.

    Furthermore, I can spec out a mini-ATX board for even cheaper that has these general specs:

    Celeron J1800
    4GB RAM
    2x Intel EXPI9301CTBLK (82574L Controller)

    Anyone know if this will suffice for PPPoE 900Mbit+ Wan-to-LAN?

    Other questions I have:

    1. Is it true that a fast enough single-core performance can overcome the igb limitation?
    2. Does the em driver truly not have this issue?
    3. Can an em driver card achieve 900Mbit+ while using a low-power processor like a bay-trail celeron or atom device? Or will a desktop class proc still be necessary?

    Thanks for reading and your time!



  • 2x Intel EXPI9301CTBLK (82574L Controller)

    Consider a dual port 82571 NIC instead, this also uses the em driver. Most likely cheaper, saves a slot and slightly better 'server' chipset, 82574L is a 'desktop' part.
    Very common on fleabay for <$20 found in parts such as Intel Pro 1000, EXPi9402PT, HP NC360T, Dell X3939, etc. They also come in quad versions if you want more than WAN+LAN.



  • After reading some threads at the pfsense boards, SNB boards, etc., it would seem that Intel gigabit cards using the igb driver cannot perform full 900MBit+ Wan-to-Lan speeds due to the bug noted at the link below - UNLESS using a desktop-class processor which can overcome the limitation:

    This is not really all about as I see it right! The real problem is more related to the circumstance that
    pfSense is actual only using one CPU core at the WAN interface on using the PPPoE, thats it.

    And so that CPU Core should be or must be then strong enough to route the 1 GBit/s throughput
    at the WAN interface of the pfSense firewall. And on top of this must be count;

    • Each TCP/IP traffic is producing overhead and that must be count on top of the measured traffic
    • Working through or out the NAT process and also the firewall rules will also take time and narrows
      down the entire throughput.

    This Jetway board with 8 GB of RAM is able to route ~936 MBit/s (pending up and down) at the WAN
    interface and plus the overhead and processing NAT and firewall rules it is nearly 1 GBit/s in total.

    Routing performance will be not only tended to one point, it is more tended to many points that
    are working together likes the CPU, the RAM speed and the NIC plus the driver support of the NICs.
    The CPU must be strong enough to route 1 GBit/s at the WAN with only one core pending on PPPoE
    If I had to guess, you're being limited also by your speed more than anything else.

    The packet filter, the IP forwarding parts, and even NAT (part of pf, but run at a different phase)
    this all hits the memory system. It's likely not that the CPU can't keep up, it's that your memory
    system is saturated. And so a more modern Intel Atom C2558 or C2758 CPU with DDR3-1600MHz
    RAM and good driver supported Intel NICs as LAN Ports will be the best bet at this time in my eyes.



  • @Aluminum:

    2x Intel EXPI9301CTBLK (82574L Controller)

    Consider a dual port 82571 NIC instead, this also uses the em driver. Most likely cheaper, saves a slot and slightly better 'server' chipset, 82574L is a 'desktop' part.
    Very common on fleabay for <$20 found in parts such as Intel Pro 1000, EXPi9402PT, HP NC360T, Dell X3939, etc. They also come in quad versions if you want more than WAN+LAN.

    Thanks for the recommendation. In that particular napkin example the micro-ATX mobo only had 2 1x PCI-E slots, but if I end up having support for it I will definitely look into a multi-port server card as you suggested.

    @BlueKobold:

    After reading some threads at the pfsense boards, SNB boards, etc., it would seem that Intel gigabit cards using the igb driver cannot perform full 900MBit+ Wan-to-Lan speeds due to the bug noted at the link below - UNLESS using a desktop-class processor which can overcome the limitation:

    This is not really all about as I see it right! The real problem is more related to the circumstance that
    pfSense is actual only using one CPU core at the WAN interface on using the PPPoE, thats it.

    And so that CPU Core should be or must be then strong enough to route the 1 GBit/s throughput
    at the WAN interface of the pfSense firewall. And on top of this must be count;

    • Each TCP/IP traffic is producing overhead and that must be count on top of the measured traffic
    • Working through or out the NAT process and also the firewall rules will also take time and narrows
      down the entire throughput.

    This Jetway board with 8 GB of RAM is able to route ~936 MBit/s (pending up and down) at the WAN
    interface and plus the overhead and processing NAT and firewall rules it is nearly 1 GBit/s in total.

    Routing performance will be not only tended to one point, it is more tended to many points that
    are working together likes the CPU, the RAM speed and the NIC plus the driver support of the NICs.
    The CPU must be strong enough to route 1 GBit/s at the WAN with only one core pending on PPPoE
    If I had to guess, you're being limited also by your speed more than anything else.

    The packet filter, the IP forwarding parts, and even NAT (part of pf, but run at a different phase)
    this all hits the memory system. It's likely not that the CPU can't keep up, it's that your memory
    system is saturated. And so a more modern Intel Atom C2558 or C2758 CPU with DDR3-1600MHz
    RAM and good driver supported Intel NICs as LAN Ports will be the best bet at this time in my eyes.

    For this build I am really only looking for basic router functionality (at least initially). I'm coming from embedded consumer routers (ASUS line) running Tomato firmware so pfSense opens up a lot more possibilities for sure. Initially I am simply trying to get basic NAT, firewall, and WAN PPPoE auth on the pfSense box that supports my new speed.

    When you say "good driver" are you referring to the igb, which appears to be the more modern driver for modern Intel NICs?

    See this board:

    http://www.amazon.com/Supermicro-MiniITX-Retail-Motherboards-MBD-A1SRI-2758F-O/dp/B00FM4M7TQ/ref=sr_1_1?s=electronics&ie=UTF8&qid=1400397161&sr=1-1&keywords=A1SRi-2758F

    This appears to have a quad NIC using the igb driver. In your opinion, would a system with a decent amount of RAM based on this board be able to get maximum speed (e.g. equivalent speed of the CenturyLink provided router for gigabit PPPoE over fiber), considering the PPPoE limitation and basic usage I described above, while still having potential for growth?

    Thanks for your suggestions!



  • 1. Is it true that a fast enough single-core performance can overcome the igb limitation?

    There is no real limitation, but on PPPoE only one CPU core is running and must do the whole work at
    the WAN interface, and this CPU core must be strong enough to handle 1 GBit/s and please don´t forget
    the overhead and the NAT or firewall rule work to count on.

    2. Does the em driver truly not have this issue?

    em driver at the WAN port will be better running pending on the better driver support nothing more or less.
    But if the CPU is not strong enough it is not really interesting what kind of driver is loaded or NIC is in usage.

    3. Can an em driver card achieve 900Mbit+ while using a low-power processor like a bay-trail celeron or atom device? Or will a desktop class proc still be necessary?

    It is like it is, there are many CPU´s that are really powerful enough, to handle this load because e CPU is not
    a CPU core, not all CPUs are at the same level or same strong. There are some you could surely go with;

    • Intel Celeron G3260T @3,2GHz
    • Intel Core i3 & Core i5 @3,0GHz (bigger models)
    • Dual or Quad Core Intel Xeon E3-12xxv3

    The SG-xxxx models from the pfSense store comes with a pre-tuned pfSense image or version and they are
    pretty good, an SG-4860 is able to archive ~500+ MBit/s of IPSec throughput as an example.

    For this build I am really only looking for basic router functionality (at least initially). I'm coming from embedded consumer routers (ASUS line) running Tomato firmware so pfSense opens up a lot more possibilities for sure. Initially I am simply trying to get basic NAT, firewall, and WAN PPPoE auth on the pfSense box that supports my new speed.

    pfSense is a software firewall and not a router only software likes OpenWRT, DD-WRT or RouterOS!
    it is performing and working out firewall rules that can harm the CPU power and narrow down the
    entire speed of the appliance. So it would be better that understand that you will need more horse
    power to archive 1 GBit/s at the WAN interface with a firewall then using Linux based router only software!

    This appears to have a quad NIC using the igb driver. In your opinion, would a system with a decent amount of RAM based on this board be able to get maximum speed (e.g. equivalent speed of the CenturyLink provided router for gigabit PPPoE over fiber), considering the PPPoE limitation and basic usage I described above, while still having potential for growth?

    I was showing up you some CPUs that can handle this speed for sure! And if you want to go with an
    Supermicro Atom C2000 based board and need some more power for future things that can apply or
    would be coming in use, I would be more having a closer look for the C2758 variant, this would be
    future proof within 5+ years and the end is not reached really yet. There are some other things that
    will be coming soon, likes netmap, DPDK, QuickAssist and so the real power from this board will be not
    unleashed fully till today!!! And the usage of only one CPU core will be also not beeing for ever in pfSense
    they are working on this, but I can´t tell you numbers and dates when this would be solved and is away.

    So please except that there are some CPUs at this time that are able to realize and route 1 GBit/s
    but fomr the lower end Atoms, pending on the single CPU core power you could have success or
    not really, this is pending on your personal configuration and not on the century link fiber line.

    If you need a real GBit/s at the WAN port go and buy the card as suggested by @Aluminum
    and go with a Intel Celeron G3260t @3,2GHz and 4 GB or 8 GB of RAM likes your packet installs and use case!

    Or go with a Supermicro C2758 and 8 GB of RAM and fine tune it by your own, and then life with something
    around or nearly ~920 MBit/s - ~936 MBit/s + overhead + NAT & firewall rules on top that is also 1 GBit/s.

    Or at last, go by a SG-4860 that will be sufficient to get this speed out, you can freely mail them or their
    support and you get two free support calls (email) and one you will be able to ask for your tuning and
    pimping to get 1 GBit/s at the WAN port. If not, or to expensive you should think about and go with a
    smaller router and a Linux based router software if you only need SPI & NAT.



  • With a Supermicro A1SRi-2758F board, this is what I had with my ISP, when using PPPoE as WAN:

    • igb0 onboard Intel nic:

    • em0 pci express Intel nic:

    So I threw in this em0-based card just to avoid igb driver with PPPoE combination. It's an 82574L-based one.
    The onboard igb nics are I354-based.



  • Just an FYI, I have the same supermicro a1sri-2758f board with centurylink gigabit fiber and PPPoE.
    I only get about 650-700mbit/s down with igb.  I saw this post and picked up an intel 82574L (Intel EXPI9301CT) adapter and a mini-itx case capable of fitting an external card.

    Now with the em driver running the WAN interface I get:

    I know with the crappy centurylink router they provided I was able to get around 915 when I first tested over a year ago so not sure if the limit is still the CPU bottlenecking or if it's something else.  That said, ~850mbit is pretty decent still.



  • @dopey:

    Just an FYI, I have the same supermicro a1sri-2758f board with centurylink gigabit fiber and PPPoE.
    I only get about 650-700mbit/s down with igb.  I saw this post and picked up an intel 82574L (Intel EXPI9301CT) adapter and a mini-itx case capable of fitting an external card.

    Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.



  • @VAMike:

    Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

    Oh I know.  But this a sunk cost.  I've had the rangeley for a while now before I got gigabit and it really wasn't clear that PPPoE would be an issue until after the fact.  There's no red blinking "WARNING" anywhere :)
    But still, I'm impressed at the difference between em and igb.  It looks like by default, em doesn't use queues at all, which makes me wonder why it's so much faster.



  • @dopey:

    @VAMike:

    Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

    Oh I know.  But this a sunk cost.  I've had the rangeley for a while now before I got gigabit and it really wasn't clear that PPPoE would be an issue until after the fact.  There's no red blinking "WARNING" anywhere :)
    But still, I'm impressed at the difference between em and igb.  It looks like by default, em doesn't use queues at all, which makes me wonder why it's so much faster.

    rss queues don't work with pppoe because the nic can't find the unique IPs inside of the pppoe packets. so they all get dumped in the first queue anyway. it used to be that the problem was compounded because since the packets came through a mechanism which supposedly hashed them, they had a flowid associated and wouldn't get rehashed and redistributed once they were decapsulated (whereas stuff coming in on a card without rss gets a new flow ids assigned). I think that's still the case, which is why igb (or any multiqueue card) is worse than a single queue card.



  • Aah I thought it might be the overhead of the hashing in an attempt to queue it.    That makes sense.  Thanks.



  • There are some updates in FreeBSD bug report — https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856
    Can someone test this possible solution suggested?
    In terminal do

    sysctl net.isr.dispatch=deferred
    

    Try some gigabit tests, like dslreports or whatever. Check for your speeds and report it here, please.



  • @w0w That did it for me.


  • Netgate Administrator

    What did it do? Got you up to Gigabit line rate over PPPoE?

    What speed were you seeing before/

    Steve



  • I see very little difference with the net.isr.dispatch change. Ever since the spectre/meltdown bios update I'm barely cracking 650 with my c2758.

    Anyone know if denverton is more capable for pppoe or do I really need to go into core series CPU?

    I'm really looking for low tdp (preferably fanless) and ipmi and quad nic. I've found its nearly impossible to guarantee finding a non counterfeit Intel nic aftermarket without paying more than the CPU/motherboard for it :)



  • Looking at the benchmarks it doesn't look like denverton is any faster than avaton. More power efficient but that's it. So denverton likely won't fare much better.



  • @dopey
    Did you restart firewall after change applied?
    Do you have the same result on your em card?



  • Oh duh!! I didn't switch back to the on igb NIC after making the change. I'll try that when I get a chance.



  • @w0w
    But at least it looks you have some performance drop on em card also after some changes? Is it spectre/meltdown patch?



  • Yeah, the spectre/meltdown update coincided with a pretty big drop in performance with the em driver.



  • Did a few more tests.
    With em driver
    net.isr.dispatch=deferred
    700-800mbps

    net.isr.dispatch=direct
    675-715
    most of the tests seem around 700 give or take a few

    With igb
    net.isr.dispatch=direct
    500-600mbps

    net.isr.dispatch=deferred
    650-700

    So net.isr.dispatch in both cases made a difference, but still shy of the 920 or so I should be pulling.


  • Netgate Administrator

    You can disable the Kernel PTI workaround for Meltdown in System > Advanced > Misc. You almost certainly don't need it anyway unless you are running virtual.

    The IBRS workaround for Spectre may not be active anyway but you can disable that too with the loader tunable:
    hw.ibrs_disable=1

    https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities

    Steve



  • @stephenw10 said in Gigabit PPPoE and Intel Drivers:

    You can disable the Kernel PTI workaround for Meltdown in System > Advanced > Misc. You almost certainly don't need it anyway unless you are running virtual.

    That's not correct. You need to mitigate meltdown unless you are 100% confident that there is no need for privilege separation on a system. (E.g., if you have no reason to run a web service as something other than root, or run pre-auth ssh code as an unprivileged user, etc.) If you use privilege separation as a mitigation for other vulnerabilities (e.g., bug in web script, bug in ssh, etc.) then you need meltdown mitigation in order for the privilege separation to actually be meaningful. Other speculative execution bugs like L1TF-VMM (CVE-2018-3646) are specific to virtual machines.



  • @vamike that would only really apply if there's any ability to execute malicious code within the privilege separated processes right? If the router is locked down so only trusted individuals can to access it and there are no available vulnerablities (big IF I know) there's should be no way someone can take advantage of the vulnerablities.

    I know there was some grumblings of a remote spectre like exposure but I don't know if that applies to routers.



  • @dopey said in Gigabit PPPoE and Intel Drivers:

    @vamike that would only really apply if there's any ability to execute malicious code within the privilege separated processes right? If the router is locked down so only trusted individuals can to access it and there are no available vulnerablities (big IF I know) there's should be no way someone can take advantage of the vulnerablities.

    Sure. Like any other mitigation, it's a risk based decision. OTOH, if you can be sure that you can lock things down and never have a vulnerability, why are you running a firewall at all?


  • Netgate Administrator

    Mmm, interesting. Some stuff I had not considered there.

    Anyway you can test it and see if it improves performance by any useful amount. If not leave it enabled.

    Steve



  • I'd expect the spectre mitigations to be more costly than meltdown, and arguably less relevant.



  • @vamike looking at the processes running on my router, unbound and dhcpd are the only two things not running as root. So given that it seems that avoiding meltdown/spectre on a native bare-metal install is fine. Anything that can take advantage of meltdown or spectre would likely simply take advantage of being root.



  • Kernel PTI disabled and net.isr.dispatch=deferred

    https://www.speedtest.net/result/7815707411

    A little bit better than I was getting before the meltdown patch with dispatch=direct

    Not too shabby.