Gigabit PPPoE and Intel Drivers

nojp

Greetings,

I am considering upgrades to support my new CenturyLink Gigabit Fiber service, which I understand in my area is still using PPPoE.

After reading some threads at the pfsense boards, SNB boards, etc., it would seem that Intel gigabit cards using the igb driver cannot perform full 900MBit+ Wan-to-Lan speeds due to the bug noted at the link below - UNLESS using a desktop-class processor which can overcome the limitation:

https://redmine.pfsense.org/issues/4821

With that being said, I have also found a few less clear pieces of information posted around that indicate similar setups that use older Intel Gigabit cards (e.g. Intel 82574L) which use the em driver instead of igb driver do not seem to exhibit these issues, and can achieve 900Mbit+ Wan-to-Lan. See these links for a couple examples:

https://kdemaria.wordpress.com/2015/04/22/how-to-configure-pfsense-2-2-2-for-centurylink-gigabit-seattle-edition/ (check comments)

http://www.dslreports.com/forum/r30270839-Prism-TV-HOWTO-Use-pfSense-with-CenturyLink-FTTH-and-Prism-TV-in-Seattle

I would ideally like to bypass using the CenturyLink provided router and use a pfSense box directly, but I want to make sure that if I am going to spec something out, it will work.

Here are some example hardware comparisons I was trying to make before making this post:

I was looking at this originally:

http://www.mitxpc.com/proddetail.php?prod=EKJNF9HGM350

Which seemed it would be fine - modern low power processor and quad-gigabit Intel LAN. But this Intel LAN controller uses the igb driver and this setup will likely not be sufficient - as shown in this thread:

https://forum.pfsense.org/index.php?topic=104282.0

An alternative I found:

http://www.mitxpc.com/proddetail.php?prod=JBC200F9N-E4IN-B

This has the same processor but uses a daughterboard to add 4x Intel 82574L ports. I am having a hard time deciding if JUST the switch from the igb to em driver will be 'enough' or if this processor will not suffice for PPPoE gigabit regardless of driver.

Furthermore, I can spec out a mini-ATX board for even cheaper that has these general specs:

Celeron J1800
4GB RAM
2x Intel EXPI9301CTBLK (82574L Controller)

Anyone know if this will suffice for PPPoE 900Mbit+ Wan-to-LAN?

Other questions I have:

1. Is it true that a fast enough single-core performance can overcome the igb limitation?
2. Does the em driver truly not have this issue?
3. Can an em driver card achieve 900Mbit+ while using a low-power processor like a bay-trail celeron or atom device? Or will a desktop class proc still be necessary?

Thanks for reading and your time!

Aluminum

2x Intel EXPI9301CTBLK (82574L Controller)

Consider a dual port 82571 NIC instead, this also uses the em driver. Most likely cheaper, saves a slot and slightly better 'server' chipset, 82574L is a 'desktop' part.
Very common on fleabay for <$20 found in parts such as Intel Pro 1000, EXPi9402PT, HP NC360T, Dell X3939, etc. They also come in quad versions if you want more than WAN+LAN.

Guest

After reading some threads at the pfsense boards, SNB boards, etc., it would seem that Intel gigabit cards using the igb driver cannot perform full 900MBit+ Wan-to-Lan speeds due to the bug noted at the link below - UNLESS using a desktop-class processor which can overcome the limitation:

This is not really all about as I see it right! The real problem is more related to the circumstance that
pfSense is actual only using one CPU core at the WAN interface on using the PPPoE, thats it.

And so that CPU Core should be or must be then strong enough to route the 1 GBit/s throughput
at the WAN interface of the pfSense firewall. And on top of this must be count;

Each TCP/IP traffic is producing overhead and that must be count on top of the measured traffic
Working through or out the NAT process and also the firewall rules will also take time and narrows
down the entire throughput.

This Jetway board with 8 GB of RAM is able to route ~936 MBit/s (pending up and down) at the WAN
interface and plus the overhead and processing NAT and firewall rules it is nearly 1 GBit/s in total.

Routing performance will be not only tended to one point, it is more tended to many points that
are working together likes the CPU, the RAM speed and the NIC plus the driver support of the NICs.
The CPU must be strong enough to route 1 GBit/s at the WAN with only one core pending on PPPoE
If I had to guess, you're being limited also by your speed more than anything else.

The packet filter, the IP forwarding parts, and even NAT (part of pf, but run at a different phase)
this all hits the memory system. It's likely not that the CPU can't keep up, it's that your memory
system is saturated. And so a more modern Intel Atom C2558 or C2758 CPU with DDR3-1600MHz
RAM and good driver supported Intel NICs as LAN Ports will be the best bet at this time in my eyes.

nojp

@Aluminum:

2x Intel EXPI9301CTBLK (82574L Controller)

Consider a dual port 82571 NIC instead, this also uses the em driver. Most likely cheaper, saves a slot and slightly better 'server' chipset, 82574L is a 'desktop' part.
Very common on fleabay for <$20 found in parts such as Intel Pro 1000, EXPi9402PT, HP NC360T, Dell X3939, etc. They also come in quad versions if you want more than WAN+LAN.

Thanks for the recommendation. In that particular napkin example the micro-ATX mobo only had 2 1x PCI-E slots, but if I end up having support for it I will definitely look into a multi-port server card as you suggested.

@BlueKobold:

After reading some threads at the pfsense boards, SNB boards, etc., it would seem that Intel gigabit cards using the igb driver cannot perform full 900MBit+ Wan-to-Lan speeds due to the bug noted at the link below - UNLESS using a desktop-class processor which can overcome the limitation:

This is not really all about as I see it right! The real problem is more related to the circumstance that
pfSense is actual only using one CPU core at the WAN interface on using the PPPoE, thats it.

And so that CPU Core should be or must be then strong enough to route the 1 GBit/s throughput
at the WAN interface of the pfSense firewall. And on top of this must be count;

Each TCP/IP traffic is producing overhead and that must be count on top of the measured traffic

Working through or out the NAT process and also the firewall rules will also take time and narrows
down the entire throughput.

This Jetway board with 8 GB of RAM is able to route ~936 MBit/s (pending up and down) at the WAN
interface and plus the overhead and processing NAT and firewall rules it is nearly 1 GBit/s in total.

Routing performance will be not only tended to one point, it is more tended to many points that
are working together likes the CPU, the RAM speed and the NIC plus the driver support of the NICs.
The CPU must be strong enough to route 1 GBit/s at the WAN with only one core pending on PPPoE
If I had to guess, you're being limited also by your speed more than anything else.

The packet filter, the IP forwarding parts, and even NAT (part of pf, but run at a different phase)
this all hits the memory system. It's likely not that the CPU can't keep up, it's that your memory
system is saturated. And so a more modern Intel Atom C2558 or C2758 CPU with DDR3-1600MHz
RAM and good driver supported Intel NICs as LAN Ports will be the best bet at this time in my eyes.

For this build I am really only looking for basic router functionality (at least initially). I'm coming from embedded consumer routers (ASUS line) running Tomato firmware so pfSense opens up a lot more possibilities for sure. Initially I am simply trying to get basic NAT, firewall, and WAN PPPoE auth on the pfSense box that supports my new speed.

When you say "good driver" are you referring to the igb, which appears to be the more modern driver for modern Intel NICs?

See this board:

http://www.amazon.com/Supermicro-MiniITX-Retail-Motherboards-MBD-A1SRI-2758F-O/dp/B00FM4M7TQ/ref=sr_1_1?s=electronics&ie=UTF8&qid=1400397161&sr=1-1&keywords=A1SRi-2758F

This appears to have a quad NIC using the igb driver. In your opinion, would a system with a decent amount of RAM based on this board be able to get maximum speed (e.g. equivalent speed of the CenturyLink provided router for gigabit PPPoE over fiber), considering the PPPoE limitation and basic usage I described above, while still having potential for growth?

Thanks for your suggestions!

Guest

1. Is it true that a fast enough single-core performance can overcome the igb limitation?

There is no real limitation, but on PPPoE only one CPU core is running and must do the whole work at
the WAN interface, and this CPU core must be strong enough to handle 1 GBit/s and please don´t forget
the overhead and the NAT or firewall rule work to count on.

2. Does the em driver truly not have this issue?

em driver at the WAN port will be better running pending on the better driver support nothing more or less.
But if the CPU is not strong enough it is not really interesting what kind of driver is loaded or NIC is in usage.

3. Can an em driver card achieve 900Mbit+ while using a low-power processor like a bay-trail celeron or atom device? Or will a desktop class proc still be necessary?

It is like it is, there are many CPU´s that are really powerful enough, to handle this load because e CPU is not
a CPU core, not all CPUs are at the same level or same strong. There are some you could surely go with;

Intel Celeron G3260T @3,2GHz
Intel Core i3 & Core i5 @3,0GHz (bigger models)
Dual or Quad Core Intel Xeon E3-12xxv3

The SG-xxxx models from the pfSense store comes with a pre-tuned pfSense image or version and they are
pretty good, an SG-4860 is able to archive ~500+ MBit/s of IPSec throughput as an example.

For this build I am really only looking for basic router functionality (at least initially). I'm coming from embedded consumer routers (ASUS line) running Tomato firmware so pfSense opens up a lot more possibilities for sure. Initially I am simply trying to get basic NAT, firewall, and WAN PPPoE auth on the pfSense box that supports my new speed.

pfSense is a software firewall and not a router only software likes OpenWRT, DD-WRT or RouterOS!
it is performing and working out firewall rules that can harm the CPU power and narrow down the
entire speed of the appliance. So it would be better that understand that you will need more horse
power to archive 1 GBit/s at the WAN interface with a firewall then using Linux based router only software!

This appears to have a quad NIC using the igb driver. In your opinion, would a system with a decent amount of RAM based on this board be able to get maximum speed (e.g. equivalent speed of the CenturyLink provided router for gigabit PPPoE over fiber), considering the PPPoE limitation and basic usage I described above, while still having potential for growth?

I was showing up you some CPUs that can handle this speed for sure! And if you want to go with an
Supermicro Atom C2000 based board and need some more power for future things that can apply or
would be coming in use, I would be more having a closer look for the C2758 variant, this would be
future proof within 5+ years and the end is not reached really yet. There are some other things that
will be coming soon, likes netmap, DPDK, QuickAssist and so the real power from this board will be not
unleashed fully till today!!! And the usage of only one CPU core will be also not beeing for ever in pfSense
they are working on this, but I can´t tell you numbers and dates when this would be solved and is away.

So please except that there are some CPUs at this time that are able to realize and route 1 GBit/s
but fomr the lower end Atoms, pending on the single CPU core power you could have success or
not really, this is pending on your personal configuration and not on the century link fiber line.

If you need a real GBit/s at the WAN port go and buy the card as suggested by @Aluminum
and go with a Intel Celeron G3260t @3,2GHz and 4 GB or 8 GB of RAM likes your packet installs and use case!

Or go with a Supermicro C2758 and 8 GB of RAM and fine tune it by your own, and then life with something
around or nearly ~920 MBit/s - ~936 MBit/s + overhead + NAT & firewall rules on top that is also 1 GBit/s.

Or at last, go by a SG-4860 that will be sufficient to get this speed out, you can freely mail them or their
support and you get two free support calls (email) and one you will be able to ask for your tuning and
pimping to get 1 GBit/s at the WAN port. If not, or to expensive you should think about and go with a
smaller router and a Linux based router software if you only need SPI & NAT.

robi

With a Supermicro A1SRi-2758F board, this is what I had with my ISP, when using PPPoE as WAN:

igb0 onboard Intel nic:
em0 pci express Intel nic:

So I threw in this em0-based card just to avoid igb driver with PPPoE combination. It's an 82574L-based one.
The onboard igb nics are I354-based.

dopey

Just an FYI, I have the same supermicro a1sri-2758f board with centurylink gigabit fiber and PPPoE.
I only get about 650-700mbit/s down with igb. I saw this post and picked up an intel 82574L (Intel EXPI9301CT) adapter and a mini-itx case capable of fitting an external card.

Now with the em driver running the WAN interface I get:

I know with the crappy centurylink router they provided I was able to get around 915 when I first tested over a year ago so not sure if the limit is still the CPU bottlenecking or if it's something else. That said, ~850mbit is pretty decent still.

VAMike

@dopey:

Just an FYI, I have the same supermicro a1sri-2758f board with centurylink gigabit fiber and PPPoE.
I only get about 650-700mbit/s down with igb. I saw this post and picked up an intel 82574L (Intel EXPI9301CT) adapter and a mini-itx case capable of fitting an external card.

Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

dopey

@VAMike:

Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

Oh I know. But this a sunk cost. I've had the rangeley for a while now before I got gigabit and it really wasn't clear that PPPoE would be an issue until after the fact. There's no red blinking "WARNING" anywhere :)
But still, I'm impressed at the difference between em and igb. It looks like by default, em doesn't use queues at all, which makes me wonder why it's so much faster.

VAMike

@dopey:

@VAMike:

Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

Oh I know. But this a sunk cost. I've had the rangeley for a while now before I got gigabit and it really wasn't clear that PPPoE would be an issue until after the fact. There's no red blinking "WARNING" anywhere :)
But still, I'm impressed at the difference between em and igb. It looks like by default, em doesn't use queues at all, which makes me wonder why it's so much faster.

rss queues don't work with pppoe because the nic can't find the unique IPs inside of the pppoe packets. so they all get dumped in the first queue anyway. it used to be that the problem was compounded because since the packets came through a mechanism which supposedly hashed them, they had a flowid associated and wouldn't get rehashed and redistributed once they were decapsulated (whereas stuff coming in on a card without rss gets a new flow ids assigned). I think that's still the case, which is why igb (or any multiqueue card) is worse than a single queue card.

dopey

Aah I thought it might be the overhead of the hashing in an attempt to queue it. That makes sense. Thanks.

w0w

There are some updates in FreeBSD bug report — https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856
Can someone test this possible solution suggested?
In terminal do

sysctl net.isr.dispatch=deferred

Try some gigabit tests, like dslreports or whatever. Check for your speeds and report it here, please.

mwave

@w0w That did it for me.

stephenw10

What did it do? Got you up to Gigabit line rate over PPPoE?

What speed were you seeing before/

Steve

dopey

I see very little difference with the net.isr.dispatch change. Ever since the spectre/meltdown bios update I'm barely cracking 650 with my c2758.

Anyone know if denverton is more capable for pppoe or do I really need to go into core series CPU?

I'm really looking for low tdp (preferably fanless) and ipmi and quad nic. I've found its nearly impossible to guarantee finding a non counterfeit Intel nic aftermarket without paying more than the CPU/motherboard for it :)

dopey

Looking at the benchmarks it doesn't look like denverton is any faster than avaton. More power efficient but that's it. So denverton likely won't fare much better.

w0w

@dopey
Did you restart firewall after change applied?
Do you have the same result on your em card?

dopey

Oh duh!! I didn't switch back to the on igb NIC after making the change. I'll try that when I get a chance.

w0w

@w0w
But at least it looks you have some performance drop on em card also after some changes? Is it spectre/meltdown patch?

dopey

Yeah, the spectre/meltdown update coincided with a pretty big drop in performance with the em driver.