Gigabit PPPoE and Intel Drivers

VAMike

Just an FYI, I have the same supermicro a1sri-2758f board with centurylink gigabit fiber and PPPoE.
I only get about 650-700mbit/s down with igb. I saw this post and picked up an intel 82574L (Intel EXPI9301CT) adapter and a mini-itx case capable of fitting an external card.

Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

dopey

@VAMike:

Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

Oh I know. But this a sunk cost. I've had the rangeley for a while now before I got gigabit and it really wasn't clear that PPPoE would be an issue until after the fact. There's no red blinking "WARNING" anywhere :)
But still, I'm impressed at the difference between em and igb. It looks like by default, em doesn't use queues at all, which makes me wonder why it's so much faster.

VAMike

@dopey:

@VAMike:

Honestly, an avoton is the wrong CPU if you're trying to do gigabit pppoe–you'd be better off with a higher clocked i3 or pentium. Your cheap ISP router probably did pppoe in hardware, so it's an apples-to-oranges comparison.

Oh I know. But this a sunk cost. I've had the rangeley for a while now before I got gigabit and it really wasn't clear that PPPoE would be an issue until after the fact. There's no red blinking "WARNING" anywhere :)
But still, I'm impressed at the difference between em and igb. It looks like by default, em doesn't use queues at all, which makes me wonder why it's so much faster.

rss queues don't work with pppoe because the nic can't find the unique IPs inside of the pppoe packets. so they all get dumped in the first queue anyway. it used to be that the problem was compounded because since the packets came through a mechanism which supposedly hashed them, they had a flowid associated and wouldn't get rehashed and redistributed once they were decapsulated (whereas stuff coming in on a card without rss gets a new flow ids assigned). I think that's still the case, which is why igb (or any multiqueue card) is worse than a single queue card.

dopey

Aah I thought it might be the overhead of the hashing in an attempt to queue it. That makes sense. Thanks.

w0w

There are some updates in FreeBSD bug report — https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856
Can someone test this possible solution suggested?
In terminal do

sysctl net.isr.dispatch=deferred

Try some gigabit tests, like dslreports or whatever. Check for your speeds and report it here, please.

mwave

@w0w That did it for me.

stephenw10

What did it do? Got you up to Gigabit line rate over PPPoE?

What speed were you seeing before/

Steve

dopey

I see very little difference with the net.isr.dispatch change. Ever since the spectre/meltdown bios update I'm barely cracking 650 with my c2758.

Anyone know if denverton is more capable for pppoe or do I really need to go into core series CPU?

I'm really looking for low tdp (preferably fanless) and ipmi and quad nic. I've found its nearly impossible to guarantee finding a non counterfeit Intel nic aftermarket without paying more than the CPU/motherboard for it :)

dopey

Looking at the benchmarks it doesn't look like denverton is any faster than avaton. More power efficient but that's it. So denverton likely won't fare much better.

w0w

@dopey
Did you restart firewall after change applied?
Do you have the same result on your em card?

dopey

Oh duh!! I didn't switch back to the on igb NIC after making the change. I'll try that when I get a chance.

w0w

@w0w
But at least it looks you have some performance drop on em card also after some changes? Is it spectre/meltdown patch?

dopey

Yeah, the spectre/meltdown update coincided with a pretty big drop in performance with the em driver.

dopey

Did a few more tests.
With em driver
net.isr.dispatch=deferred
700-800mbps

net.isr.dispatch=direct
675-715
most of the tests seem around 700 give or take a few

With igb
net.isr.dispatch=direct
500-600mbps

net.isr.dispatch=deferred
650-700

So net.isr.dispatch in both cases made a difference, but still shy of the 920 or so I should be pulling.

stephenw10

You can disable the Kernel PTI workaround for Meltdown in System > Advanced > Misc. You almost certainly don't need it anyway unless you are running virtual.

The IBRS workaround for Spectre may not be active anyway but you can disable that too with the loader tunable:
hw.ibrs_disable=1

https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities

Steve

VAMike

@stephenw10 said in Gigabit PPPoE and Intel Drivers:

You can disable the Kernel PTI workaround for Meltdown in System > Advanced > Misc. You almost certainly don't need it anyway unless you are running virtual.

That's not correct. You need to mitigate meltdown unless you are 100% confident that there is no need for privilege separation on a system. (E.g., if you have no reason to run a web service as something other than root, or run pre-auth ssh code as an unprivileged user, etc.) If you use privilege separation as a mitigation for other vulnerabilities (e.g., bug in web script, bug in ssh, etc.) then you need meltdown mitigation in order for the privilege separation to actually be meaningful. Other speculative execution bugs like L1TF-VMM (CVE-2018-3646) are specific to virtual machines.

dopey

@vamike that would only really apply if there's any ability to execute malicious code within the privilege separated processes right? If the router is locked down so only trusted individuals can to access it and there are no available vulnerablities (big IF I know) there's should be no way someone can take advantage of the vulnerablities.

I know there was some grumblings of a remote spectre like exposure but I don't know if that applies to routers.

VAMike

@dopey said in Gigabit PPPoE and Intel Drivers:

@vamike that would only really apply if there's any ability to execute malicious code within the privilege separated processes right? If the router is locked down so only trusted individuals can to access it and there are no available vulnerablities (big IF I know) there's should be no way someone can take advantage of the vulnerablities.

Sure. Like any other mitigation, it's a risk based decision. OTOH, if you can be sure that you can lock things down and never have a vulnerability, why are you running a firewall at all?

stephenw10

Mmm, interesting. Some stuff I had not considered there.

Anyway you can test it and see if it improves performance by any useful amount. If not leave it enabled.

Steve

VAMike

I'd expect the spectre mitigations to be more costly than meltdown, and arguably less relevant.