PPPoE traffic speed and single core use on i211AT NICs

zgtc

Hello,

New member here, I started using pfsense last year, I consider myself quite newbie. My hardware is an Odyssey x86 (Intel Celeron quad core J4105 and 2x Intel i211AT NIC). My ISP connection is 600Mbps. I get the full 600Mbps up/down and the hardware seems to be dealing fine (not much % of CPU, heat, etc). All "good"... Until I also tried to setup Suricata, and the moment I started using IPS, with very few basic rules, pfsense would collapse. I thought it was a problem of the CPU and even started thinking of building my own.

Then I came across the PPPoE with Multi-Queue NICs note in pfsense documentation. There are lots of posts about this, but after reading a lot on this subject, I don't think I found a clear solution. My conclusion is that pfsense should clearly state that igb based NICs should not be used (at least when using PPPoE). These igb NICs include the common i210, i211, i350... that are actually recommended everywhere. Actually, FreeBSD listed igb based NIC in the compatible hardware list until version 11 (up to pfsense 2.4), but dropped them in the v12 and v13 release (pfsense 2.5 and up). This change has not been reflected in the pfsense documentation, nor have I found it in any post, blog, forum, etc. Also FreeBSD/pfsense propose the use of net.isr.dispatch=deferred but this wont fix the problem, just improve the speed a bit. To be clear, the problem seems to be PPPoE WANs packets are only received on one NIC driver queue (queue0), so in practice, these NICs do not take full advantage of multi-core/thread when using PPPoE (it's not a driver problem). FreeBSD recommends the use of NIC capable of multiple receive queues (RSS) like Intel 82576.

So, as a newbie I am asking, what to do?

Is it a good assumption that a newer CPU (with better single core speed) would at least improve the overall performance and maybe even be able to cope with basic IPS load?
Another option would be to build my own mini ITX using other NICs that support to distribute PPPoE traffic per-queue. Do these NIC exist? I have read everywhere to avoid Realtek, and apparently the Intel ones using the em driver (mainly Intel PRO/1000 825xx models) should not be affected by the PPPoE single queue. Is that true? Curious enough, i210 i211 i350 and i354 where listed in the igb group in FreeBSD v11, and switched to em group from v12. Weird.
Forget about Suricata and stay with the basic use of pfsense (still improves over ISP router as I can use VLANs).
Final option would be to move to a country where ISP don't use PPPoE. Or is this universal?
Any other option?

Thank you so much and sorry for the length. I tried to document all links, did my homework the best I could.
Sergi

stephenw10

Just to be clear this is nothing to do with igb NICs or any other NICs. The issue with PPPoE loading only using a single core applies to all NICs/drivers. The only reason this is often seen in igb is that they are multiqueue so the throughput reduction when that cannot be used is high.

igb NICs are still supported, we ship devices with them ourselves.
From FreeBSD 12 onward the drivers were re-written to use iflib and the em and igb drivers were merged. You can still see all the same NIC listed there under the em driver.

A CPU with better single core speed will give you better PPPoE througput.
There are no NICs that allow multiple receive queues for PPPoE. It's not a NIC issue.
I would expect to be able to use Suricata still because on a 4 core CPU like that most of the existing loading will be on on core only. What exactly happens when you enable it?
Switching ISPs to one the doesn't use PPPoE would work. As would moving the PPPoE session to an external device if that's possible.

I would expect this to work with what you have if it's configured correctly.
Define "pfsense would collapse".

Steve

zgtc

Hi and thank you for your replies

@stephenw10 said in PPPoE traffic speed and single core use on i211AT NICs:

Just to be clear this is nothing to do with igb NICs or any other NICs. The issue with PPPoE loading only using a single core applies to all NICs/drivers. The only reason this is often seen in igb is that they are multiqueue so the throughput reduction when that cannot be used is high.
igb NICs are still supported, we ship devices with them ourselves.

OK I had the misunderstanding from this post in the FreeBSD issue tracker, but what you're saying makes sense as well.

From FreeBSD 12 onward the drivers were re-written to use iflib and the em and igb drivers were merged. You can still see all the same NIC listed there under the em driver.

Yes, I saw that these NICs were moved to em driver from FreeBSD v12, as stated in my previous post. Thanks for pointing out anyway.

A CPU with better single core speed will give you better PPPoE througput.

OK in case my tests with net.isr.dispatch=deferred and Suricata go wrong again, I will probably think about a DIY with an i5.

There are no NICs that allow multiple receive queues for PPPoE. It's not a NIC issue.

Noted

I would expect to be able to use Suricata still because on a 4 core CPU like that most of the existing loading will be on on core only. What exactly happens when you enable it?

This was like 1 yr ago so I don't remember the specifics, but everything collapsed, was not able to handle the traffic and in the practice, I had no router. Will try again, hopefully this weekend. Since I am not able to download same version 2.4 I will have to do the tests on the new 2.6 though (in a different SSD, just in case).

Switching ISPs to one the doesn't use PPPoE would work.

Sadly, not a real chance these days in this country.

As would moving the PPPoE session to an external device if that's possible.

Please tell me more about this. I am not sure how this would work. I understand using the ISP router in bridge mode would not convert PPPoE to ethernet traffic, right? I mean, I think I should use the ISP router and then add my pfsense at the output. This would come with the double NAT problem though. But if just by using the ISP router in bridge mode fixes this...

I would expect this to work with what you have if it's configured correctly.
Define "pfsense would collapse".

Will try this weekend again.

Thank you again for all the support
Sergi

stephenw10

It's been a while since I looked into it but I used to have a DSL modem that could be a PPP to dhcp bridge of sorts. I never used that mode but it involved having the PPP session on the 'modem'.
You could certainly do a DMZ style setup where the ISP modem/router handles the PPPoE and just forwards all traffic to pfSense. That does mean the ISP router is still routing though and any limitations it might have would still apply.

I could imagine that setting net.isr.dispatch=deferred might not play nicely with Suricata, especially if it's running in in-line mode.

Steve

zgtc

@stephenw10 said in PPPoE traffic speed and single core use on i211AT NICs:

Define "pfsense would collapse".
Steve

Hi again Steve. Finally had the time to play with this. I actually swapped the SATA disk for a NVMe (mainly to keep a copy of the 2.4.5 just in case, but I will keep the NVMe since it's faster and I don't have another system for it ATM). Installed v2.6.0 from scratch, using ZFS, restored backup, tested speed, all good 600Mbps.

Then I installed Suricata in Legacy (no Inline mode) and some basic rules:

Emerging Threats Open Rules
Snort Subscriber Rules (free user rules)
Snort GPLv2 Community Rules
Feodo Tracker Botnet C2 IP Rules
ABUSE.ch SSL Blacklist Rules

I have no clue what I am doing, but these seemed like sensible settings... and it works :)

I don't know why it collapsed last year, but it handles just the same 600Mbps speed (how is this possible?) and only seen a small increase in memory and CPU from around 3-5% to 15%, roughly.

So, all in all, a success. Still need to learn about how to properly setup the Suricata engine and maybe fine tune some rules.

Thank you again.

stephenw10

Ah, good result then!

Yeah you almost certainly don't need all those rules. Also those are rulesets and you probably don't have all the rules in them loaded because that would be a lot of rules!

Steve