400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4

[Konan 0]

PPPoE is not in use on the interface in question.

Been working through what is an unexpected bottleneck on this hardware. On a single TCP download, i'm showing 100% interrupt time on a single core and unable to get much over 400mbps

After working out that igb interfaces are still handled by hw.em - I looked at the hw.em.max_interrupt_rate and found it was set to 8000. However, watching it in vmstat I was seeing igb1:rxq1 (igb1 being WAN) going over 8000.

I've adjusted it to 32000.

I've tinkered with the various HW offloads being on or off. Disabled some of the interface power savings. Run over the network tuning guide. No observed changes in behavior.

I've set net.isr.dispatch=deferred, which appears to have picked performance up from around 375 to 410mbps, which makes me think I'm looking in the right area.

This hardware replaced an old Atom D525 / e1000 based system, which could also shift about 400mbps, but that was definitely a CPU limit.

There are a large number of limiters in place in the programing. My testing is being performed with a simple CURL to /dev/null from the shell. I've checked the limiter stats and I'm 99% sure the traffic isn't hitting them. My gut feeling is this isn't going to be pfsense, but BSD/Hardware/Driver related, but I think it's worth mentioning.

I know the WAN I'm testing with is good for at least 800mbps. These are actually a pair using CARP, both have this issue and both can pull 400mbps from the WAN at the same time.

[netblues]

@Konan-0 Still you are benchmarking from inside pfsense.

Try running the same curl from the lan side on another box and see what happens.

pfsense is known not to perform well from console, wasn't designed or optimized for that.

As you probably know, n2930 was released in 2014 and maximum expected throughput is less than 1G. plain uov4 doing routing or nat

[Gertjan]

@netblues said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

pfsense is known not to perform well from console, wasn't designed or optimized for that.

Not just pfSense, every router actually.

When testing the real ^^ throughput, this implies : traffic comes into an interface, and leaves another.
Now, take teh eving of, go here Core Dumped and here Computerphile and watch all the video's.
When done, you've paved the way : you are close to becoming a kernel expert. Look outside : over 100 k people know less about kernel as you do.

Now you are ready to understand what @netblues said.
Normal routing : traffic comes into an interface, and it send on another.
This means : (we go close to bottom rock level here) ;
The NIC, the chip, has a packet. It interrupts the processor.
The processor NIC interrupt handler set up the DMA to empty out the NIC, and checks where to place it. The destination of the packet exist in the routing table, so the DMA is told to do a NIC to NIC transfer.
Done.
The processor itself can go on doing it's other stuff (kernel, multi task, sleeping, playing a game, whatever).

When you initiate a speed test on pfSense, an end-user process does the actual transfer, using the kernel of course.
This time, when packets come in (or go out) on every packet, the data has to be transfer into 'user' (not kernel) space. This takes 'a lot' of time. Context switching is costly.
Then the user speed - test process is informed and does things with the date. Like making stats, charts whatever. The rapid (you said 8000 time per second ?) interrupts now generate extra local processor activity, as the " user speed - test process" now has to intercut with the data. if it's a one core/process task, this one core will get hammered.

All this is normally not a big deal as a router is not a web server, file server, mail server, or doing speed tests.
A router's goal is routing, and doing some firewall activity while doing so.
( ones in a while is seems that Netgate forgot about that one ^^)

Solution :
Do less with your router and/or get bigger iron.

[Konan 0]

@netblues said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

@Konan-0 Still you are benchmarking from inside pfsense.

Try running the same curl from the lan side on another box and see what happens.

pfsense is known not to perform well from console, wasn't designed or optimized for that.

This is the testing I've been doing because of an observed limit to the throughput from the LAN - I didn't jump straight to testing from the firewall to find the speed, I'm doing it that way to try to isolate it from being a configuration issue.

I've used a download to /dev/null as a quick and dirty test when troubleshooting and it's never presented a problem with getting near wire speed before. On much lower end hardware.

Anyway - here's what we see pulling the file down from a client on the LAN. 100% interrupt on one core and around 40% on the LAN side. Actually, it was useful to go back and look because if we take the 100% as gigabit, then the 40% matches the 400mbps. May be coincidence but there may be something odd going on with the interface WAN side.

@netblues said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

As you probably know, n2930 was released in 2014 and maximum expected throughput is less than >1G. plain uov4 doing routing or nat

I expect it to be able to route 1gbps. If that's unrealistic, I'd at least expect it to be faster than a 2010 atom that's approx half the single threaded speed and down 2 cores.

@Gertjan said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

Now, take teh eving of, go here Core Dumped and here Computerphile and watch all the video's.
When done, you've paved the way : you are close to becoming a kernel expert. Look outside : over 100 k people know less about kernel as you do.

Now you are ready to understand what @netblues said.

Just read that back and think about how it comes across.

@Gertjan said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

Solution :
Do less with your router and/or get bigger iron.

For the sake of wanting some more helpful insights, whilst 'just change the hardware' might be what's done, but I'd like to dial down where the actual issue is.

@netblues post has got me thinking (and looking at other systems) and, for example, this system, also with i211 NICs, appears to spread across both cores (it's SMT, ignore the other 2). Same test, a CURL to /dev/null - the system I have an issue with will only ever use 1 core of it's 4.

[Gertjan]

@Konan-0

I would exclude the VM from the equation.
I know, that means you have to 'just change the hardware'. But it would take any old (desktop) PC, 'less then 50€$' device, and your NICs, and you'll be good.

VMs are nice to test 'if it works'.
Afterwards, you should assign pfSense a dedicated device if 'speed' and arguably security are important for you.

[Konan 0]

@Gertjan What VM?

[Gertjan]

@Konan-0 said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

@Gertjan What VM?

Yeah, right, I thought I saw a 'VM'.... Sorry. Let's call it Friday, late in the afternoon (here), time to get a break ^^

[stephenw10]

I would expect to see 1Gbps through that box. But definitely test from some client on the LAN. There's a good chance the test itself on pfSense is using more of the available CPU than forwarding packets and that's the limit you're seeing.

I would also check you have speedstep/speedshift enabled. Some boards with embedded CPUs like that default to the lowest CPU speed if it's not enabled.

[Konan 0]

@stephenw10 said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

I would expect to see 1Gbps through that box. But definitely test from some client on the LAN. There's a good chance the test itself on pfSense is using more of the available CPU than forwarding packets and that's the limit you're seeing.

I would also check you have speedstep/speedshift enabled. Some boards with embedded CPUs like that default to the lowest CPU speed if it's not enabled.

Already tested from LAN (That's how the issue cropped up. I didn't start by testing from the box itself, it's just what I tend to do for simplicity).

said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

Anyway - here's what we see pulling the file down from a client on the LAN. 100% interrupt on one core and around 40% on the LAN side. Actually, it was useful to go back and look because if we take the 100% as gigabit, then the 40% matches the 400mbps. May be coincidence but there may be something odd going on with the interface WAN side.

I don't see any speedstep stuff in the UI. Would it be in with PowerD if enabled? I suspect it's maybe off in the BIOS. The dashboard widget shows 1.83GHz - which is the default top speed without burst.

[stephenw10]

Hmm, probably OK then. You can check directly using sysctl like:

[26.07-BETA][admin@5100.stevew.lan]/root: sysctl dev.cpu | grep freq
dev.cpu.3.freq_levels: 2200/0 2100/0 2000/0 1900/0 1800/0 1700/0 1600/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.cpu.3.freq: 1100
dev.cpu.2.freq_levels: 2200/0 2100/0 2000/0 1900/0 1800/0 1700/0 1600/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.cpu.2.freq: 1100
dev.cpu.1.freq_levels: 2200/0 2100/0 2000/0 1900/0 1800/0 1700/0 1600/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.cpu.1.freq: 1100
dev.cpu.0.freq_levels: 2200/0 2100/0 2000/0 1900/0 1800/0 1700/0 1600/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.cpu.0.freq: 1100

How many queues are shown for each NIC at boot? Something like speedtest.net will open multiple connections which should spread that load across the cores/queues.

[netblues]

@stephenw10 said in 400mbps bottleneck 100% interrupt on single core - Intel Celeron N2930 - Intel i211 x 4:

sysctl dev.cpu | grep freq

Strange enough systctl dev.cpu has no freq entry

[stephenw10]

You'll only see that is powerd is enabled and has set those of speedstep.

[Konan 0]

@stephenw10

I am seeing frequency options but the frequency isn't moving, so I'm fairly happy it's not dropping to the lower rates

dev.cpu.3.freq_levels: 1827/2000 1826/2000 1660/1825 1494/1650 1328/1475 1162/1300 996/1125 830/950 664/775 498/600
dev.cpu.3.freq: 1826
dev.cpu.2.freq_levels: 1827/2000 1826/2000 1660/1825 1494/1650 1328/1475 1162/1300 996/1125 830/950 664/775 498/600
dev.cpu.2.freq: 1826
dev.cpu.1.freq_levels: 1827/2000 1826/2000 1660/1825 1494/1650 1328/1475 1162/1300 996/1125 830/950 664/775 498/600
dev.cpu.1.freq: 1826
dev.cpu.0.freq_levels: 1827/2000 1826/2000 1660/1825 1494/1650 1328/1475 1162/1300 996/1125 830/950 664/775 498/600
dev.cpu.0.freq: 1826

Looking at

sysctl -a | grep 'queues'

I get

igb0: Using 2 RX queues 2 TX queues
<6>igb0: netmap queues/slots: TX 2/1024, RX 2/1024
igb1: Using 2 RX queues 2 TX queues
<6>igb1: netmap queues/slots: TX 2/1024, RX 2/1024
igb2: Using 2 RX queues 2 TX queues
<6>igb2: netmap queues/slots: TX 2/1024, RX 2/1024
igb3: Using 2 RX queues 2 TX queues
<6>igb3: netmap queues/slots: TX 2/1024, RX 2/1024
igb0: Using 2 RX queues 2 TX queues
<6>igb0: netmap queues/slots: TX 2/1024, RX 2/1024
igb1: Using 2 RX queues 2 TX queues
<6>igb1: netmap queues/slots: TX 2/1024, RX 2/1024
igb2: Using 2 RX queues 2 TX queues
<6>igb2: netmap queues/slots: TX 2/1024, RX 2/1024
igb3: Using 2 RX queues 2 TX queues
<6>igb3: netmap queues/slots: TX 2/1024, RX 2/1024
igb0: Using 2 RX queues 2 TX queues
<6>igb0: netmap queues/slots: TX 2/1024, RX 2/1024
igb1: Using 2 RX queues 2 TX queues
<6>igb1: netmap queues/slots: TX 2/1024, RX 2/1024
igb2: Using 2 RX queues 2 TX queues
<6>igb2: netmap queues/slots: TX 2/1024, RX 2/1024
igb3: Using 2 RX queues 2 TX queues
<6>igb3: netmap queues/slots: TX 2/1024, RX 2/1024
igb0: Using 2 RX queues 2 TX queues
<6>igb0: netmap queues/slots: TX 2/1024, RX 2/1024
igb1: Using 2 RX queues 2 TX queues
<6>igb1: netmap queues/slots: TX 2/1024, RX 2/1024
igb2: Using 2 RX queues 2 TX queues
<6>igb2: netmap queues/slots: TX 2/1024, RX 2/1024
igb3: Using 2 RX queues 2 TX queues
<6>igb3: netmap queues/slots: TX 2/1024, RX 2/1024

I'll go back and verify - but the bottleneck was originally spotted on a busy LAN, with multiple downloads/clients/speedtests running side by side, so I'm going to be it's 100% on one core.

[stephenw10]

Hmm at least 2 queues on each NIC then. I'd expect to see load spread across at least two cores. Unless that load is actually something like traffic shaping that can appear as interrupt on the NIC.

[Konan 0]

@stephenw10

I'll take a look. One of the reasons I tend to test from the console is an effort to avoid traffic shaping (which IIRC, you'd need a floating rule to catch outgoing from the firewall itself?).

The BW management is all done using limiters, which I didn't think used the network driver (dummynet in software?).

[stephenw10]

It depends which shaping you use. Limiters require a firewall rule to put traffic into the queue and that would usually never happen from the firewall itself. But AltQ based shaping uses outbound queues on each NIC. The default queue will catch traffic from the firewall itself.

[Konan 0]

@stephenw10 Thanks - that's as I understood it. I've also had the limiters on monitoring (in the UI) and they appear to show zero traffic when I'm testing.

No AltQ in use.

But I'm also feeling like I'm looking for something odd. Might flatten one of these units and just test it 'out of the box', as it were.

[stephenw10]

Yeah that's always a good test. Just to make sure something isn't setting something obscure.

[Konan 0]

Bit of progress, although I obviously need to look in more detail I don't think I need to look at the driver/hardware for the NIC now.

Out of furstration, just to narrow it down, I ran it with

pfctl -d

Boom - 1Gbps

So I think I need to concentrate on pf itself. I think now I know it's pf that i'm less fussed about it being bound to one core - although the LAN with multiple TCP connection apparently doing that remains a mystery to keep in mind.

I do have some floating rules that (i think) would catch traffic generated by the firewall itself - disabled these just to test but it didn't make any difference.

[stephenw10]

What sort of CPU usage did you see with pf disabled? If it was, maybe, less than 50% on every core then I'd expect to still be able to pass it with pf.