PSA for anyone with a hyperthreading CPU, or one with P-cores + E-cores…

[luckman212]

I was debugging a very high interrupt (intr process in top/ps/btop etc) when pushing high amounts of data across my 10G ixl interfaces. After much research (I'll spare you the boring parts) I discovered that the FreeBSD iflib driver was spreading the load across multiple cores of my "10-core" Intel i7-1255U CPU, treating each one equally. In fact, only core#0 and #2 were true "P-cores" (performance) -- the rest were just HT cores or low perf E-cores.

I figured out that by adding these sysctls to /boot/loader.conf.local, I could force those queues to use the P-cores and my CPU usage when pushing 10G traffic dropped from >25% to ~11%.

dev.ixl.0.iflib.override_qs_enable=1
dev.ixl.0.iflib.core_offset=0
dev.ixl.0.iflib.override_nrxqs=2
dev.ixl.0.iflib.override_ntxqs=2
dev.ixl.1.iflib.override_qs_enable=1
dev.ixl.1.iflib.core_offset=0
dev.ixl.1.iflib.override_nrxqs=2
dev.ixl.1.iflib.override_ntxqs=2

Your overrides may be different if you have a different CPU, so double check.

After tuning this, you want to kick off some high bandwidth transfers and watch your iterrupts, check that they are hitting the P-cores with something like systat -vmstat 1 and top -HPST -p $(pgrep -Sx intr):

I couldn't find anything about this in the docs (probably because it doesn't really affect current Netgate hardware) so passing this along. Hope it helps someone.

[KOM]

I specifically avoid Intel CPUS now (and have for years since the sidechannel attacks) due to that and this sort of nonsense. AMD all the way. They're not perfect but I was just sick & tired of Intel's garbage. IIRC VMware had an issue with mixed cores but Proxmox handled them just fine. I just avoid mixed cores (and Intel) whenever I can.

[AndyRH]

@KOM AMD will be doing 3 core types. That is the way all will be. Apple is doing it with ARM CPUs.

[KOM]

@AndyRH Dammit. I don't want p cores, e cores, and lpe cores all being juggled by a scheduler that may or may not handle them efficiently and transparently.

[stephenw10]

Just use one very fast core. Solved!