pfSense CPU Interrupts bottleneck during SYN FLOOD DDOS

madhatterfounder · Oct 29, 2019, 9:41 PM

Hello,

I work for a hosting company and we run a Dell Poweredge with..

Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
16 CPUs: 2 package(s) x 8 core(s)
AES-NI CPU Crypto: Yes (active)

On average, we have a constant 800k states, but when we get a Syn Flood DDOS attack we have 8 million. (10 million max)

We are hitting a CPU Interrupts bottleneck, which puts our CPU at 99% and we start getting packet loss. Bandwidth is not a factor.

We can block, no log, kill states on the subnets responsible and that brings it back down.

We are implementing a DDOS solution, but in the meantime we want to throw more horsepower at the problem. I have seen debates about More Ghz vs more cores.

But for the sake of the CPU interrupts bottleneck, would it help to have more cores? 32 or 64 with lower Ghz, or 16 Cores with 4.8Ghz (with Turbo)

Thanks,

Dan

stephenw10 · Oct 29, 2019, 10:18 PM

Depends what limit you're hitting now. Is 99% shown on the dashboard? Try running top -aSH at the command line to see how that load is spread across the cores.

If it is showing 100% on the dash though it's probably using 100% on all CPU cores in which case anything will help.

Steve

madhatterfounder · Oct 30, 2019, 12:05 AM

That's correct, 99% on the dashboard. I don't think I kept any screenshots of top, but I will grab one the next time it happens.

Thanks for your input.

Thanks,

Dan

Derelict · Oct 30, 2019, 12:48 AM

What kind of NICs?

madhatterfounder · Oct 30, 2019, 3:21 PM

Chelsio 10Gb SFP+ T520.

Here's my thoughts. And they do get out there sometimes so correct me if I am wrong.

To me, it doesn't matter how WIDE (cores) we go, because the interrupts are going to keep building until the IPFW can process the packets that the NIC is receiving. All cores become consumed with the t5nex0.

I have heard that the IPFW operates best with higher clock speeds, like it may be somewhat single threaded.

Please confirm.

Thanks,

Dan

Derelict · Oct 30, 2019, 3:23 PM

Why are you talking about ipfw? Are you running Captive Portal?

madhatterfounder · Oct 30, 2019, 3:24 PM

Sorry, I was trying to edit my post, I meant the Firewall packet filer.

Thanks,

Dan

stephenw10 · Oct 30, 2019, 5:55 PM

It doesn't scale linearly certainly. More cores will help as long as you have queues provided by the driver for them to service. The Chelsio driver provides 16 Tx queues and 8 Rx queues so it's unlikely more Cores than that will help.
16 faster cores will probably be able to provide more throughput. That's speculating somewhat though.

Steve

madhatterfounder · Oct 30, 2019, 3:40 PM

Thanks you,

I assume you meant 16Rx and 8Tx queues?

It sounds like you are saying that each queue is limited to one core, and if so, could it help to have at least 24 cores? (16+8).

Thanks,

Dan

stephenw10 · Oct 30, 2019, 5:58 PM

Sorry, yeah, typo'd that. It's 8 Rx cores.

I'm not sure how that load would spread across 24 cores. There may be some work required to get the appropriate core affinity.

Steve