PFSense crashes with TCP Segmentation Offload or Large Receive Offload Enabled

willk · Feb 1, 2019, 5:32 AM

Hi All,

We have three PFSense machines. Two were purchased directly from NetGate and work fine. The last is a SuperMicro 1018D-FRN8T. It has an X10SDV-7TP8F motherboard with both Intel I210 and Intel I350 NICs.

We found that under a UDP attack there would be a huge number of CPU interrupts, even though CPU usage remained below 30%, but the PF would stop being able to process packets quickly enough. This is exactly what LRO was designed to prevent. However, when we turn LRO on, the system crashes within a few minutes every time, this was with about 400k states which should be no problems with the system (32 Core Xeon).

We have 128GB of RAM in the machine, and the MBUF usage was not an issue.

We had also changed the defaults in the system tunables to these settings with no change in behaviour:
kern.ipc.nmbclusters 2000000
kern.ipc.nmbjumbop 1000000
net.inet.tcp.sendbuf_max 4194304
net.inet.tcp.recvbuf_max 4194304

Note that we are not testing under an attack, just switching LRO on with our regular load (about 200-250k states and under 500mb/s bandwidth) causes the system to crash.

Any advice on getting the most out of this system with PF would be greatly appreciated as we are at a loss at how to improve it's capabilities and mitigate UDP flood attacks with the PF.

Thanks,
Will

stephenw10 · Feb 1, 2019, 2:55 PM

Was it blocked UDP packets?

Did you see some Cores pegged at 100% during that time?

What is the crash you see with LRO enabled? We usually recommend leaving that disabled though for just this reason, it can be unstable.

Steve