Snort 4.1.4_3 -- Snort Inline mode crashes WAN interfaces

Dule

Yesterday I upgraded my pfSense to 2.5.2 from 2.4.5(p1). One of the most interesting features for me was to test Snort Inline mode. After the upgrade I configured Snort Inline mode following steps bellow:

Snort Interfaces > WAN > IDS/IPS mode Inline - after that I got popup that I should enable Hardware checksum offload, Hardware TCP Segmentation Offloading and Hardware Large Receive Offloading.
I did it in Advanced > Networking and enable these three settings
Reboot

After reboot everything looked perfect. So I started to configure new rules through SID Management. Suddenly on my WAN interface GW become unreachable because packet los on that interface. During my investigation pfSense suddenly crushed. What I saw was these logs in crash report

710.855879 [4034] netmap_transmit       ixl0 full hwcur 1008 hwtail 1009 qlen 1022

In the end of the log I have this:

Fatal trap 9: general protection fault while in kernel mode
...
panic: general protection fault
cpuid = 3
time = 1633021711
KDB: enter: panic
panic.txt0600003014125367417  7143 ustarrootwheelgeneral protection faultversion.txt0600006714125367417  7550 ustarrootwheelFreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense

So I started to investigate further and found interesting config changes which I did:

Change WAN interface Speed/Duplex setting to autoselect
Added hw.igb.num_queues=1 in my /boot/loader.conf file

These changes just delayed crashes for one up to two hours, but they were present until I changed my IDS/IPS mode to legacy.

Right now I'm in legacy mode and everything works fine, so far. I'll test it better during the weekend.

From hardware perspective I've Dell R340 with 32GB RAM, 2 x intel X710 with 2x10Gbps SFP network cards and Intel(R) Xeon(R) E-2186G CPU @ 3.80GHz

I'm using Snort with Talos certified rules and OpenAppID.

Is there anyone facing similar issue? Am I missing something in configuration which makes my pfSense to acts like this. I'm looking forward to understand better and solve this problem and I can not find better place than this community!

Thanks in advanced!

bmeeks

The ixl driver works with a number of slightly different Intel chipsets from the same family. That's not uncommon for a FreeBSD driver. Netmap operation (which is required for Inline IPS Mode) requires cooperation from the NIC hardware driver. Prior to FreeBSD 12.x, netmap support had to be included by the hardware driver itself. With the release of FreeBSD 12, a new API was introduced for NIC drivers to use called iflib. Netmap support is now baked into iflib. But I don't think every single NIC driver has been rewritten to use iflib. Some still may not. And, finally, there have been a few bugs identified in the iflib wrapper library since it was released.

So all of the above is a long story to say it is very possible that your particular Intel NIC flavor is not well supported when attempting to use netmap. And without netmap support, Inline IPS can't work. I have a Netgate SG-5100 appliance. That appliance has Intel igb and ix NICs. I use Snort on that box with Inline IPS Mode enabled without issue. I seem to recall the ixl NIC driver as having at least one open bug posted on the FreeBSD bug reporting site.

You probably will have to stay with Legacy Mode Blocking so long as you have that particular NIC (unless an updated FreeBSD driver is released).

bmeeks

Here is the FreeBSD bug report for the ixl driver that I mentioned in my previous post: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230465. Until this bug is fixed upstream, I would not expect the ixl NIC driver to work well (if at all) using netmap operation with Inline IPS Mode.

Dule

@bmeeks Thank you for your responses. I turned off Inline mode and looks like firewall works good. I faced one more problem with IPSec for mobile client but I managed to fix it.

What do you think should I leave offloads disabled or should I revert that configuration and enable offloads once again?

I'm not sure that I need it anymore since I'll keep using legacy mode? I do not know how these offloads work in depth so probably thats the reason I'm confused

bmeeks

@dule said in Snort 4.1.4_3 -- Snort Inline mode crashes WAN interfaces:

@bmeeks Thank you for your responses. I turned off Inline mode and looks like firewall works good. I faced one more problem with IPSec for mobile client but I managed to fix it.

What do you think should I leave offloads disabled or should I revert that configuration and enable offloads once again?

I'm not sure that I need it anymore since I'll keep using legacy mode? I do not know how these offloads work in depth so probably thats the reason I'm confused

When using either of the two IDS/IPS packages, whether with Inline IPS Mode or with Legacy Blocking Mode, you should leave all hardware offloads disabled. With today's highspeed CPUs, there is very little impact from letting the software stack handle those chores.

The problem with hardware offloading being enabled is that the NIC then sends "weird" and somewhat non-standard packets through. And those weird packets can trip up and fool the IDS/IPS. Some forms of hardware offloading can also result in packets completely bypassing the IDS/IPS engine altogether, thus those packets are not even inspected by the engine.