Netmap (Suricata) cause crash

giyahban

Hello, Hope all is well
Im running pfsense latest version virtualized on esxi (DL360G8, 128GB, 2x2680v2)
The pfsense vm has 2 nics one with for lan another one for wan all other lan interfaces are vlans of the parent interface.
everything works great but in recent days I tried to run suricata and now Im getting random crashes
Im using suricata inline mode with igx interfaces (VMXNET3), has done all of the recommended options such as disable offloading and checksum with shellcmd

Configuring pfSense/netmap for Suricata Inline IPS mode on em/igb interfaces

but it still Im getting netmap_transmit error and whole pfsense crashesh nothing routed, in the topic it says that :

However, if you see this filling your log, then it's an indication that Suricata (or whatever application is using netmap) cannot process the packets fast enough. In other words: you need to get a faster CPU or disable some of your rules. You can buy yourself a tiny bit of burst by modifying buffers, but all you are doing is buying a few seconds of breathing room if you are saturating your link.

pfsense hase 8GB of ram and 24core of cpu, isnt it enough ? or there is some other problem ?

note that I set disabled checksum with shellcmd just for my lan interface not my wan. Is it necessary too ?

bmeeks

Need some more info in order to make an educated guess at the problem.

What version of pfSense and the Suricata package are you running?
What exactly is the nature of the "crash"? Is pfSense itself crashing, or is just the Suricata instance on the LAN interface crashing?
Post any Suricata related error messages you see from the following two log files:
a. pfSense system log
b. suricata.log file for the LAN interface (available under LOGS VIEW tab).

giyahban

@bmeeks
Thanks for your reply,

pfSense+ 23.05.1 and suricata 6.0.13
pfSense dosent route and nat anything, but still respond to icmp. cannot access gui or ssh from lan. cannot access from wan ovpn server the only way to access it is through physical access to console and you will see it constantly log netmap_transmit
pfSense system log

Aug  4 07:04:43 pfSense suricata[61550]: [Drop] [1:2260002:1] SURICATA Applayer Detect protocol only one direction [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 80.94.
95.184:3810 -> 192.168.53.15:587
Aug  4 07:04:55 pfSense suricata[61550]: [Drop] [1:2500026:6610] ET COMPROMISED Known Compromised or Hostile Host Traffic group 14 [Classification: Misc Attack] [Priority: 2] {TCP} 77.90.185.
18:51424 -> 192.168.53.15:587
Aug  4 12:10:38 pfSense syslogd: kernel boot file is /boot/kernel/kernel
Aug  4 12:10:38 pfSense kernel: 839 [4336] netmap_transmit           vmx1 full hwcur 235 hwtail 236 qlen 510
Aug  4 12:10:38 pfSense kernel: 715.006887 [4336] netmap_transmit           vmx1 full hwcur 235 hwtail 236 qlen 510

unfortunately suricata.log is wiped after restarting

bmeeks

Yes, the suricata.log file is cleaned with each restart from the GUI.

Are you running on Netgate hardware or a white box with a pfSense Plus license?

Do you have any VLANs configured on that interface? If so, Suricata running with Inline IPS Mode hates VLANs and does not work well with them. This is due to how the netmap device interacts with virtual interfaces in the kernel's network stack.

That error message does mean that the netmap buffers are not being emptied out after being populated with packet data. Not sure why that would be, but typically this is a result of a problem with the underlying NIC driver.

I am running pfSense CE on two test virtual machines under VMware Workstation using the VMXNET3 drivers. I have the interfaces running with Inline IPS Mode without issue. However, those interfaces do NOT have any VLANs defined and nothing else special such as a Bridge or LAGG. Just plain vanilla individual Ethernet interfaces. I have not tested under ESXi, but I would suspect the VMXNET3 virtual NICs are very similar if not identical to those in VMware Workstation.

If you have whitebox hardware, it's possible that the actual physical NIC and ESXi might have an issue with each other.

giyahban

@bmeeks
No Im running it on a hp dl380g8 with home lab license
yeah actually I have 5 different vlans on the same interface. I didnt know its not recommended to have vlan with inline mode.

Thanks for sharing your experience I will move it to physical interface and see what will happen

bmeeks

@giyahban said in Netmap (Suricata) cause crash:

didnt know its not recommended to have vlan with inline mode.

Inline IPS Mode has some limitations. The biggest is that VLANs and other virtual interfaces are not currently well supported. Things like a Bridge or LAGG setup will not work well. VLANs are especially problematic. There is some work happening within FreeBSD's netmap code to make things better, but none of those experimental updates are present in the pfSense kernel yet.

If you want to use Inline IPS Mode, you should only deploy it on plain-vanilla Ethernet interfaces (meaning no VLANs defined and not a member of a LAGG or Bridge). You may get by with running Suricata on the physical parent interface only and NOT on each defined VLAN interface.