Enabling blocking offenders results in net down and lost access to the GUI

Draghmar · Oct 3, 2020, 3:07 PM

Hi
I was playing with Snort that is set up on LAN. I choose some rules from the list (free VRT, Community, ET and OpenAppID but I didn't use any rules from that last one) with flowbits enabled and some additional ETs. The rest of the config is probably at default. I was observing alerts for some time and now decided to turn on inline blocking mode. At first I thought everything went well but few seconds later I lost net access and also access to pfSense GUI. SSH also was unreachable. The only thing I could do was restarting machine. It behave the same way after booting - few seconds of everything ok and when Snort was started everything went down. I had to manually stopped Snort to have everything working again.

Weird thing is that I was previously testing Suricata with the same exact result, with the difference that I choose the Legacy Mode instead there.

I'm pretty sure it has to be some config issue because people are using this packages but I have no idea where to look. I don't see anything in logs but then again webGUI show only last 50 entries so I may missed something. Snort was even alerting after each boot (I was bootig few times to check).
I decided to go with inline mode because as far I understand it wouldn't block anything unless I choose to which was great for me...So that probably wasn't it.

Have ever encountered something like this?

Draghmar · Oct 3, 2020, 5:42 PM

Maybe I should also mention that I have pfBlockerNG installed and NAT Reflection currently enabled (NAT + proxy) - waiting for a little bit of time to make config work without it.

bmeeks · Oct 5, 2020, 6:51 AM

When you enable Inline Blocking Mode, a warning box pops up telling you that not all NICs behave with Inline IPS Mode and issues up to and including a complete system crash can occur with some hardware when Inline IPS Mode is enabled.

Sounds like, from your description, your hardware is one of those. The solution is to use Legacy Blocking Mode only. You said that mode works with Suricata, so it should work with Snort. The hardware NICs in your firewall will not support netmap operation, and thus won't support Inline IPS Mode either if you get a complete freeze-up when you enable that mode.

Draghmar · Oct 5, 2020, 6:55 AM

@bmeeks Thanks for suggestion but from what I understand my NICs should be supported. I have em (I217LM) for WAN and igp (I211AT) for LAN, both of which are mentioned here as a "good to go".
Maybe I wasn't clear enough with Suricata when I meant the results were the same. Using Legacy Mode in Suricata also made complete lockdown in the same exact way: few seconds of being fine, when even logs got alerts/blocks and then everything stopped working. Those few seconds gave me enough time to stop Snort/Suricata service thus preventing crash. Actually if Suricata wouldn't crash my system I wouldn't even try Snort.

bmeeks · Oct 5, 2020, 4:15 PM

@Draghmar said in Enabling blocking offenders results in net down and lost access to the GUI:

@bmeeks Thanks for suggestion but from what I understand my NICs should be supported. I have em (I217LM) for WAN and igp (I211AT) for LAN, both of which are mentioned here as a "good to go".
Maybe I wasn't clear enough with Suricata when I meant the results were the same. Using Legacy Mode in Suricata also made complete lockdown in the same exact way: few seconds of being fine, when even logs got alerts/blocks and then everything stopped working. Those few seconds gave me enough time to stop Snort/Suricata service thus preventing crash. Actually if Suricata wouldn't crash my system I wouldn't even try Snort.

That is not how I understood your initial post. I took it that Legacy Mode worked but Inline IPS did not.

If you are getting a complete lockup with Legacy Mode, then you have something fundamentally wrong in your configuration. I would start by removing everything (including the abomination, NAT reflection) and starting over. Add just one package, configure it and get it working. Let it run for several hours to be 100% sure it is working. Then add another step, etc. With an IDS/IPS, pfBlocker, NAT reflection and proxy, you have a lot of places where a misconfiguration can cause your problem.

Draghmar · Oct 5, 2020, 1:46 PM

@bmeeks Thanks, I get your general idea for troubleshooting.
Would disabling pfBlockerNG service be enough for this test? Or are you suggesting removing it completely? I have only few packages installed: pfBlockerNG, iperf (disabled), arpwatch and nmap. (Snort is out for now) I can disable or uninstall all of them (I think pfBlockerNG has its config retained?) and try to install only Snort and checkout if it crashes with the config that was left after uninstalling.

I can disable NAT Reflection for testing, if you're suggesting I should. I have it on because I needed to have domains resolving on my local webserver and didn't have yet time to "fight" with DNS Resolver config as was suggested. (Yup, you're not the first to point out about the evilness of NAT Reflection, not to mention documentation hints :P)

bmeeks · Oct 5, 2020, 1:53 PM

In most cases, disabling a package is enough and full uninstall is not usually necessary. The idea, as you have surmised, is to start clean with a known working system and then add things one at a time to see what actually causes the "break".

Snort, Suricata and pfBlocker will all maintain their config information and will restore it when you reinstall the package. That can be a good thing, or a bad thing if that config is corrupt. For those packages, there is an option to remove the configuration when uninstalling the package. You can use that if you wanted to start with a completely clean slate. I would only do that if my first attempt at say installing and running just Snort or Suricata failed. The option is located on the GLOBAL SETTINGS tab in Snort and Suricata down towards the bottom of the page. It is a checkbox that is "checked" by default to preserve the configuration. Uncheck the box to erase the configuration when removing the package.

Draghmar · Oct 5, 2020, 2:29 PM

@bmeeks Ok, thanks for all suggestions. I'll wait until weekend to test everything out and will report how it went. :)

buggz · Oct 6, 2020, 10:29 AM

I had to change the following for INLINE mode to work here, there was a popup somewhere that told me to change these settings:
System/Advanced/Networking

Network Interfaces
CHECK - Hardware Checksum Offloading/Disable hardware checksum offload
CHECK - Hardware TCP Segmentation Offloading/Disable hardware TCP segmentation offload
CHECK - Hardware Large Receive Offloading/Disable hardware large receive offload

for me,
motherboard IF - em0 - WAN
cheap Chinese 4 port Intel adapter
igb0 - LAN
igb1 - not used
igb2 - not used
igb3 - not used

Draghmar · Oct 6, 2020, 10:29 AM

@buggz Thanks for suggestion. I have second and third disabled. First is not but from what I read what it can cause it false positives because of timestamp differences. Anyway I had all three of them disabled when I was checking Suricata and yet I had crash, so I don't think it's related. But I have to remember to disable them when testing to make sure there is nothing that could impact the result. :)

buggz · Oct 6, 2020, 10:34 AM

Just a note:
I got warnings displayed on the console if those settings were not checked, and it would not work.

Maybe I will try them one by one to if it is only one setting.

I'll have to do this sometime much later though, as I am enjoying the use of the system.

Draghmar · Oct 6, 2020, 10:36 AM

@buggz As far I remember the message I had when turning on Snort all three of them should be disabled. Message didn't say which one exactly was enabled. Only to check and disable.

bmeeks · Oct 6, 2020, 3:01 PM

All three offloading settings should always be disabled for all modes of the IDS/IPS packages. That's because the NIC will create packets that are too large when these offloading options are not disabled. With those offloading options enabled, the NIC itself is reassembling packets instead of letting the pfSense kernel do so. The IDS/IPS packages expect standard size packets (typically 1500 bytes or so). Having the offloading options enabled will result in packets that are too large for the kernel-configured netmap buffer size. That will lead to netmap errors.

Users need to understand that Inline Mode for both packages (Snort and Suricata) uses a special kernel-provided networking device called the netmap adapter. That adapter, especially on FreeBSD-11 and earlier, requires the NIC hardware driver be cognizant of and operate with the kernel's netmap device. If not, problems will occur. FreeBSD-12 is a little better as the netmap interoperability was moved into the new iflib wrapper for network drivers. But that wrapper code is new and still getting the occasional bug fix on the FreeBSD side.

What version of pfSense are you running?

Draghmar · Oct 6, 2020, 3:03 PM

@bmeeks "Should" still doesn't mean "must", which is especially important when you'll consider that there's no warning before actually starting blocking mode. The fact I knew about that was only because in Suritaca I had a lot timestamp alerts, which Internet told me, was related to the hardware checksum being enabled. ;) But like I said - I did turn all of them off with Suricata. I didn't know that it's also needed for Snort until I turned on blocking mode, because there was no indication for that before.

I will turn them off for the testing phase! :)

pfSense version: 2.4.5-RELEASE-p1/FreeBSD 11.3-STABLE

BTW Thanks for the explanation!