Suricata Inline IPS blocks LAN
-
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
ok i unistalled suricata again, and unchecked the keep settings
installed again, and did not enable any ruleset, no sid management
still blocked :(
new suricata log:30/11/2021 -- 21:23:54 - <Notice> -- This is Suricata version 6.0.3 RELEASE running in SYSTEM mode 30/11/2021 -- 21:23:54 - <Info> -- CPUs/cores online: 2 30/11/2021 -- 21:23:54 - <Info> -- HTTP memcap: 67108864 30/11/2021 -- 21:23:54 - <Info> -- Netmap: Setting IPS mode 30/11/2021 -- 21:23:54 - <Info> -- fast output device (regular) initialized: alerts.log 30/11/2021 -- 21:23:54 - <Info> -- http-log output device (regular) initialized: http.log 30/11/2021 -- 21:23:54 - <Info> -- 1 rule files processed. 310 rules successfully loaded, 0 rules failed 30/11/2021 -- 21:23:54 - <Info> -- Threshold config parsed: 0 rule(s) found 30/11/2021 -- 21:23:54 - <Info> -- 310 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 135 inspect application layer, 107 are decoder event only 30/11/2021 -- 21:23:54 - <Info> -- Going to use 1 thread(s) for device em1 30/11/2021 -- 21:23:54 - <Info> -- devname [fd: 6] netmap:em1/R em1 opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 7] netmap:em1^/T em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- Going to use 1 thread(s) for device em1^ 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 8] netmap:em1^/R em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 9] netmap:em1/T em1 opened 30/11/2021 -- 21:23:55 - <Notice> -- all 2 packet processing threads, 2 management threads initialized, engine started.I see no block in that log output. What are you saying?
If you mean the interface locked up and is not passing traffic, then I need to see the output of
procstat -t <pid>like I asked for earlier. I need to see if a Suricata thread is locked or not.
-
@bmeeks said in Suricata Inline IPS blocks LAN:
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
ok i unistalled suricata again, and unchecked the keep settings
installed again, and did not enable any ruleset, no sid management
still blocked :(
new suricata log:30/11/2021 -- 21:23:54 - <Notice> -- This is Suricata version 6.0.3 RELEASE running in SYSTEM mode 30/11/2021 -- 21:23:54 - <Info> -- CPUs/cores online: 2 30/11/2021 -- 21:23:54 - <Info> -- HTTP memcap: 67108864 30/11/2021 -- 21:23:54 - <Info> -- Netmap: Setting IPS mode 30/11/2021 -- 21:23:54 - <Info> -- fast output device (regular) initialized: alerts.log 30/11/2021 -- 21:23:54 - <Info> -- http-log output device (regular) initialized: http.log 30/11/2021 -- 21:23:54 - <Info> -- 1 rule files processed. 310 rules successfully loaded, 0 rules failed 30/11/2021 -- 21:23:54 - <Info> -- Threshold config parsed: 0 rule(s) found 30/11/2021 -- 21:23:54 - <Info> -- 310 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 135 inspect application layer, 107 are decoder event only 30/11/2021 -- 21:23:54 - <Info> -- Going to use 1 thread(s) for device em1 30/11/2021 -- 21:23:54 - <Info> -- devname [fd: 6] netmap:em1/R em1 opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 7] netmap:em1^/T em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- Going to use 1 thread(s) for device em1^ 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 8] netmap:em1^/R em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 9] netmap:em1/T em1 opened 30/11/2021 -- 21:23:55 - <Notice> -- all 2 packet processing threads, 2 management threads initialized, engine started.I see no block in that log output. What are you saying?
If you mean the interface locked up and is not passing traffic, then I need to see the output of
procstat -t <pid>like I asked for earlier. I need to see if a Suricata thread is locked or not.
yes it locks up
procstat looks the same as earlier:23482 100532 suricata - -1 120 sleep nanslp 23482 100718 suricata W#01-em1 -1 120 sleep select 23482 100725 suricata W#01-em1^ -1 120 sleep select 23482 100726 suricata FM#01 -1 120 sleep nanslp 23482 100727 suricata FR#01 -1 121 sleep nanslp -
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks said in Suricata Inline IPS blocks LAN:
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
ok i unistalled suricata again, and unchecked the keep settings
installed again, and did not enable any ruleset, no sid management
still blocked :(
new suricata log:30/11/2021 -- 21:23:54 - <Notice> -- This is Suricata version 6.0.3 RELEASE running in SYSTEM mode 30/11/2021 -- 21:23:54 - <Info> -- CPUs/cores online: 2 30/11/2021 -- 21:23:54 - <Info> -- HTTP memcap: 67108864 30/11/2021 -- 21:23:54 - <Info> -- Netmap: Setting IPS mode 30/11/2021 -- 21:23:54 - <Info> -- fast output device (regular) initialized: alerts.log 30/11/2021 -- 21:23:54 - <Info> -- http-log output device (regular) initialized: http.log 30/11/2021 -- 21:23:54 - <Info> -- 1 rule files processed. 310 rules successfully loaded, 0 rules failed 30/11/2021 -- 21:23:54 - <Info> -- Threshold config parsed: 0 rule(s) found 30/11/2021 -- 21:23:54 - <Info> -- 310 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 135 inspect application layer, 107 are decoder event only 30/11/2021 -- 21:23:54 - <Info> -- Going to use 1 thread(s) for device em1 30/11/2021 -- 21:23:54 - <Info> -- devname [fd: 6] netmap:em1/R em1 opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 7] netmap:em1^/T em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- Going to use 1 thread(s) for device em1^ 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 8] netmap:em1^/R em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 9] netmap:em1/T em1 opened 30/11/2021 -- 21:23:55 - <Notice> -- all 2 packet processing threads, 2 management threads initialized, engine started.I see no block in that log output. What are you saying?
If you mean the interface locked up and is not passing traffic, then I need to see the output of
procstat -t <pid>like I asked for earlier. I need to see if a Suricata thread is locked or not.
yes it locks up
procstat looks the same as earlier:23482 100532 suricata - -1 120 sleep nanslp 23482 100718 suricata W#01-em1 -1 120 sleep select 23482 100725 suricata W#01-em1^ -1 120 sleep select 23482 100726 suricata FM#01 -1 120 sleep nanslp 23482 100727 suricata FR#01 -1 121 sleep nanslpHmm... this does not look like the same thing as the flow manager bug. In that one, the thread would show locked with a state of "umtxn" (kernel lock). I see no thread with that status, so your issue is something else entirely. Not sure what it could be.
If Snort works without issue for you, then perhaps swap over to using it.
-
@bmeeks said in Suricata Inline IPS blocks LAN:
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks said in Suricata Inline IPS blocks LAN:
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
ok i unistalled suricata again, and unchecked the keep settings
installed again, and did not enable any ruleset, no sid management
still blocked :(
new suricata log:30/11/2021 -- 21:23:54 - <Notice> -- This is Suricata version 6.0.3 RELEASE running in SYSTEM mode 30/11/2021 -- 21:23:54 - <Info> -- CPUs/cores online: 2 30/11/2021 -- 21:23:54 - <Info> -- HTTP memcap: 67108864 30/11/2021 -- 21:23:54 - <Info> -- Netmap: Setting IPS mode 30/11/2021 -- 21:23:54 - <Info> -- fast output device (regular) initialized: alerts.log 30/11/2021 -- 21:23:54 - <Info> -- http-log output device (regular) initialized: http.log 30/11/2021 -- 21:23:54 - <Info> -- 1 rule files processed. 310 rules successfully loaded, 0 rules failed 30/11/2021 -- 21:23:54 - <Info> -- Threshold config parsed: 0 rule(s) found 30/11/2021 -- 21:23:54 - <Info> -- 310 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 135 inspect application layer, 107 are decoder event only 30/11/2021 -- 21:23:54 - <Info> -- Going to use 1 thread(s) for device em1 30/11/2021 -- 21:23:54 - <Info> -- devname [fd: 6] netmap:em1/R em1 opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 7] netmap:em1^/T em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- Going to use 1 thread(s) for device em1^ 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 8] netmap:em1^/R em1^ opened 30/11/2021 -- 21:23:55 - <Info> -- devname [fd: 9] netmap:em1/T em1 opened 30/11/2021 -- 21:23:55 - <Notice> -- all 2 packet processing threads, 2 management threads initialized, engine started.I see no block in that log output. What are you saying?
If you mean the interface locked up and is not passing traffic, then I need to see the output of
procstat -t <pid>like I asked for earlier. I need to see if a Suricata thread is locked or not.
yes it locks up
procstat looks the same as earlier:23482 100532 suricata - -1 120 sleep nanslp 23482 100718 suricata W#01-em1 -1 120 sleep select 23482 100725 suricata W#01-em1^ -1 120 sleep select 23482 100726 suricata FM#01 -1 120 sleep nanslp 23482 100727 suricata FR#01 -1 121 sleep nanslpHmm... this does not look like the same thing as the flow manager bug. In that one, the thread would show locked with a state of "umtxn" (kernel lock). I see no thread with that status, so your issue is something else entirely. Not sure what it could be.
If Snort works without issue for you, then perhaps swap over to using it.
snort works... for now
suricata worked fine for a couple of days, then started doing this, locking in moments :(
couldn't be some sort of tuning or workaround with the nic the problem? ihave all the hardware offloading disabled i think
will try again with snort and monitor things
thank you very much for your time!
if you have any more ideas, i'll try them all
thanks -
Sorry, had to be away for a while.
Weird that it worked fine for a few days and then started misbehaving. Nothing has changed in the package obviously, as no update has been posted.
I assume you are using Snort with Inline IPS Mode as well? If not, then that might explain Snort working while Suricata does not (in Inline IPS Mode). The default for both packages, when blocking is enabled, is Legacy Mode.
The Inline IPS Mode uses the netmap device within FreeBSD. Did you by chance make any other configuration change in pfSense such as adding a limiter or enabling traffic shaping? Those two features are not compatible with the netmap kernel device used when Inline IPS Mode is utilized, and they could cause some weird issues if enabled alongside Inline IPS Mode.
-
@bmeeks
yes, snort is in ips mode, seems ok
i have a couple of limiters defined bur they are not used right now, they are for a vlan that is not working right now, and i even deleted the whole vlan for a test, no change -
@bmeeks
just did a fresh install, configured only minimal things, installed suricata, same thing, it locks the interface.
so no vlans, no limiters, no suricata rules, just minimum. still blocks -
@cobrax2 Curious why your Intel NIC using A Mellanox driver instead of Intel pro driver...have you gotten that sorted?
-
@nollipfsense
yes, was just me using wrong command -
@bmeeks
hmm
i have a few days now working with snort, all good. but it seems that vlans do not work lol. the packets get somehow blocked, the tagging of them i mean. i see in the sniffer the dhcp reques, the answer from the dhcp server gives the wrong ip range (it gives the lan range) and nothing gets through. this is the capture on lan interface. on vlan interface seems nothing gets captured. if i stop snort process, vlans work. what gives?
thanks -
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
hmm
i have a few days now working with snort, all good. but it seems that vlans do not work lol. the packets get somehow blocked, the tagging of them i mean. i see in the sniffer the dhcp reques, the answer from the dhcp server gives the wrong ip range (it gives the lan range) and nothing gets through. this is the capture on lan interface. on vlan interface seems nothing gets captured. if i stop snort process, vlans work. what gives?
thanksThe netmap kernel device used for Inline IPS Mode does not honor VLAN tags. That's just another limitation of that driver. It does not work well with limiters, traffic shapers, or VLANs.
You can try running Snort on the parent VLAN interface. It puts the interface in promiscuous mode anyway and thus will see all traffic crossing the physical device.
-
@bmeeks
snort is already on em1 lan interface :(
vlan is on em1.1
is there some workaround? on the old pc it worked, i had 2 older em intel pci nics -
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
snort is already on em1 lan interface :(
vlan is on em1.1
is there some workaround? on the old pc it worked, i had 2 older em intel pci nicsWhat mode were you using on the older PC? Was it Inline IPS Mode, and were you using Snort there, too?
I do not know your whole story here, so crafting replies is a bit of a shot in the dark.
Inline IPS Mode in both Snort and Suricata requires the use of the netmap kernel device inside FreeBSD. This is not a pfSense thing. It's a FreeBSD thing. The netmap device does not play well with several other networking features due to the way it is plumbed internally into the FreeBSD network stack. As I mentioned before, VLANs, shapers and limiters are some of the things that either don't work at all, or work only "sort of" with netmap.
The specific flavor of network card chipset in use also has a big influence on how the netmap device behaves. That's because the netmap device speaks directly to the NIC hardware driver so it can intercept packets. This link to NIC hardware drivers is in the process of being abstracted a bit with the move to the iflib wrapper API library in FreeBSD. As more NIC cards get mapped over to using that wrapper, netmap compatibility should improve. And hopefully even features that current are dodgy with netmap will later get fixed.
Right now, it sounds to me like you need to either put those old NIC cards into your new PC, or else switch to Legacy Blocking Mode if you continue using the newer NICs. Or you could try to purchase some new NICs. Are you 100% positive the ones you currently have are genuine Intel? There are a number of counterfits out there, so be wary if you see any for sale "at a great price" ...
. -
@bmeeks
i can't put the old cards back, i don't have pci slots in this pc
any idea when the new drivers will be available?
thanks again! -
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
i can't put the old cards back, i don't have pci slots in this pc
any idea when the new drivers will be available?
thanks again!I have no idea about timetables for NIC driver updates. That's up to the manufacturer guys that write them and the FreeBSD maintainers who add them to the distro. They are also outside the control of the Netgate/pfSense team as well.
Based on what you've said thus far, you may have a NIC that is not genuinely Intel. Or at least it is not highly compatible with the generic em driver in FreeBSD. Other folks with the same generic driver are not having issues as severe as yours. Or if they are, nobody is reporting it here.
One thing you need to be sure you've done is to disable all the offloading options under SYSTEM > ADVANCED > NETWORKING.
You have tried two packages, and both are giving you issues with Inline IPS Mode operation. So at this point you will need to either switch to Legacy Mode blocking in the IDS/IPS packages, or cease using the packages if you don't want to use Legacy Mode blocking.
-
@bmeeks ok, thank you, will try to find another nic
-
@bmeeks said in Suricata Inline IPS blocks LAN:
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
i can't put the old cards back, i don't have pci slots in this pc
any idea when the new drivers will be available?
thanks again!I have no idea about timetables for NIC driver updates. That's up to the manufacturer guys that write them and the FreeBSD maintainers who add them to the distro. They are also outside the control of the Netgate/pfSense team as well.
Based on what you've said thus far, you may have a NIC that is not genuinely Intel. Or at least it is not highly compatible with the generic em driver in FreeBSD. Other folks with the same generic driver are not having issues as severe as yours. Or if they are, nobody is reporting it here.
One thing you need to be sure you've done is to disable all the offloading options under SYSTEM > ADVANCED > NETWORKING.
You have tried two packages, and both are giving you issues with Inline IPS Mode operation. So at this point you will need to either switch to Legacy Mode blocking in the IDS/IPS packages, or cease using the packages if you don't want to use Legacy Mode blocking.
well i activated the onboard lan, it is an Intel
I219-V.
looks like it is still an em driver, it behaves exactly as the pcie one that i have, snort still blocks the vlan :( if i disable snort, all works. so the driver is the problem, or snort? not the nic :(
thanksedit: or some freak setting somewhere in my config? i have already disabled in networking all the hw things. its strange that only i report this...
-
@cobrax2 said in Suricata Inline IPS blocks LAN:
edit: or some freak setting somewhere in my config? i have already disabled in networking all the hw things. its strange that only i report this...
I strongly suspect it is something specific to just your setup. Could be hardware, or it might be a configuration issue.
If I may ask, is English your primary language, or is it a second language? I'm asking because of the way you are using the term "block". That is a bit confusing to me because in the context of the two IPS packages, "block" has a very specific meaning related to blocking certain IP addresses only. If that is the case, where only certain clients are having issues while other clients and traffic are not, then a configuration problem is likely the cause. If, on the other hand, ALL traffic through the IPS interface just stops and absolutely nothing gets through, then that would indicate something hardware related in terms of the driver software. You might want to do some Google research on the netmap device in FreeBSD to better understand what I am talking about when discussing how certain hardware NIC drivers interact with the kernel's netmap device (when the netmap device is active).
There are a lot of Snort and Suricata installations out there using the em driver without issue in Inline IPS Mode. In fact, that is the driver I use frequently in my VMware virtual machines when testing updates to both Snort and Suricata.
-
@bmeeks
no english is not my primary language
sorry about my bad explanations. the issues are different from suricata to snort. suricata locks/ drops all traffic pn physical interface at some point, no matter vlan or lan.
snort does something to the packets coming tagged, the dhcp server on pfsense "sees" the udp req coming on physical lan and returns a lan address to it, but the device probably does not receive the reply as it goes back untagged? the normal physical lan, untagged portion works fine. also this does not block/ lock/drop further packets on neither lan or vlan. but packet capture on vlan shows nothing...
thanks -
@cobrax2 said in Suricata Inline IPS blocks LAN:
@bmeeks
no english is not my primary language
sorry about my bad explanations. the issues are different from suricata to snort. suricata locks/ drops all traffic pn physical interface at some point, no matter vlan or lan.
snort does something to the packets coming tagged, the dhcp server on pfsense "sees" the udp req coming on physical lan and returns a lan address to it, but the device probably does not receive the reply as it goes back untagged? the normal physical lan, untagged portion works fine. also this does not block/ lock/drop further packets on neither lan or vlan. but packet capture on vlan shows nothing...
thanksThe Suricata issue is one where I really have no solution to offer. That certainly seems like the old flow manager threading bug in Suricata 6.0.x, but there specifically is a patch for that in the pfSense version of Suricata. And if that bug was in fact not fixed, everyone would be reporting an issue regardless of NIC driver type. That bug affected all drivers the same.
For Snort, perhaps hardware VLAN tagging is still enabled on the NIC driver. Some have that option, and it must be disabled using
sysctlvariables. But some versions of the Intel NIC drivers (I don't recall which at the moment), do not honor thesysctlcommands to actually disable hardware VLAN tagging. As I said before, the netmap devices and VLANs don't play well together, and most especially with hardware VLAN tagging enabled at the NIC driver level.
