Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0
-
Hi there,
I just upgraded from 2.4.4.3 to newest build, prior to the upgrade I upgraded snort and pfblocker.
After the upgrade snort would not start and I received a similar error, Ialso noticed that the snort rules failed to download at first, but i managed to download them manually.
Sep 19 03:20:10 php /tmp/snort_bge0_startcmd.php: The command '/usr/local/bin/snort -R _bge0 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_bge022424 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 22424 -c /usr/local/etc/snort/snort_22424_bge0/snort.conf -i bge0' returned exit code '1', the output was 'ERROR size 1464 != 1448'
my WAN adapter is named bge0 so nothing unusual and I do not have VLAN tagging enabled in pfsense as that is handled on the switch.
The only other thing i have changed apart from the upgrade was that I disabled PXE boot rom from the NIC at server boot.
This the first integrated adapter out of two on a HP dl385 G4 server dual Xeon CPU with 6gb Mem, it has run for years and never failed an upgrade.
Please let me know if you want any other info from the system.Kind Regards
Sebastian
-
@mikrosec said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Hi there,
I just upgraded from 2.4.4.3 to newest build, prior to the upgrade I upgraded snort and pfblocker.
After the upgrade snort would not start and I received a similar error, Ialso noticed that the snort rules failed to download at first, but i managed to download them manually.
Sep 19 03:20:10 php /tmp/snort_bge0_startcmd.php: The command '/usr/local/bin/snort -R _bge0 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_bge022424 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 22424 -c /usr/local/etc/snort/snort_22424_bge0/snort.conf -i bge0' returned exit code '1', the output was 'ERROR size 1464 != 1448'
my WAN adapter is named bge0 so nothing unusual and I do not have VLAN tagging enabled in pfsense as that is handled on the switch.
The only other thing i have changed apart from the upgrade was that I disabled PXE boot rom from the NIC at server boot.
This the first integrated adapter out of two on a HP dl385 G4 server dual Xeon CPU with 6gb Mem, it has run for years and never failed an upgrade.
Please let me know if you want any other info from the system.Kind Regards
Sebastian
That error is not related the other poster's error in this thread. His error message clearly stated the PID suffix was too long.
While it does not say so, your error seems like it might be an MTU or snaplen thing. Is this interface by chance a PPPoE interface?
LATER FOLLOW-UP:
Okay, did some Google research and found this error can be caused by having incorrect versions of some shared libraries. If you upgraded from pfSense-2.4.4_p3, that is clearly possible.
Go to the GLOBAL SETTINGS tab and verify that the "Keep Settings After Deinstall" setting is enabled (box checked). That is the default, but check to be sure. Then go to the SYSTEM > PACKAGE MANAGER tab and then Installed Packages and delete the Snort package. Let the package removal complete. Then go to the Available Packages tab and install it again. Report back on how that works.
-
It's a normal Ethernet Link nothing special, but you are absolutely right it is not related I just found this error:
98858 FATAL ERROR: Failed to initialize dynamic preprocessor: SF_MODBUS version 1.1.1 (-2)
I will investigate and open another thread if necessary, sorry for barging in.
KInd Regards
-
@mikrosec said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
It's a normal Ethernet Link nothing special, but you are absolutely right it is not related I just found this error:
98858 FATAL ERROR: Failed to initialize dynamic preprocessor: SF_MODBUS version 1.1.1 (-2)
I will investigate and open another thread if necessary, sorry for barging in.
KInd Regards
Follow my instructions in the "LATER FOLLOW-UP" edit I made to my previous post above. That should correct your problem.
-
@bmeeks Thank you for that!
I checked that "Click to retain Snort settings after package removal." was enabled and deinstalled and reinstalled smoothly without any errors. And it started up smoothly, like it has for 8 years now :-)
Saved my night
Cheers.
-
@mikrosec said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks Thank you for that!
I checked that "Click to retain Snort settings after package removal." was enabled and deinstalled and reinstalled smoothly without any errors. And it started up smoothly, like it has for 8 years now :-)
Saved my night
Cheers.
The
pkg
utilty that FreeBSD uses to install and update packages sometimes misses updating certain shared libraries, particularly when changing base OS versions such as when updating from pfSense-2.4.4 to 2.4.5 (that was a FreeBSD operating system update). In this case, some of the Snort preprocessor libs did not get deleted and the new versions installed. -
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
I found the change that is causing your problem. However, that code has been running over in pfSense-2.5 DEVEL for over a year, so it's weird that nobody running Snort 4.x on the DEVELOPMENT snapshots has encountered this. It's caused by the extra long name of your real interfaces with the VLAN ID tacked on the end.
A typical NIC real interface name is three or four characters such as em0, bce1, etc. An interface with that name length won't hit the suffix name limit defined by the Snort binary, even when a VLAN ID is tacked on (unless the VLAN ID is really long). Your NIC real interface shows up as vtnet2, so that is a bit longer than normal. That, plus your three digit VLAN ID, is what puts you over the limit when the underscore character is included.
The code change was made in the 4.x branch of Snort to address some other issues with the older technique used in the 3.x Snort branch of passing a randomly-generated GUID as part of the PID file name. I can possibly revert back to that method, but I need to carefully review all of the 4.x PHP code to be sure I find and fix all the places that reference the PID filename.
The only thing you can do in the interim is shorten that real interface name. The only method I think you have is to shorten the VLAN ID, but that of course likely requires other changes in your system that you may not wish to undertake.
Thanks for the quick check! :)
Can you estimate how long you will need for the fixing this?
-
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
Is there any way to roll back to the old package version?
-
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
I am unable to reproduce your problem. I changed a rule from DROP to ALERT and it showed up as "user-forced Alert" on the RULES tab.
You do understand that when making a change on the ALERTS tab that the icon for the past alert will not change. If the rule was DROP when the rule was triggered, that is the action written to the log file and thus will be the action displayed. However, it will be different on the RULES tab if you force an action change on the ALERTS tab and then go check the same rule on the RULES tab.
Lastly, the action change dialog pops up as what Bootstrap calls a Modal Dialog. It's possible that some security settings in your browser might be preventing that from displaying. Are you saying you never see the Modal Dialog box, or you see the box and make a change but it does not "stick"?
Is there any way to roll back to the old package version?
No, there is no way to roll back. The old package is removed from the repository.
-
@Beerman said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Can you estimate how long you will need for the fixing this?
It will be several days. Perhaps as long as a week. Once I make the change, the update has to be reviewed and approved by the pfSense team before it gets posted in the package repositories.
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
I am unable to reproduce your problem. I changed a rule from DROP to ALERT and it showed up as "user-forced Alert" on the RULES tab.
You do understand that when making a change on the ALERTS tab that the icon for the past alert will not change. If the rule was DROP when the rule was triggered, that is the action written to the log file and thus will be the action displayed. However, it will be different on the RULES tab if you force an action change on the ALERTS tab and then go check the same rule on the RULES tab.
Lastly, the action change dialog pops up as what Bootstrap calls a Modal Dialog. It's possible that some security settings in your browser might be preventing that from displaying. Are you saying you never see the Modal Dialog box, or you see the box and make a change but it does not "stick"?
There is nothing happens, when I click to any icon...
What for security settings do you mean?Is there any way to roll back to the old package version?
No, there is no way to roll back. The old package is removed from the repository.
Is there any plan to implement this. I had a lot of issues in the past years, and think it would help to make pfsense more professional. -
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
I am unable to reproduce your problem. I changed a rule from DROP to ALERT and it showed up as "user-forced Alert" on the RULES tab.
You do understand that when making a change on the ALERTS tab that the icon for the past alert will not change. If the rule was DROP when the rule was triggered, that is the action written to the log file and thus will be the action displayed. However, it will be different on the RULES tab if you force an action change on the ALERTS tab and then go check the same rule on the RULES tab.
Lastly, the action change dialog pops up as what Bootstrap calls a Modal Dialog. It's possible that some security settings in your browser might be preventing that from displaying. Are you saying you never see the Modal Dialog box, or you see the box and make a change but it does not "stick"?
There is nothing happens, when I click to any icon...
What for security settings do you mean?Is there any way to roll back to the old package version?
No, there is no way to roll back. The old package is removed from the repository.
Is there any plan to implement this. I had a lot of issues in the past years, and think it would help to make pfsense more professional.I assume that you do know you can't modify the action of a rule when you are using Legacy Mode Blocking. The option to modify a rule's action is only enabled when you use Inline IPS Mode blocking. When using Legacy Mode Blocking, you can't change the action of rules and thus the icons are not "clickable". The Inline IPS Mode option is new to this version of Snort on pfSense-2.4.5 RELEASE. Previously this option was only available in pfSense-2.5 DEVEL.
-
Thanks @bmeeks for all this information, I really appreciate it!
I was able to make some progress after coming across these couple interesting links last night:
https://lists.freebsd.org/pipermail/freebsd-net/2017-March/047384.html
https://lists.freebsd.org/pipermail/svn-src-all/2016-June/126504.htmlIt turns out that Chelsio offers direct netmap support via virtual interfaces which need to be enabled separately with a sysctl tunable. The regular interfaces don't seem to support netmap natively so when I enabled inline mode in Snort, netmap was actually being emulated, which explains the poor performance I was seeing. Unfortunately I missed this key log entry the first time around, which offered a clue:
Sep 19 12:47:08 kernel 028.141828 [ 318] generic_netmap_register Emulated adapter for cxl1 activated
In any case, I went ahead and enabled the virtual inferfaces by setting the sysctl tunable
hw.cxgbe.num_vis=2
and then confirmed with dmesg that I now had netmap support for the Chelsio card:vcxl1: netmap queues/slots: TX 8/1023, RX 8/1024 vcxl1: 8 txq, 8 rxq (NIC); 8 txq, 8 rxq (netmap)
I then reassigned interface to use the virtual interface instead (i.e. instead of using cxl1 for the interface I assigned vcxl1). Everything was working well at this point, traffic was passing fine, and performance was as expected.
However, as soon I switched over from legacy mode to inline mode in Snort and started using netmap, all traffic ceased on the vcxl1 interface. If I turn Snort off or go back to legacy mode, everything works fine and traffic passes. At this point I'm stumped - I still have offloading, checksumming, flow control disabled. It does appear like all traffic is being dropped for some reason, but I'm unsure why. @bmeeks are you aware of any additional logging facilities in Snort or otherwise I could look into to try to investigate this further? Or would it maybe make sense to give Suricatta and its inline mode a try instead (if for some reason there is a problem with the Snort netmap implementation)?
Thanks again for all your help, I really appreciate it.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
However, as soon I switched over from legacy mode to inline mode in Snort and started using netmap, all traffic ceased on the vcxl1 interface. If I turn Snort off or go back to legacy mode, everything works fine and traffic passes. At this point I'm stumped - I still have offloading, checksumming, flow control disabled. It does appear like all traffic is being dropped for some reason, but I'm unsure why. @bmeeks are you aware of any additional logging facilities in Snort or otherwise I could look into to try to investigate this further? Or would it maybe make sense to give Suricatta and its inline mode a try instead (if for some reason there is a problem with the Snort netmap implementation)?
Thanks again for all your help, I really appreciate it.
Unfortunately there is no additional logging within Snort relative to netmap operation. Netmap is a kernel device, and Snort is basically ignorant of what happens internally within the device. All the Snort binary does is call an API to open an interface in netmap device mode. It gets a "success" or "failure" message, and that's about it.
Having said that, there are continuing changes happening within the netmap code in FreeBSD. This is particularly true in the new iflib wrapper API found in FreeBSD-12 and up (which is what pfSense-2.5 DEVELOPMENT is based upon). And I do know that in the not too distant past a bug was fixed in the code where the netmap device interacts with virtual devices. Perhaps that bug (which I assume still exists in the legacy FreeBSD code base) is what is biting your Chelsio virtual configuration ???
Netmap has proven to be pretty problematic with both Snort and Suricata. And it is not just on pfSense, either. The "other" firewall distro based on FreeBSD has also had the same types of netmap problems with Inline IPS Mode. Things are getting better, but the code in FreeBSD is not yet 100% there. As a result, you will find some NICs that work fine while others won't work at all. If your NIC hardware won't work with Inline IPS Mode, then you will just have to switch over to Legacy Mode Blocking. The situation is marginally better on FreeBSD-12 with the new iflib wrapper, but even there it is not yet perfected.
-
Thanks @bmeeks for this additional info - it's very helpful!
I do have some good news! I was able to make some more progress and got everything working in native netmap mode on the Chelsio interfaces. It turns out the that the missing tunable was
hw.cxgbe.fl_pktshift
which needs to be set to 0:https://lists.freebsd.org/pipermail/freebsd-net/2016-January/044433.html
I had considered setting this tunable before but didn't change it because I thought the default value was already 0. Turns out I was reading the wrong version of the FreeBSD documentation and in 11.3 it is actually set to 2 by default (Doh! It's always something simple isn't it? :)). In any case, once I set the pktshift value to 0, traffic started passing fine.
So to summarize, to use native netmap support with Chelsio cards set the following tunables in your
loader.conf.local
:hw.cxgbe.num_vis=2 hw.cxgbe.fl_pktshift=0
This will create a virtual interface for each physical interface, e.g.
cxl0
will havevcxl0
and so on. These virtual interfaces support netmap natively. You can confirm this by running dmesg and looking for netmap in the output, e.g.:dmesg | grep netmap
You should see netmap support for the Chelsio virtual interfaces, including any other interfaces in your firewall that support netmap natively. For sample output, please see my previous post above. Finally you need to assign the virtual interface(s) to your network segment(s) and enable Snort Inline mode.
==================
Unfortunately, performance is not much better for me. Even when using netmap natively on the Chelsio card (as opposed to emulated mode) and enabling Snort Inline (netmap) mode, I was still only able to pass about 200-300Mbit/s between two different 10Gbit LAN interfaces (separate physical subnets, no VLAN's) using an iperf3 test. This is very similar to what I saw in emulated netmap mode. With Snort legacy (pcap) mode, I usually get around 4Gbit/s. WAN performance is even worse and limited to 50-80 Mbit/s down/up on a symmetric 1Gbit fiber connection. I do not think the issue lies with the Chelsio netmap support because if I enable Snort inline IPS mode solely on the WAN interface which is Intel
igb
, and not on any of the Chelsio interfaces, I see the exact same WAN down/up speed limitation.@bmeeks you had mentioned this above:
Lastly, there is a different netmap interface API version in FreeBSD-11 versus FreeBSD-12. Only FreeBSD-12 supports multiple host rings in netmap. FreeBSD-11 exposes only a single pair of Tx/Rx host rings.
Do you think I may be hitting this (single host ring) limitation? Is there any way to tune Snort? Also, do you think it might be worth trying Suricata instead if they handle this interfacing differently (better?) Or would I be better off trying pfSense 2.5.0 at this point?
Thanks again for your help and insight I really appreciate it.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks you had mentioned this above:
Lastly, there is a different netmap interface API version in FreeBSD-11 versus FreeBSD-12. Only FreeBSD-12 supports multiple host rings in netmap. FreeBSD-11 exposes only a single pair of Tx/Rx host rings.
Do you think I may be hitting this (single host ring) limitation? Is there any way to tune Snort? Also, do you think it might be worth trying Suricata instead if they handle this interfacing differently (better?) Or would I be better off trying pfSense 2.5.0 at this point?
Thanks again for your help and insight I really appreciate it.
Perhaps, but there is also the issue of Snort being single-threaded. I'm not a kernel threading guru, so I'm not sure whether multi-threaded operation would help. My gut says "yeah", but there are other places where bottlenecks can occur besides just with the netmap packet processing. I don't know if you have duplicate hardware, but if you did, you could test using pfSense-2.5 snapshots. They are based on FreeBSD-12 and have the multiple host ring support (but only if the native NIC also exposes multiple rings).
Also, with Legacy Mode Blocking using PCAP, you would need to enable all the Snort stats to see if any packets are in fact getting dropped by Snort. You would not notice this on your speed test throughput because Snort is working on copies of the packets while the originals have gone on to the kernel when using Legacy Blocking. So the PCAP engine might be dropping some of the copied packets at high line rates. I think there are some stats for that when Statistics Mode is enabled (on the PREPROCESSORS tab). But I might be confusing some Suricata features here, so I don't want to be quoted on that.
When I first deployed the Inline IPS option with netmap on pfSense-2.5, the Netgate team did some testing of their own. If I recall correctly, they were able to get close to Gigabit line rates, but I don't know how many rules they had enabled. I'm thinking probably not very many. The Snort binary is likely not optimized for inline operation. Remember that Snort's history has been IDS first and foremost. The inline mode was added later as I understand it. And it's likely that code is not streamlined. Snort natively, through its DAQ library, supports two different inline modes on FreeBSD. One uses
ipfw
while the other usesnetmap
. The change I made to DAQ on pfSense was to allow thenetmap
mode to support host stack interfaces. Natively, DAQ only supported physical interfaces, so setting up an inline IPS mode between a NIC and the OS itself was not possible. You could only specify two different physical NIC ports as thenetmap
endpoints. This was not optimum for pfSense, so I added the ability to open and use host stack rings in DAQ. -
Thanks @bmeeks - I decided to take a step back and just enable inline mode on the WAN interface (Intel igb) and ran iperf3 test between my network and a cloud based VM. Doing so I see the same 200-300Mbit/s up/down transfer limit I saw when doing local testing between the two Chelsio interfaces (usually I can get ~900/900 up/down to this VM when I have Snort legacy or pcap mode enabled). This makes me wonder now if the limitation simply comes down to Snort not being fully optimized for IPS mode, my firewall's CPU is not fast enough, or a combination of both. The CPU in my firewall is a Xeon D-1518, which is a quad core 2.2GHz chip with Hyperthreading. I do see CPU usage close to maxed when using Snort inline (netmap) mode on the igb interface, but I also see the same when using Snort legacy (pcap) mode. So I'm not fully convinced yet that it's just raw CPU cycles that are limiting performance.
As such, do you think it's worth giving Suricata a try? I did have a couple questions:
- Suricata does support multi-threading, correct?
- Is Suricata better suited to be an IPS than Snort?
- Is it easy enough to migrate enough between the two, i.e. would my suppression list(s) transfer over pretty easily?
Thanks again for all your help.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
As such, do you think it's worth giving Suricata a try? I did have a couple questions:
- Suricata does support multi-threading, correct?
- Is Suricata better suited to be an IPS than Snort?
- Is it easy enough to migrate enough between the two, i.e. would my suppression list(s) transfer over pretty easily?
Thanks again for all your help.
You can certainly test, but I'm not sure Suricata will fare much better. Even its multithreaded operation still has certain bottlenecks. This used to be the big arguing point between the Snort and Suricata guys when Suricata first came out. The Snort team poo-pooed multithreaded operation -- that is until they decided to introduce it in Snort3, where they now play it up ... , so who knows what to believe.
Suricata is not necessarily better than Snort in IPS mode. Also, Suricata currently does NOT have multiple host rings support on FreeBSD-12. They are still using the older NETMAP_API version 13 interface. Only NETMAP_API version 14 has the multiple host rings.
As for your Suppress Lists, they can be freely copied and pasted if you save them off on your client PC, but Suricata will not automatically "find them" and use them when you install it. Snort and Suricata are totally independent in terms of configuration parameters. And the Suppress List is stored within the
config.xml
file of the firewall in the configuration section associated with the particular package the list is used with. So when I say copy and paste, I literally mean open your list for editing in Snort, copy all of the contents out and paste into say a Notepad file in Windows. Then install Suricata, create a Suppress List, and paste in the text from the Notepad file. If you do that, don't forget to go to the INTERFACE SETTINGS tab and assign the Suppress List to the Suricata interface where it belongs. -
Thanks @bmeeks - I did end up running a few more speed tests with inline mode enabled on just the WAN (igb) interface and (oddly) saw quite a range of numbers: From as little as 100Mbit/s up/down (speedtest.net) to almost full line speed (Netflix's Fast.com) with several tests coming in between 200 to 300 Mbit/s (similar to what I saw with iperf3). I did only have a handful of ET rules enabled on the WAN interface, but even adding in the Connectivity rule set didn't have too big of an impact on speed (just a slight decrease). I also took some time to study the netmap documentation:
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4&manpath=FreeBSD+11.3-RELEASE
Thanks for providing some additional insight above about how you choose to change DAQ to use a netmap connection to the host stack. I'm trying to visualize how this all fits together so I can narrow down the relevant bottlechecks - would be it be something like this?
NIC --> Netmap --> Snort --> Netmap --> Host Stack
I'm still not convinced yet that a lack of processing power is the primary issue for me - the bottleneck could very well be the single host ring between netmap and the host stack, as the (Chelsio NIC) does support multiple netmap queues on it's virtual interfaces. I did enable preprocessing statistics as well on the WAN (igb) interface and looked at the output, but it's not giving me much insight where the majority of CPU time is being spent (to help narrow down potentially alleviate any bottlenecks). I thought I also read somewhere that there is a way to profile Snort in order to see that - do you know if that's possible to do on the pfSense / FreeBSD implementation?
In any case, I think further exploratory analysis is probably warranted before switching over to Suricata. I do have some additional hardware lying around here to build a test system for pfSense 2.5, including an additional Intel and Chelsio NIC, along with a CPU that will have better single thread performance (Intel i3-8100 3.6GHz quad core). I would first need to test with 2.4.5 to see if the CPU is indeed the primary bottleneck and then move up to 2.5 to see if now having the additional host rings will increase throughput. In the meantime, I will probably explore the netmap tunable parameters a bit more to see if any of them might make a noticeable difference on performance (especially parameters related to buffer and ring sizing).
Thanks again for all your help!
-
@tman222
I don't know the specific setup of youriperf3
testing, but when usingiperf
you should test THROUGH the firewall and not TO the firewall. In other words, do not put either theiperf
client or server on the firewall. They both should instead be on workstations or servers hanging off two different firewall interfaces so that you test through the firewall.When you run
ipferf
on the firewall itself, whether it's the client or server piece, there is some CPU time and potential thoughput chewed up by theiperf
executable.I mention this just because some other users in the past have tested with
iperf
installed on the firewall and have gotten misleading throughput numbers as a result.