Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0
-
Look for some changes in the Snort package in the near future. I've posted updates for the pfSense team to review and merge.
The updates include the latest Snort-2.9.16.1 binary from upstream. However, the biggest change will be to users of Snort on pfSense-2.4.5 RELEASE. The new update will finally bring Inline IPS Mode using netmap to Snort on pfSense-2.4.5 systems! This is the same type of IPS operation currently available with Snort when using pfSense-2.5.0 DEVEL, and it's the same IPS mode available in the Suricata package. The new Inline IPS Mode will be especially useful to security admins using the OpenAppID rules where you would like to drop selected traffic to a host, but not block all traffic from that host. Inline IPS Mode lets you do that by selectively modifying the Action of rules you select from ALERT to DROP. This is most efficiently done using the SID MGMT tab options.
For users on pfSense-2.5.0 DEVEL, the new Snort package includes support for multiple host rings when the physical NIC in the netmap connection pair exposes multiple rings. Not all NIC drivers currently do this, but a few do. The multiple host rings feature is limited to FreeBSD-12.x and higher, so unfortunately the multiple host rings capability is just not available on pfSense-2.4.5 because it uses FreeBSD-11.x.
-
Hi,
thx, for the new version.
One Interface is failing to start. I get this error message in the system logs:
Sep 18 20:23:55 php: /tmp/snort_vtnet2.400_startcmd.php: The command '/usr/local/bin/snort -R _vtnet2.400 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_vtnet2.40049222 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 49222 -c /usr/local/etc/snort/snort_49222_vtnet2.400/snort.conf -i vtnet2.400' returned exit code '1', the output was ''
Sep 18 20:23:55 snort[63989]: FATAL ERROR: Invalid pidfile suffix: _vtnet2.400. Suffix must less than 11 characters and not have ".." or "/" in the name.What can I do?
pfSense: 2.4.5-RELEASE-p1
Snort pkg: 4.1.2Thx! :)
-
@Beerman said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Hi,
thx, for the new version.
One Interface is failing to start. I get this error message in the system logs:
Sep 18 20:23:55 php: /tmp/snort_vtnet2.400_startcmd.php: The command '/usr/local/bin/snort -R _vtnet2.400 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_vtnet2.40049222 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 49222 -c /usr/local/etc/snort/snort_49222_vtnet2.400/snort.conf -i vtnet2.400' returned exit code '1', the output was ''
Sep 18 20:23:55 snort[63989]: FATAL ERROR: Invalid pidfile suffix: _vtnet2.400. Suffix must less than 11 characters and not have ".." or "/" in the name.What can I do?
pfSense: 2.4.5-RELEASE-p1
Snort pkg: 4.1.2Thx! :)
First question is was this a working install on the previous Snort version, or is this a new installation, or have you added this new VLAN interface recently?
That particular error is being thrown by the Snort binary itself, I believe. It means literally what it says -- when your VLAN ID of 400 is tacked on to the package generated string for the interface PID filename, it results in a string longer than the 11-character limit.
-
Thank you for the quick answer.
It was an update. I did not change anything in the configuration, before and after the update.
In the previous version I did not have this error, there I had no problems of this kind.
-
@Beerman said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Thank you for the quick answer.
It was an update. I did not change anything in the configuration, before and after the update.
In the previous version I did not have this error, there I had no problems of this kind.
Okay, let me carefully compare the PHP code between this version and previous one. I did copy over the sections from the pfSense-2.5 DEVEL branch that supported Inline IPS Mode operation. Perhaps something happened there. If so, I can submit a fix in the near future.
-
Thx, for your support! :)
-
Hi @bmeeks -
Just went through the upgrade here and everything went fine. I wish, however, I could say the same for using inline mode instead of legacy mode:
- My system has 4 port Chelsio T540 10Gbit SFP+ network expansion card (for the LAN interfaces) and an Intel i210 based 1Gbit WAN interface. I was able to enable inline mode on all these interfaces (including those with VLAN's) without issues.
- Unfortunately with inline mode enabled performance really nose-dived. I used to be able to push 4Gbit/s+ between two different (LAN) network segments (both with Snort's Balanced policy enabled), using iperf3 in legacy mode. However, with inline mode enabled instead (and still using the Balanced policy) that throughput number reduced sharply to around 200Mbit/s.
- I also saw a large number of netmap transmit errors during the upload portion of WAN speed test (I have a symmetric 1Gbit fiber connection), as described here: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces Perhaps my system needs to have a faster CPU - it is Xeon D-1518 based which has worked quite well up now.
Ultimately, I decided to go back to legacy mode for now while I investigate these netmap performance issues further. Do you have any ideas why performance might be so bad? I do have all the offloading and flow control options disabled, but it didn't really seem to help.
I also have one quick question for you: While Snort is in legacy mode, does it matter what the IPS Policy Mode is set to under interface categories (i.e. either Alert or Policy)? Am I correct in thinking that it actually does not matter in legacy mode as long as Block Offenders is checked?
Thanks in advance for your help, I really appreciate it.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Hi @bmeeks -
Just went through the upgrade here and everything went fine. I wish, however, I could say the same for using inline mode instead of legacy mode:
- My system has 4 port Chelsio T540 10Gbit SFP+ network expansion card (for the LAN interfaces) and an Intel i210 based 1Gbit WAN interface. I was able to enable inline mode on all these interfaces (including those with VLAN's) without issues.
- Unfortunately with inline mode enabled performance really nose-dived. I used to be able to push 4Gbit/s+ between two different (LAN) network segments (both with Snort's Balanced policy enabled), using iperf3 in legacy mode. However, with inline mode enabled instead (and still using the Balanced policy) that throughput number reduced sharply to around 200Mbit/s.
- I also saw a large number of netmap transmit errors during the upload portion of WAN speed test (I have a symmetric 1Gbit fiber connection), as described here: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces Perhaps my system needs to have a faster CPU - it is Xeon D-1518 based which has worked quite well up now.
Ultimately, I decided to go back to legacy mode for now while I investigate these netmap performance issues further. Do you have any ideas why performance might be so bad? I do have all the offloading and flow control options disabled, but it didn't really seem to help.
I also have one quick question for you: While Snort is in legacy mode, does it matter what the IPS Policy Mode is set to under interface categories (i.e. either Alert or Policy)? Am I correct in thinking that it actually does not matter in legacy mode as long as Block Offenders is checked?
Thanks in advance for your help, I really appreciate it.
The netmap device is a complicated beast and has gone through several generations of tweaks on FreeBSD. There has also just been a big change in the overall NIC hardware driver interface introduced in FreeBSD-12. That new interface is called iflib. So in FreeBSD-12, netmap functionality is handled within iflib, and NIC manufacturers create their drivers for FreeBSD-12 to use iflib. So netmap interfacing comes without any extra work by the NIC hardware drive author. None of that is the case on FreeBSD-11. There, each hardware NIC driver vendor is responsible for their own netmap intefacing. Some do it better than others.
Another complicating issue is that netmap is not designed to work well with VLANs. So when you have VLANs, netmap is not going to be a good idea for you in terms of performance. You also will need to disable any and all hardware-level VLAN options on the NIC.
Lastly, there is a different netmap interface API version in FreeBSD-11 versus FreeBSD-12. Only FreeBSD-12 supports multiple host rings in netmap. FreeBSD-11 exposes only a single pair of Tx/Rx host rings.
Netmap is really designed to rapidly move data from one physical NIC to another physical NIC, but it does not work as well when exchanging data between a physical NIC and the host stack (or the kernel itself). The implementation on pfSense needs netmap to exchange data between a NIC and the host stack.
There have been some recent commits (software updates) made to the netmap portion of the iflib code in FreeBSD-12. Some of those changes explicitly addressed netmap transmit errors. Apparently there is a bug in the iflib code related to them. However, since pfSense-2.4.5 is based on FreeBSD-11, those iflib updates will not get into pfSense-2.4.5.
So short answer to the long story above is netmap is going to be selective feature that works well for some, but not for others. And on FreeBSD-11 it is also going to be hardware dependent. And on both pfSense-2.4.5 and pfSense-2.5, if you are using VLANs, then netmap is not an ideal choice for you with the IDS/IPS.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
I also have one quick question for you: While Snort is in legacy mode, does it matter what the IPS Policy Mode is set to under interface categories (i.e. either Alert or Policy)? Am I correct in thinking that it actually does not matter in legacy mode as long as Block Offenders is checked?
Thanks in advance for your help, I really appreciate it.
No, that setting has no influence when the operational mode is Legacy Blocking. In fact, I really meant for that option to be hidden when an interface is running in Legacy Mode. I will need to fix that so it does not even display when using Legacy Mode on an interface.
-
@tman222
I found the change that is causing your problem. However, that code has been running over in pfSense-2.5 DEVEL for over a year, so it's weird that nobody running Snort 4.x on the DEVELOPMENT snapshots has encountered this. It's caused by the extra long name of your real interfaces with the VLAN ID tacked on the end.A typical NIC real interface name is three or four characters such as em0, bce1, etc. An interface with that name length won't hit the suffix name limit defined by the Snort binary, even when a VLAN ID is tacked on (unless the VLAN ID is really long). Your NIC real interface shows up as vtnet2, so that is a bit longer than normal. That, plus your three digit VLAN ID, is what puts you over the limit when the underscore character is included.
The code change was made in the 4.x branch of Snort to address some other issues with the older technique used in the 3.x Snort branch of passing a randomly-generated GUID as part of the PID file name. I can possibly revert back to that method, but I need to carefully review all of the 4.x PHP code to be sure I find and fix all the places that reference the PID filename.
The only thing you can do in the interim is shorten that real interface name. The only method I think you have is to shorten the VLAN ID, but that of course likely requires other changes in your system that you may not wish to undertake.
-
Hi there,
I just upgraded from 2.4.4.3 to newest build, prior to the upgrade I upgraded snort and pfblocker.
After the upgrade snort would not start and I received a similar error, Ialso noticed that the snort rules failed to download at first, but i managed to download them manually.
Sep 19 03:20:10 php /tmp/snort_bge0_startcmd.php: The command '/usr/local/bin/snort -R _bge0 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_bge022424 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 22424 -c /usr/local/etc/snort/snort_22424_bge0/snort.conf -i bge0' returned exit code '1', the output was 'ERROR size 1464 != 1448'
my WAN adapter is named bge0 so nothing unusual and I do not have VLAN tagging enabled in pfsense as that is handled on the switch.
The only other thing i have changed apart from the upgrade was that I disabled PXE boot rom from the NIC at server boot.
This the first integrated adapter out of two on a HP dl385 G4 server dual Xeon CPU with 6gb Mem, it has run for years and never failed an upgrade.
Please let me know if you want any other info from the system.Kind Regards
Sebastian
-
@mikrosec said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Hi there,
I just upgraded from 2.4.4.3 to newest build, prior to the upgrade I upgraded snort and pfblocker.
After the upgrade snort would not start and I received a similar error, Ialso noticed that the snort rules failed to download at first, but i managed to download them manually.
Sep 19 03:20:10 php /tmp/snort_bge0_startcmd.php: The command '/usr/local/bin/snort -R _bge0 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_bge022424 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 22424 -c /usr/local/etc/snort/snort_22424_bge0/snort.conf -i bge0' returned exit code '1', the output was 'ERROR size 1464 != 1448'
my WAN adapter is named bge0 so nothing unusual and I do not have VLAN tagging enabled in pfsense as that is handled on the switch.
The only other thing i have changed apart from the upgrade was that I disabled PXE boot rom from the NIC at server boot.
This the first integrated adapter out of two on a HP dl385 G4 server dual Xeon CPU with 6gb Mem, it has run for years and never failed an upgrade.
Please let me know if you want any other info from the system.Kind Regards
Sebastian
That error is not related the other poster's error in this thread. His error message clearly stated the PID suffix was too long.
While it does not say so, your error seems like it might be an MTU or snaplen thing. Is this interface by chance a PPPoE interface?
LATER FOLLOW-UP:
Okay, did some Google research and found this error can be caused by having incorrect versions of some shared libraries. If you upgraded from pfSense-2.4.4_p3, that is clearly possible.
Go to the GLOBAL SETTINGS tab and verify that the "Keep Settings After Deinstall" setting is enabled (box checked). That is the default, but check to be sure. Then go to the SYSTEM > PACKAGE MANAGER tab and then Installed Packages and delete the Snort package. Let the package removal complete. Then go to the Available Packages tab and install it again. Report back on how that works.
-
It's a normal Ethernet Link nothing special, but you are absolutely right it is not related I just found this error:
98858 FATAL ERROR: Failed to initialize dynamic preprocessor: SF_MODBUS version 1.1.1 (-2)
I will investigate and open another thread if necessary, sorry for barging in.
KInd Regards
-
@mikrosec said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
It's a normal Ethernet Link nothing special, but you are absolutely right it is not related I just found this error:
98858 FATAL ERROR: Failed to initialize dynamic preprocessor: SF_MODBUS version 1.1.1 (-2)
I will investigate and open another thread if necessary, sorry for barging in.
KInd Regards
Follow my instructions in the "LATER FOLLOW-UP" edit I made to my previous post above. That should correct your problem.
-
@bmeeks Thank you for that!
I checked that "Click to retain Snort settings after package removal." was enabled and deinstalled and reinstalled smoothly without any errors. And it started up smoothly, like it has for 8 years now :-)
Saved my night
Cheers.
-
@mikrosec said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks Thank you for that!
I checked that "Click to retain Snort settings after package removal." was enabled and deinstalled and reinstalled smoothly without any errors. And it started up smoothly, like it has for 8 years now :-)
Saved my night
Cheers.
The
pkg
utilty that FreeBSD uses to install and update packages sometimes misses updating certain shared libraries, particularly when changing base OS versions such as when updating from pfSense-2.4.4 to 2.4.5 (that was a FreeBSD operating system update). In this case, some of the Snort preprocessor libs did not get deleted and the new versions installed. -
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
I found the change that is causing your problem. However, that code has been running over in pfSense-2.5 DEVEL for over a year, so it's weird that nobody running Snort 4.x on the DEVELOPMENT snapshots has encountered this. It's caused by the extra long name of your real interfaces with the VLAN ID tacked on the end.
A typical NIC real interface name is three or four characters such as em0, bce1, etc. An interface with that name length won't hit the suffix name limit defined by the Snort binary, even when a VLAN ID is tacked on (unless the VLAN ID is really long). Your NIC real interface shows up as vtnet2, so that is a bit longer than normal. That, plus your three digit VLAN ID, is what puts you over the limit when the underscore character is included.
The code change was made in the 4.x branch of Snort to address some other issues with the older technique used in the 3.x Snort branch of passing a randomly-generated GUID as part of the PID file name. I can possibly revert back to that method, but I need to carefully review all of the 4.x PHP code to be sure I find and fix all the places that reference the PID filename.
The only thing you can do in the interim is shorten that real interface name. The only method I think you have is to shorten the VLAN ID, but that of course likely requires other changes in your system that you may not wish to undertake.
Thanks for the quick check! :)
Can you estimate how long you will need for the fixing this?
-
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
Is there any way to roll back to the old package version?
-
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
I am unable to reproduce your problem. I changed a rule from DROP to ALERT and it showed up as "user-forced Alert" on the RULES tab.
You do understand that when making a change on the ALERTS tab that the icon for the past alert will not change. If the rule was DROP when the rule was triggered, that is the action written to the log file and thus will be the action displayed. However, it will be different on the RULES tab if you force an action change on the ALERTS tab and then go check the same rule on the RULES tab.
Lastly, the action change dialog pops up as what Bootstrap calls a Modal Dialog. It's possible that some security settings in your browser might be preventing that from displaying. Are you saying you never see the Modal Dialog box, or you see the box and make a change but it does not "stick"?
Is there any way to roll back to the old package version?
No, there is no way to roll back. The old package is removed from the repository.
-
@Beerman said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Can you estimate how long you will need for the fixing this?
It will be several days. Perhaps as long as a week. Once I make the change, the update has to be reviewed and approved by the pfSense team before it gets posted in the package repositories.
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
I am unable to reproduce your problem. I changed a rule from DROP to ALERT and it showed up as "user-forced Alert" on the RULES tab.
You do understand that when making a change on the ALERTS tab that the icon for the past alert will not change. If the rule was DROP when the rule was triggered, that is the action written to the log file and thus will be the action displayed. However, it will be different on the RULES tab if you force an action change on the ALERTS tab and then go check the same rule on the RULES tab.
Lastly, the action change dialog pops up as what Bootstrap calls a Modal Dialog. It's possible that some security settings in your browser might be preventing that from displaying. Are you saying you never see the Modal Dialog box, or you see the box and make a change but it does not "stick"?
There is nothing happens, when I click to any icon...
What for security settings do you mean?Is there any way to roll back to the old package version?
No, there is no way to roll back. The old package is removed from the repository.
Is there any plan to implement this. I had a lot of issues in the past years, and think it would help to make pfsense more professional. -
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@dotsch said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
After upgrade to the new version, I am unsable to perform a "rule action" to modify rules within the Snort alerts. Happens with Firefox and Safari.
Edit: Happens also directly within the rules editor of the interfaces.
I am unable to reproduce your problem. I changed a rule from DROP to ALERT and it showed up as "user-forced Alert" on the RULES tab.
You do understand that when making a change on the ALERTS tab that the icon for the past alert will not change. If the rule was DROP when the rule was triggered, that is the action written to the log file and thus will be the action displayed. However, it will be different on the RULES tab if you force an action change on the ALERTS tab and then go check the same rule on the RULES tab.
Lastly, the action change dialog pops up as what Bootstrap calls a Modal Dialog. It's possible that some security settings in your browser might be preventing that from displaying. Are you saying you never see the Modal Dialog box, or you see the box and make a change but it does not "stick"?
There is nothing happens, when I click to any icon...
What for security settings do you mean?Is there any way to roll back to the old package version?
No, there is no way to roll back. The old package is removed from the repository.
Is there any plan to implement this. I had a lot of issues in the past years, and think it would help to make pfsense more professional.I assume that you do know you can't modify the action of a rule when you are using Legacy Mode Blocking. The option to modify a rule's action is only enabled when you use Inline IPS Mode blocking. When using Legacy Mode Blocking, you can't change the action of rules and thus the icons are not "clickable". The Inline IPS Mode option is new to this version of Snort on pfSense-2.4.5 RELEASE. Previously this option was only available in pfSense-2.5 DEVEL.
-
Thanks @bmeeks for all this information, I really appreciate it!
I was able to make some progress after coming across these couple interesting links last night:
https://lists.freebsd.org/pipermail/freebsd-net/2017-March/047384.html
https://lists.freebsd.org/pipermail/svn-src-all/2016-June/126504.htmlIt turns out that Chelsio offers direct netmap support via virtual interfaces which need to be enabled separately with a sysctl tunable. The regular interfaces don't seem to support netmap natively so when I enabled inline mode in Snort, netmap was actually being emulated, which explains the poor performance I was seeing. Unfortunately I missed this key log entry the first time around, which offered a clue:
Sep 19 12:47:08 kernel 028.141828 [ 318] generic_netmap_register Emulated adapter for cxl1 activated
In any case, I went ahead and enabled the virtual inferfaces by setting the sysctl tunable
hw.cxgbe.num_vis=2
and then confirmed with dmesg that I now had netmap support for the Chelsio card:vcxl1: netmap queues/slots: TX 8/1023, RX 8/1024 vcxl1: 8 txq, 8 rxq (NIC); 8 txq, 8 rxq (netmap)
I then reassigned interface to use the virtual interface instead (i.e. instead of using cxl1 for the interface I assigned vcxl1). Everything was working well at this point, traffic was passing fine, and performance was as expected.
However, as soon I switched over from legacy mode to inline mode in Snort and started using netmap, all traffic ceased on the vcxl1 interface. If I turn Snort off or go back to legacy mode, everything works fine and traffic passes. At this point I'm stumped - I still have offloading, checksumming, flow control disabled. It does appear like all traffic is being dropped for some reason, but I'm unsure why. @bmeeks are you aware of any additional logging facilities in Snort or otherwise I could look into to try to investigate this further? Or would it maybe make sense to give Suricatta and its inline mode a try instead (if for some reason there is a problem with the Snort netmap implementation)?
Thanks again for all your help, I really appreciate it.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
However, as soon I switched over from legacy mode to inline mode in Snort and started using netmap, all traffic ceased on the vcxl1 interface. If I turn Snort off or go back to legacy mode, everything works fine and traffic passes. At this point I'm stumped - I still have offloading, checksumming, flow control disabled. It does appear like all traffic is being dropped for some reason, but I'm unsure why. @bmeeks are you aware of any additional logging facilities in Snort or otherwise I could look into to try to investigate this further? Or would it maybe make sense to give Suricatta and its inline mode a try instead (if for some reason there is a problem with the Snort netmap implementation)?
Thanks again for all your help, I really appreciate it.
Unfortunately there is no additional logging within Snort relative to netmap operation. Netmap is a kernel device, and Snort is basically ignorant of what happens internally within the device. All the Snort binary does is call an API to open an interface in netmap device mode. It gets a "success" or "failure" message, and that's about it.
Having said that, there are continuing changes happening within the netmap code in FreeBSD. This is particularly true in the new iflib wrapper API found in FreeBSD-12 and up (which is what pfSense-2.5 DEVELOPMENT is based upon). And I do know that in the not too distant past a bug was fixed in the code where the netmap device interacts with virtual devices. Perhaps that bug (which I assume still exists in the legacy FreeBSD code base) is what is biting your Chelsio virtual configuration ???
Netmap has proven to be pretty problematic with both Snort and Suricata. And it is not just on pfSense, either. The "other" firewall distro based on FreeBSD has also had the same types of netmap problems with Inline IPS Mode. Things are getting better, but the code in FreeBSD is not yet 100% there. As a result, you will find some NICs that work fine while others won't work at all. If your NIC hardware won't work with Inline IPS Mode, then you will just have to switch over to Legacy Mode Blocking. The situation is marginally better on FreeBSD-12 with the new iflib wrapper, but even there it is not yet perfected.
-
Thanks @bmeeks for this additional info - it's very helpful!
I do have some good news! I was able to make some more progress and got everything working in native netmap mode on the Chelsio interfaces. It turns out the that the missing tunable was
hw.cxgbe.fl_pktshift
which needs to be set to 0:https://lists.freebsd.org/pipermail/freebsd-net/2016-January/044433.html
I had considered setting this tunable before but didn't change it because I thought the default value was already 0. Turns out I was reading the wrong version of the FreeBSD documentation and in 11.3 it is actually set to 2 by default (Doh! It's always something simple isn't it? :)). In any case, once I set the pktshift value to 0, traffic started passing fine.
So to summarize, to use native netmap support with Chelsio cards set the following tunables in your
loader.conf.local
:hw.cxgbe.num_vis=2 hw.cxgbe.fl_pktshift=0
This will create a virtual interface for each physical interface, e.g.
cxl0
will havevcxl0
and so on. These virtual interfaces support netmap natively. You can confirm this by running dmesg and looking for netmap in the output, e.g.:dmesg | grep netmap
You should see netmap support for the Chelsio virtual interfaces, including any other interfaces in your firewall that support netmap natively. For sample output, please see my previous post above. Finally you need to assign the virtual interface(s) to your network segment(s) and enable Snort Inline mode.
==================
Unfortunately, performance is not much better for me. Even when using netmap natively on the Chelsio card (as opposed to emulated mode) and enabling Snort Inline (netmap) mode, I was still only able to pass about 200-300Mbit/s between two different 10Gbit LAN interfaces (separate physical subnets, no VLAN's) using an iperf3 test. This is very similar to what I saw in emulated netmap mode. With Snort legacy (pcap) mode, I usually get around 4Gbit/s. WAN performance is even worse and limited to 50-80 Mbit/s down/up on a symmetric 1Gbit fiber connection. I do not think the issue lies with the Chelsio netmap support because if I enable Snort inline IPS mode solely on the WAN interface which is Intel
igb
, and not on any of the Chelsio interfaces, I see the exact same WAN down/up speed limitation.@bmeeks you had mentioned this above:
Lastly, there is a different netmap interface API version in FreeBSD-11 versus FreeBSD-12. Only FreeBSD-12 supports multiple host rings in netmap. FreeBSD-11 exposes only a single pair of Tx/Rx host rings.
Do you think I may be hitting this (single host ring) limitation? Is there any way to tune Snort? Also, do you think it might be worth trying Suricata instead if they handle this interfacing differently (better?) Or would I be better off trying pfSense 2.5.0 at this point?
Thanks again for your help and insight I really appreciate it.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks you had mentioned this above:
Lastly, there is a different netmap interface API version in FreeBSD-11 versus FreeBSD-12. Only FreeBSD-12 supports multiple host rings in netmap. FreeBSD-11 exposes only a single pair of Tx/Rx host rings.
Do you think I may be hitting this (single host ring) limitation? Is there any way to tune Snort? Also, do you think it might be worth trying Suricata instead if they handle this interfacing differently (better?) Or would I be better off trying pfSense 2.5.0 at this point?
Thanks again for your help and insight I really appreciate it.
Perhaps, but there is also the issue of Snort being single-threaded. I'm not a kernel threading guru, so I'm not sure whether multi-threaded operation would help. My gut says "yeah", but there are other places where bottlenecks can occur besides just with the netmap packet processing. I don't know if you have duplicate hardware, but if you did, you could test using pfSense-2.5 snapshots. They are based on FreeBSD-12 and have the multiple host ring support (but only if the native NIC also exposes multiple rings).
Also, with Legacy Mode Blocking using PCAP, you would need to enable all the Snort stats to see if any packets are in fact getting dropped by Snort. You would not notice this on your speed test throughput because Snort is working on copies of the packets while the originals have gone on to the kernel when using Legacy Blocking. So the PCAP engine might be dropping some of the copied packets at high line rates. I think there are some stats for that when Statistics Mode is enabled (on the PREPROCESSORS tab). But I might be confusing some Suricata features here, so I don't want to be quoted on that.
When I first deployed the Inline IPS option with netmap on pfSense-2.5, the Netgate team did some testing of their own. If I recall correctly, they were able to get close to Gigabit line rates, but I don't know how many rules they had enabled. I'm thinking probably not very many. The Snort binary is likely not optimized for inline operation. Remember that Snort's history has been IDS first and foremost. The inline mode was added later as I understand it. And it's likely that code is not streamlined. Snort natively, through its DAQ library, supports two different inline modes on FreeBSD. One uses
ipfw
while the other usesnetmap
. The change I made to DAQ on pfSense was to allow thenetmap
mode to support host stack interfaces. Natively, DAQ only supported physical interfaces, so setting up an inline IPS mode between a NIC and the OS itself was not possible. You could only specify two different physical NIC ports as thenetmap
endpoints. This was not optimum for pfSense, so I added the ability to open and use host stack rings in DAQ. -
Thanks @bmeeks - I decided to take a step back and just enable inline mode on the WAN interface (Intel igb) and ran iperf3 test between my network and a cloud based VM. Doing so I see the same 200-300Mbit/s up/down transfer limit I saw when doing local testing between the two Chelsio interfaces (usually I can get ~900/900 up/down to this VM when I have Snort legacy or pcap mode enabled). This makes me wonder now if the limitation simply comes down to Snort not being fully optimized for IPS mode, my firewall's CPU is not fast enough, or a combination of both. The CPU in my firewall is a Xeon D-1518, which is a quad core 2.2GHz chip with Hyperthreading. I do see CPU usage close to maxed when using Snort inline (netmap) mode on the igb interface, but I also see the same when using Snort legacy (pcap) mode. So I'm not fully convinced yet that it's just raw CPU cycles that are limiting performance.
As such, do you think it's worth giving Suricata a try? I did have a couple questions:
- Suricata does support multi-threading, correct?
- Is Suricata better suited to be an IPS than Snort?
- Is it easy enough to migrate enough between the two, i.e. would my suppression list(s) transfer over pretty easily?
Thanks again for all your help.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
As such, do you think it's worth giving Suricata a try? I did have a couple questions:
- Suricata does support multi-threading, correct?
- Is Suricata better suited to be an IPS than Snort?
- Is it easy enough to migrate enough between the two, i.e. would my suppression list(s) transfer over pretty easily?
Thanks again for all your help.
You can certainly test, but I'm not sure Suricata will fare much better. Even its multithreaded operation still has certain bottlenecks. This used to be the big arguing point between the Snort and Suricata guys when Suricata first came out. The Snort team poo-pooed multithreaded operation -- that is until they decided to introduce it in Snort3, where they now play it up ... , so who knows what to believe.
Suricata is not necessarily better than Snort in IPS mode. Also, Suricata currently does NOT have multiple host rings support on FreeBSD-12. They are still using the older NETMAP_API version 13 interface. Only NETMAP_API version 14 has the multiple host rings.
As for your Suppress Lists, they can be freely copied and pasted if you save them off on your client PC, but Suricata will not automatically "find them" and use them when you install it. Snort and Suricata are totally independent in terms of configuration parameters. And the Suppress List is stored within the
config.xml
file of the firewall in the configuration section associated with the particular package the list is used with. So when I say copy and paste, I literally mean open your list for editing in Snort, copy all of the contents out and paste into say a Notepad file in Windows. Then install Suricata, create a Suppress List, and paste in the text from the Notepad file. If you do that, don't forget to go to the INTERFACE SETTINGS tab and assign the Suppress List to the Suricata interface where it belongs. -
Thanks @bmeeks - I did end up running a few more speed tests with inline mode enabled on just the WAN (igb) interface and (oddly) saw quite a range of numbers: From as little as 100Mbit/s up/down (speedtest.net) to almost full line speed (Netflix's Fast.com) with several tests coming in between 200 to 300 Mbit/s (similar to what I saw with iperf3). I did only have a handful of ET rules enabled on the WAN interface, but even adding in the Connectivity rule set didn't have too big of an impact on speed (just a slight decrease). I also took some time to study the netmap documentation:
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4&manpath=FreeBSD+11.3-RELEASE
Thanks for providing some additional insight above about how you choose to change DAQ to use a netmap connection to the host stack. I'm trying to visualize how this all fits together so I can narrow down the relevant bottlechecks - would be it be something like this?
NIC --> Netmap --> Snort --> Netmap --> Host Stack
I'm still not convinced yet that a lack of processing power is the primary issue for me - the bottleneck could very well be the single host ring between netmap and the host stack, as the (Chelsio NIC) does support multiple netmap queues on it's virtual interfaces. I did enable preprocessing statistics as well on the WAN (igb) interface and looked at the output, but it's not giving me much insight where the majority of CPU time is being spent (to help narrow down potentially alleviate any bottlenecks). I thought I also read somewhere that there is a way to profile Snort in order to see that - do you know if that's possible to do on the pfSense / FreeBSD implementation?
In any case, I think further exploratory analysis is probably warranted before switching over to Suricata. I do have some additional hardware lying around here to build a test system for pfSense 2.5, including an additional Intel and Chelsio NIC, along with a CPU that will have better single thread performance (Intel i3-8100 3.6GHz quad core). I would first need to test with 2.4.5 to see if the CPU is indeed the primary bottleneck and then move up to 2.5 to see if now having the additional host rings will increase throughput. In the meantime, I will probably explore the netmap tunable parameters a bit more to see if any of them might make a noticeable difference on performance (especially parameters related to buffer and ring sizing).
Thanks again for all your help!
-
@tman222
I don't know the specific setup of youriperf3
testing, but when usingiperf
you should test THROUGH the firewall and not TO the firewall. In other words, do not put either theiperf
client or server on the firewall. They both should instead be on workstations or servers hanging off two different firewall interfaces so that you test through the firewall.When you run
ipferf
on the firewall itself, whether it's the client or server piece, there is some CPU time and potential thoughput chewed up by theiperf
executable.I mention this just because some other users in the past have tested with
iperf
installed on the firewall and have gotten misleading throughput numbers as a result. -
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@tman222
I don't know the specific setup of youriperf3
testing, but when usingiperf
you should test THROUGH the firewall and not TO the firewall. In other words, do not put either theiperf
client or server on the firewall. They both should instead be on workstations or servers hanging off two different firewall interfaces so that you test through the firewall.When you run
ipferf
on the firewall itself, whether it's the client or server piece, there is some CPU time and potential thoughput chewed up by theiperf
executable.I mention this just because some other users in the past have tested with
iperf
installed on the firewall and have gotten misleading throughput numbers as a result.Hi @bmeeks - sorry for not being clear in my original post, but the iperf3 test was performed between a (bare metal) Linux host on my home network and a cloud based VM (i.e. I was testing through the firewall).
One thing I forgot to ask - when you have a moment, do you mind asking the folks at Netgate what hardware and any specific sysctl tuning they used that allowed them to get close to 1Gbit/s line rate using Snort in IPS (inline) mode? I realize it was for 2.5, but it would be great to have another data point to calibrate my results to and help me further narrow down what might be causing the performance issues. Thanks again!
-
@bmeeks - well, I might have been wrong. It is starting to look like the CPU may be the bottleneck here. I experimented and reduced the number of rules on the WAN interface and network throughput started to increase. By time I was just using the basic Connectivity IPS policy (and no ET rules), I was able to get very close to line speed using an iperf3 test (again between a local Linux host and cloud based VM). As I started to increase the number of ET rules, throughput dropped again accordingly.
From all this I gather that Snort needs a quite a beefy CPU with great single thread performance to run in IPS (inline) mode and be able to achieve high throughput (I'm actually starting to wonder now whether 1Gbit/s is even possible with a modest rule set and Snort operating in inline mode). Perhaps it's worth trying out Suricata after all to see if the multi-threading would help?
Thanks again for your help and insight.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks - well, I might have been wrong. It is starting to look like the CPU may be the bottleneck here. I experimented and reduced the number of rules on the WAN interface and network throughput started to increase. By time I was just using the basic Connectivity IPS policy (and no ET rules), I was able to get very close to line speed using an iperf3 test (again between a local Linux host and cloud based VM). As I started to increase the number of ET rules, throughput dropped again accordingly.
From all this I gather that Snort needs a quite a beefy CPU with great single thread performance to run in IPS (inline) mode and be able to achieve high throughput (I'm actually starting to wonder now whether 1Gbit/s is even possible with a modest rule set and Snort operating in inline mode). Perhaps it's worth trying out Suricata after all to see if the multi-threading would help?
Thanks again for your help and insight.
The number of enabled rules and exactly what those rules are doing will have a large impact on throughput. That's why I preach to folks to carefully select the rules they enable so that they match the potential threats faced without bogging down the CPU and thus throttling throughput. Rules that are performing complex pattern matching are naturally going to be more processor intensive than some simple rule triggering on an IP or something similarly simple to examine.
One of my favorite examples to illustrate the "enable only what you need to address the attack surfaces you expose" mantra is to not enable the DNS server, web server or email server rules if you are not running such services behind your firewall. If you don't run a DNS server, then attacks against DNS servers are of no concern in your network. And there are other examples from the inventory of rules.
-
@bmeeks I am hitting the pidfile suffix length issue as well:
FATAL ERROR: Invalid pidfile suffix: _vtnet0.100. Suffix must less than 11 characters and not have ".." or "/" in the name.
I suspect it never came up in testing because it appears to require a combination of:
- pfSense be running in a VM with the
virtio
network adapter (this is the Netgate recommended NIC type for Proxmox) - VLANs be handled in the guest, instead of the hypervisor presenting multiple virtual NICs (there are good reasons to do this in some setups)
- A 3 or 4-digit VLAN ID (perhaps most people use short VLAN IDs?)
Unfortunately, it seems it isn't really possible to rename the interface on pfSense, as it gets reverted every reboot. And re-configuring the site's VLANs isn't exactly an ideal solution either.
Instead of changing the pfSense package to create a random pidfile suffix, is the Snort team open to increasing the maximum length? 12 characters would allow 10
virtio
NICs and the maximum VLAN ID of 4095. 13 characters would allow 100virtio
NICs, which "should be enough for anybody" (famous last words). - pfSense be running in a VM with the
-
@DAVe3283 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks I am hitting the pidfile suffix length issue as well:
FATAL ERROR: Invalid pidfile suffix: _vtnet0.100. Suffix must less than 11 characters and not have ".." or "/" in the name.
I suspect it never came up in testing because it appears to require a combination of:
- pfSense be running in a VM with the
virtio
network adapter (this is the Netgate recommended NIC type for Proxmox) - VLANs be handled in the guest, instead of the hypervisor presenting multiple virtual NICs (there are good reasons to do this in some setups)
- A 3 or 4-digit VLAN ID (perhaps most people use short VLAN IDs?)
Unfortunately, it seems it isn't really possible to rename the interface on pfSense, as it gets reverted every reboot. And re-configuring the site's VLANs isn't exactly an ideal solution either.
Instead of changing the pfSense package to create a random pidfile suffix, is the Snort team open to increasing the maximum length? 12 characters would allow 10
virtio
NICs and the maximum VLAN ID of 4095. 13 characters would allow 100virtio
NICs, which "should be enough for anybody" (famous last words).I think I have a fix. I'm testing now to be sure there are no adverse side effects.
The 11-character limit imposed by the Snort binary stems, most likely, from the 16-character limit on process names in most Unix-based operating systems. The Snort binary prepends "snort" to any PID file suffix the user provides. So that's 5 characters right there, leaving 11 for the user's suffix. As you say, with certain combinations of long interface names and VLAN IDs, the user-supplied suffix can exceed 11 characters.
The only real need is to be able to uniquely identify the PID file for a given interface so the GUI can control the Snort binary. I can accomplish that using a different interface value generated each time you create a Snort interface. That is the UUID field. It's a random value between 1 and 65535, so it will never exceed a 5-character string and thus will be safely within the 11-character Snort limit. It is random when generated at the time the Snort interface is created, but stays static after that.
Look for the fix with this change in the near future. Hopefully later today if the pfSense team can quickly get the change reviewed and merged.
- pfSense be running in a VM with the
-
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@Beerman and @DAVe3283, plus any others affected by the Snort startup failure with the "PID filename suffix must be less than 11 characters" error.
The fix for this has been posted for review and merging by the pfSense developer team. Look for Snort package version 4.1.2_1 in the near future.
Thx!
-
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
Hi @bmeeks - after this update, Snort will no longer start for me. I see these errors:
/tmp/snort_vcxl0_startcmd.php: The command '/usr/local/bin/snort -R _3137 -D -q --suppress-config-log -Q --daq netmap -l /var/log/snort/snort_vcxl03137 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 3137 -c /usr/local/etc/snort/snort_3137_vcxl0/snort.conf -i vcxl0^:vcxl0' returned exit code '1', the output was ''
and
FATAL ERROR: /usr/local/etc/snort/snort_3137_vcxl0/snort.conf(174) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Any ideas?
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
Hi @bmeeks - after this update, Snort will no longer start for me. I see these errors:
/tmp/snort_vcxl0_startcmd.php: The command '/usr/local/bin/snort -R _3137 -D -q --suppress-config-log -Q --daq netmap -l /var/log/snort/snort_vcxl03137 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 3137 -c /usr/local/etc/snort/snort_3137_vcxl0/snort.conf -i vcxl0^:vcxl0' returned exit code '1', the output was ''
and
FATAL ERROR: /usr/local/etc/snort/snort_3137_vcxl0/snort.conf(174) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Any ideas?
Easy fix. Your
unicode.map
file got clobbered somehow. Simply delete the Snort package and install it again. Do NOT use the "update" icon. Use the Delete icon and then reinstall from the Available Packages tab.Not sure why some users hit this error. I have never gotten it in all the hundreds of times I've upgraded and/or green-field installed Snort on my test virtual machines.
The only possible explanation might be if you are not using any Snort Subscriber Rules. Those rules contain this file. The Emerging Threats rules do not. So if this file gets clobbered, and you don't have the Snort Subscriber Rules enabled, then the only way to get the file back is to reinstall Snort.