Suricata 2.0.3 pkg v2.0.2 - Release Notes

bmeeks

Well, it was running OK until sometime this AM when it seemingly randomly crashed and fails to restart. The only recent change to the system was enabling the DHCPv6 server, I disabled that again but it did not fix the issue.. Rebooting and reinstalling so far haven't helped. The only relevant error I found is at the end of the suricata.log below

22/9/2014 -- 10:54:02 - <notice>-- This is Suricata version 2.0.3 RELEASE
22/9/2014 -- 10:54:02 - <info>-- CPUs/cores online: 4
22/9/2014 -- 10:54:02 - <info>-- Live rule reloads enabled
22/9/2014 -- 10:54:02 - <info>-- 'default' server has 'request-body-minimal-inspect-size' set to 33882 and 'request-body-inspect-window' set to 4053 after randomization.
22/9/2014 -- 10:54:02 - <info>-- 'default' server has 'response-body-minimal-inspect-size' set to 33695 and 'response-body-inspect-window' set to 4218 after randomization.
22/9/2014 -- 10:54:02 - <info>-- HTTP memcap: 67108864
22/9/2014 -- 10:54:02 - <info>-- DNS request flood protection level: 500
22/9/2014 -- 10:54:02 - <info>-- DNS per flow memcap (state-memcap): 524288
22/9/2014 -- 10:54:02 - <info>-- DNS global memcap: 16777216
22/9/2014 -- 10:54:02 - <info>-- allocated 1572864 bytes of memory for the defrag hash... 65536 buckets of size 24
22/9/2014 -- 10:54:02 - <info>-- preallocated 65535 defrag trackers of size 144
22/9/2014 -- 10:54:02 - <info>-- defrag memory usage: 11009904 bytes, maximum: 33554432
22/9/2014 -- 10:54:02 - <info>-- AutoFP mode using "Active Packets" flow load balancer
22/9/2014 -- 10:54:02 - <info>-- preallocated 1024 packets. Total memory 3508224
22/9/2014 -- 10:54:03 - <info>-- allocated 262144 bytes of memory for the host hash... 4096 buckets of size 64
22/9/2014 -- 10:54:03 - <info>-- preallocated 1000 hosts of size 88
22/9/2014 -- 10:54:03 - <info>-- host memory usage: 366144 bytes, maximum: 16777216
22/9/2014 -- 10:54:03 - <info>-- allocated 4194304 bytes of memory for the flow hash... 65536 buckets of size 64
22/9/2014 -- 10:54:03 - <info>-- preallocated 10000 flows of size 240
22/9/2014 -- 10:54:03 - <info>-- flow memory usage: 6674304 bytes, maximum: 33554432
22/9/2014 -- 10:54:03 - <info>-- IP reputation disabled
22/9/2014 -- 10:54:03 - <info>-- Added "35" classification types from the classification file
22/9/2014 -- 10:54:03 - <info>-- Added "19" reference types from the reference.config file
22/9/2014 -- 10:54:03 - <info>-- using magic-file /usr/share/misc/magic
22/9/2014 -- 10:54:03 - <info>-- Delayed detect disabled
22/9/2014 -- 10:54:06 - <info>-- 2 rule files processed. 3736 rules successfully loaded, 0 rules failed
22/9/2014 -- 10:54:07 - <info>-- 3736 signatures processed. 190 are IP-only rules, 1004 are inspecting packet payload, 2886 inspect application layer, 0 are decoder event only
22/9/2014 -- 10:54:07 - <info>-- building signature grouping structure, stage 1: preprocessing rules... complete
22/9/2014 -- 10:54:07 - <info>-- building signature grouping structure, stage 2: building source address list... complete
22/9/2014 -- 10:54:12 - <info>-- building signature grouping structure, stage 3: building destination address lists... complete
22/9/2014 -- 10:54:13 - <info>-- Threshold config parsed: 3 rule(s) found
22/9/2014 -- 10:54:13 - <info>-- Core dump size is unlimited.
22/9/2014 -- 10:54:13 - <info>-- alert-pf output device (regular) initialized: block.log
22/9/2014 -- 10:54:13 - <error>-- [ERRCODE: SC_ERR_INVALID_ARGUMENTS(52)] - prefix or user NULL</error></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></notice>

running v2.0.2 on 2.1.5-RELEASE (amd64).

Actually, my issue ended up being the IPV6 addresses of my DNS servers added to the pass list from "Add WAN DNS servers to the list." I guess it took a day or two to update the pass lists automatically since I re-enabled IPv6 for the first time in a few months a couple days ago and added the DNS servers shortly after.

Values in a PASS LIST are only updated with a Suricata restart. They cannot be dynamically updated due to limitations within the Suricata binary. So when you make a change to a PASS LIST, you must stop and restart Suricata for those changes to become active. Could be that an overnight automatic rule update (assuming you have those enabled) finally restarted Suricata and it picked up something in the updated PASS LIST. The rules update process only restarts Suricata if an actual rules package download and update is necessary (that is, the new rules have been posted by the rules vendor).

Bill

gogglespisano

I have a new install of pfSense and Suricata. This is the only version of Suricata I've used.

Services: Suricata 2.0.3 pkg v2.0.2 - Intrusion Detection System

2.1.5-RELEASE (amd64)
built on Mon Aug 25 07:44:45 EDT 2014
FreeBSD pfsense.bedrock 8.3-RELEASE-p16 FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:27:11 EDT 2014

I hit the CSV comma bug in block.log that I thought was fixed in the previous release, so I'm confused.

This is the entry in block.log

10/09/2014-21:10:46.431197,Block Src,1,2001972,17,ET SCAN Behavioral Unusually fast Terminal Server Traffic, Potential Scan or Infection (Inbound),Misc activity,3,TCP,124.232.152.37,18785

This is the error in the system log

php: /suricata/suricata_blocked.php: [suricata] Failed block.log entry fields are: Array ( [0] => 10/09/2014-21:10:46.431197 [1] => Block Src [2] => 1 [3] => 2001972 [4] => 17 [5] => ET SCAN Behavioral Unusually fast Terminal Server Traffic [6] => Potential Scan or Infection (Inbound) [7] => Misc activity [8] => 3 [9] => TCP [10] => 124.232.152.37 [11] => 18785 )

I do see alerts and blocks in the tabs, but the error keeps getting logged. I deleted block.log and that fixed it for now, but I thought this wasn't supposed to happen again with this version?

-Stuart

bmeeks

@gogglespisano:

I have a new install of pfSense and Suricata. This is the only version of Suricata I've used.

Services: Suricata 2.0.3 pkg v2.0.2 - Intrusion Detection System

2.1.5-RELEASE (amd64)
built on Mon Aug 25 07:44:45 EDT 2014
FreeBSD pfsense.bedrock 8.3-RELEASE-p16 FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:27:11 EDT 2014

I hit the CSV comma bug in block.log that I thought was fixed in the previous release, so I'm confused.

This is the entry in block.log
10/09/2014-21:10:46.431197,Block Src,1,2001972,17,ET SCAN Behavioral Unusually fast Terminal Server Traffic, Potential Scan or Infection (Inbound),Misc activity,3,TCP,124.232.152.37,18785
This is the error in the system log
php: /suricata/suricata_blocked.php: [suricata] Failed block.log entry fields are: Array ( [0] => 10/09/2014-21:10:46.431197 [1] => Block Src [2] => 1 [3] => 2001972 [4] => 17 [5] => ET SCAN Behavioral Unusually fast Terminal Server Traffic [6] => Potential Scan or Infection (Inbound) [7] => Misc activity [8] => 3 [9] => TCP [10] => 124.232.152.37 [11] => 18785 )
I do see alerts and blocks in the tabs, but the error keeps getting logged. I deleted block.log and that fixed it for now, but I thought this wasn't supposed to happen again with this version?

-Stuart

Well, the error specifically fixed was in the ALERTS tab. The BLOCKED tab gets its data from a log file that is still in CSV format. I did put a hack in so it should not corrupt the HTML display, and it logs the offending entry to the system log as you see. I will put this fix on my TODO list.

Bill

gogglespisano

Thanks Bill for the correction. I missed the that distinction. I now see the difference in the format of alerts.log and block.log.

Being new to pfSense and Suricata, I found the learning curve quite short and the product very good, a pleasant surprise for a firewall / IDS!

I do have a couple of feature suggestions.

It would be nice to be able to set the highest priority level to block. I ended up suppressing some priority 3 rules so that I could enable blocking. I'd rather still see/log most of those priority 3 alerts but have only priority 1 and 2 alerts cause blocks to be generated.

When installing the Snort VRT rules you need to enter the rules filename. It took a while to find the filename to use and be confident I had the correct one (which turned out was the same as the GUI example). In the release description of the change to add the filename field, there was a good explanation of why it was needed, where to get it and how frequently it would change. It would be nice to capture that info into the GUI.

Thanks,
-Stuart

bmeeks

@gogglespisano:

Thanks Bill for the correction. I missed the that distinction. I now see the difference in the format of alerts.log and block.log.

Being new to pfSense and Suricata, I found the learning curve quite short and the product very good, a pleasant surprise for a firewall / IDS!

I do have a couple of feature suggestions.

It would be nice to be able to set the highest priority level to block. I ended up suppressing some priority 3 rules so that I could enable blocking. I'd rather still see/log most of those priority 3 alerts but have only priority 1 and 2 alerts cause blocks to be generated.

When installing the Snort VRT rules you need to enter the rules filename. It took a while to find the filename to use and be confident I had the correct one (which turned out was the same as the GUI example). In the release description of the change to add the filename field, there was a good explanation of why it was needed, where to get it and how frequently it would change. It would be nice to capture that info into the GUI.

Thanks,
-Stuart

I intend to fix the Block log problem. Might can sneak it into the next update. If not, it will go in the following one. That log is written by the custom blocking plugin I created for Suricata, so it's familiar territory for me.

I had actually thought about adding an option to block by priority. If more folks are interested, I could add it. I will see what I can do about the VRT filename. The Sourcefire guys messed up a good thing when they discontinued their "edge" file that always contained the latest rule update. The problem I face on the GUI is somewhat limited space, but maybe there is enough room without making it too cluttered to put a URL link to the rules file or something.

Bill

fsansfil

Related to the block.log… I get this only with long IPv6 address : ERROR: block.log entry failed to parse correctly with too many or not enough CSV entities, skipping this entry...

Oct 11 10:16:36 php-fpm[44596]: /suricata/suricata_blocked.php: [suricata] Failed block.log entry fields are: Array ( [0] => 10/09/2014-21:09:57.398584 [1] => Block Dst [2] => 1 [3] => 770058 [4] => 1 [5] => PB-PROTOCOL ICMP for IPv6 RFC 4443 [6] => RFC 4884 [7] => Detection of a Non-Standard Protocol or Event [8] => 2 [9] => IPV6-ICMP [10] => ff02:0000:0000:0000:0000:0000:0000:0001 [11] => 0 )
Oct 11 10:16:36 php-fpm[44596]: /suricata/suricata_blocked.php: [suricata] ERROR: block.log entry failed to parse correctly with too many or not enough CSV entities, skipping this entry…

F.

bmeeks

@fsansfil:

Related to the block.log… I get this only with long IPv6 address : ERROR: block.log entry failed to parse correctly with too many or not enough CSV entities, skipping this entry...

Oct 11 10:16:36 php-fpm[44596]: /suricata/suricata_blocked.php: [suricata] Failed block.log entry fields are: Array ( [0] => 10/09/2014-21:09:57.398584 [1] => Block Dst [2] => 1 [3] => 770058 [4] => 1 [5] => PB-PROTOCOL ICMP for IPv6 RFC 4443 [6] => RFC 4884 [7] => Detection of a Non-Standard Protocol or Event [8] => 2 [9] => IPV6-ICMP [10] => ff02:0000:0000:0000:0000:0000:0000:0001 [11] => 0 )
Oct 11 10:16:36 php-fpm[44596]: /suricata/suricata_blocked.php: [suricata] ERROR: block.log entry failed to parse correctly with too many or not enough CSV entities, skipping this entry…

F.

Can you also post text from the ALERTS and BLOCK log for this exact event? You can correlate the timestamp to be sure you get the exact event. I'm working now on coding a fix for this problem on the BLOCK tab. You can find these two files in the /var/log/suricata subdirectory for the interface, or you can go to the LOGS VIEW tab and select the interface and log file to view.

Bill

simby

bmeeks: do we have plane to update to suricata 2.0.4 or 2.1 ? :)

gogglespisano

Bill, can you change the blocks tab to show most recent first like the alerts tab?

-Stuart

BBcan177

@gogglespisano:

Bill, can you change the blocks tab to show most recent first like the alerts tab?

-Stuart

The column headers are sortable. Just click on any one of the headers.

gogglespisano

@BBcan177:

@gogglespisano:

Bill, can you change the blocks tab to show most recent first like the alerts tab?

-Stuart

The column headers are sortable. Just click on any one of the headers.

There's no "Date" column on Blocks like there is on "Alerts", so nothing to click to get most recent first.

I have auto refresh turned on, so having the default would be nice since the column sort doesn't stick through an auto refresh.

What is the default sort order of the Blocks? It seems mostly "recent last", but the first part of the list doesn't quite match that. Is it ordered by "earliest seen last"? (I have a 1hr expiration on blocks)

fsansfil

Logs rotated from that event, but heres a new one, same error and this time not from a custom rule.

Cheers.

sys.log

Oct 12 14:10:11	php-fpm[5131]: /suricata/suricata_blocked.php: [suricata] Failed block.log entry fields are: Array ( [0] => 10/11/2014-21:27:04.782000 [1] => Block Src [2] => 1 [3] => 22 [4] => 1 [5] => FILE pdf claimed [6] => but not pdf [7] => (null) [8] => 3 [9] => TCP [10] => 128.119.103.XXX [11] => 80 )
Oct 12 14:10:11	php-fpm[5131]: /suricata/suricata_blocked.php: [suricata] ERROR: block.log entry failed to parse correctly with too many or not enough CSV entities, skipping this entry...

alert.log

10/11/2014-21:27:04.782000  [**] [1:22:1] FILE pdf claimed, but not pdf [**] [Classification: (null)] [Priority: 3] {TCP} 128.119.103.XXX:80 -> 192.168.1.XXX:47798

block.log

10/11/2014-21:27:04.782000,Block Src,1,22,1,FILE pdf claimed, but not pdf,(null),3,TCP,128.119.103.XXX,80

rule

alert http any any -> any any (msg:"FILE pdf claimed, but not pdf"; flow:established,to_client; fileext:"pdf"; filemagic:!"PDF document"; filestore; sid:22; rev:1;)

bmeeks

@fsansfil:

Logs rotated from that event, but heres a new one, same error and this time not from a custom rule.

Cheers.

sys.log

Oct 12 14:10:11	php-fpm[5131]: /suricata/suricata_blocked.php: [suricata] Failed block.log entry fields are: Array ( [0] => 10/11/2014-21:27:04.782000 [1] => Block Src [2] => 1 [3] => 22 [4] => 1 [5] => FILE pdf claimed [6] => but not pdf [7] => (null) [8] => 3 [9] => TCP [10] => 128.119.103.XXX [11] => 80 )
Oct 12 14:10:11	php-fpm[5131]: /suricata/suricata_blocked.php: [suricata] ERROR: block.log entry failed to parse correctly with too many or not enough CSV entities, skipping this entry...

alert.log

10/11/2014-21:27:04.782000  [**] [1:22:1] FILE pdf claimed, but not pdf [**] [Classification: (null)] [Priority: 3] {TCP} 128.119.103.XXX:80 -> 192.168.1.XXX:47798

block.log

10/11/2014-21:27:04.782000,Block Src,1,22,1,FILE pdf claimed, but not pdf,(null),3,TCP,128.119.103.XXX,80

rule

alert http any any -> any any (msg:"FILE pdf claimed, but not pdf"; flow:established,to_client; fileext:"pdf"; filemagic:!"PDF document"; filestore; sid:22; rev:1;)

I'm working on the fix, but this is really just a cosmetic error. The custom module that inserts a block also writes some of the fields provided by the Suricata alert module to a log file (the block.log file). To make it easy to parse in PHP, I configured the blocking module to separate those Suricata fields with commas. PHP has a built-in function for reading a CSV text file and placing each field into an array. This technique has been used for ages in the Snort package. However, in Suricata an unforeseen problem popped up because a tiny handful of the text rules have embedded commas in the rule message. That messes up the CSV parser because it expects the 11 fields of data to be delimited by commas. When you have extra commas as part of the data of a field, the CSV parser winds up presenting PHP with 12, 13 or possibly more fields. That messes up the order in which the BLOCK tab code expects the fields to be in.

I have a fix working in my test lab and hope to push it out soon for the pfSense developers to review and hopefully approve.

Bill

fsansfil

Hey Bill,

Thanks again. Another question; I dont understand how the SID MGMT works.

Lets say I want to remove the fast_pattern:only from those SID 31944,29895,28807,28406,14081,21474,26722

I then edit the modifysid-sample.conf and add:
31944,29895,28807,28406,14081,21474,26722 "fast_pattern:only; " ""

Then select the proper interface (LAN) in the Interface SID Management File Assignements and select modify-sample.conf under Modify SID File, click save and nothing happens ;(

Please let me know what I doing wrong.

Thanks!

F.

BBcan177

Hi fsansfil,

I would suggest you make a new modify-sid (and enable/disable sid also) because if you use the "Sample" Ones, they will be overwritten when you do any re-install or Package Updates.

Did you click the "Rebuild" Checkboxes beside the Interface Names?

fsansfil

Hey there,

Thanks for the quick reply. Just tested it again, this time with a newly created modify.config, did checked rebuild in front of the interface (LAN), still nothing. Seems the check rebuild button dont save, every refresh it goes back to uncheck…

Thanks.

F.

BBcan177

The Rebuild Checkbox does not stay checked after you hit save…

Once you make you Sid Mgmt Changes, Click "Rebuild" then save.

Goto the Interface (ie: LAN) click "Edit"

LAN Rules: and Look for the Rule that you modified to see if the changes were made to the Rule.

gogglespisano

Here's a good example of the need to have control over blocking by the alert priority level.

These are from windows update trying to download new updates today.

10/15/2014-08:57:59.102113,Block Src,1,2015744,3,ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging),Misc activity,3,TCP,96.17.15.72,80
10/15/2014-08:59:32.716282,Block Src,1,2014819,3,ET INFO Packed Executable Download,Misc activity,3,TCP,96.17.15.96,80

If the IPS cries wolf too much it'll get turned off. Other than during the initial setup, I've only had one alert with pri < 3 that's been a false positive. I've had a couple dozen pri 3 that have been false positives since the initial setup about a week ago. The choice seems to be between constant monitoring for false positives or turning off large blocks or categories of rules, neither of which I'd like to do.

Is there a simple solution to this situation I've missed?

-Stuart

bmeeks

@gogglespisano:

Here's a good example of the need to have control over blocking by the alert priority level.

These are from windows update trying to download new updates today.
10/15/2014-08:57:59.102113,Block Src,1,2015744,3,ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging),Misc activity,3,TCP,96.17.15.72,80
10/15/2014-08:59:32.716282,Block Src,1,2014819,3,ET INFO Packed Executable Download,Misc activity,3,TCP,96.17.15.96,80
If the IPS cries wolf too much it'll get turned off. Other than during the initial setup, I've only had one alert with pri < 3 that's been a false positive. I've had a couple dozen pri 3 that have been false positives since the initial setup about a week ago. The choice seems to be between constant monitoring for false positives or turning off large blocks or categories of rules, neither of which I'd like to do.

Is there a simple solution to this situation I've missed?

-Stuart

There is currently no capability for filtering by PRIORITY for block actions in the Suricata or Snort packages. That will require rewrites of the custom patch that is compiled into both Snort and Suricata in order to implement blocking by interfacing with the pf firewall in pfSense. Just thinking about it off the cuff, I don't see too much of a problem implementing it in the near future. But that is just off the cuff without a detailed code review. Could be there is something that might prove to be an obstacle to implementing the feature.

Bill

bmeeks

@fsansfil:

Hey there,

Thanks for the quick reply. Just tested it again, this time with a newly created modify.config, did checked rebuild in front of the interface (LAN), still nothing. Seems the check rebuild button dont save, every refresh it goes back to uncheck…

Thanks.

F.

BBcan177 is correct in describing the steps. The new SID MGMT tab takes the configuration files and uses the instructions within to create a list of rules Suricata will use for inspecting traffic. Literally, the primary purpose of the SID MGMT tab itself is to let you upload, edit and assign various text configuration files to your interfaces. Saving simply does this part. The actual creation of the new list of rules for Suricata to use for network traffic inspection happens in these three cases:

1. You click the REBUILD checkbox next to the interface on the SID MGMT tab. That will not only save the file assignments as described above, but will trigger the creation of a new list of rules for the interface. Once the list is created, the running Suricata process is sent a "live rule reload" signal. You will see this in the suricata.log file viewed in the LOGS VIEW tab for the interface. As BBcan177 said, you can then go to the CATEGORIES or RULES tabs for the interface and see the new color-coded icons beside the impacted text rules.

2. Once auto-SID management is enabled, every time you click SAVE on any other interface tabs, the list of rules is rebuilt using the logic in the assigned SID MGMT configuration files.

3. Finally, when an automatic download of updated rules occurs via the scheduled update job, a new rules list is generated and Suricata is signaled to load it.

Bill