reference.config - overwritten after edit?

boobletins

@bmeeks said in reference.config - overwritten after edit?:

The suricata.yaml file in /usr/local/etc/suricata is never used by the GUI package for anything. That is simply the default file put there by the installation of the binary.

Ah, yes, I'm learning with pfSense that very little of the configuration should be done in the default config files. Thanks for being patient with me, it's appreciated.

I'm still not sure why you want to do this. How are you going to keep your local file in sync with any changes that creep in from rule updates?

The goal here is twofold:

eliminate log spam so the suricata logs are more useful to me
modify alerts page so its more useful to me (by including reference urls)

cat /var/log/suricata/suricata_igb043414/suricata.log | grep -o ERR_REFERENCE_UNKNOWN | wc -l

shows 2075 occurances of "bad" references (lots of bid and md5 references which I believe appear in the default reference file)

References get changed by the rule package vendors from time to time. That's why the GUI code rebuilds that file (along with classification.config with each rules update run).

I was judging that it would be less work to maintain the references.config.local than it would to modify the php code that creates the references config file, but of course that is foolish.

boobletins

@bmeeks said in reference.config - overwritten after edit?:

I'm still not sure why you want to do this. How are you going to keep your local file in sync with any changes that creep in from rule updates?

I probably wasn't very clear above. It sounds like I was just trying to clear up log spam. I was doing that, but I maybe I should clarify that because the reference file that the package constructs doesn't contain the necessary references, something like 2000+ rules from ET error out. In addition to polluting the logs, these rules are inactive.

Correcting this by modifying the reference.config file allows those rules to be properly parsed.

With corrected references.config.local file:
3/12/2018 -- 19:19:02 - <Info> -- 2 rule files processed. 28925 rules successfully loaded, 55 rules failed

(these failures are mostly Snort v2 rules which are invalid in Suricata)

With references.config as constructed by the Suricata package:

3/12/2018 -- 18:59:44 - <Info> -- 2 rule files processed. 26848 rules successfully loaded, 2132 rules failed

That's something like 1/8th of the ET rules that fail because the reference.config file can't account for references that appear in the rules themselves.

I haven't dug into how the package is constructing the references.config file each time, but one way to resolve this would be to add any reference keyword to references.config -- no matter what the file from ET contains.

In other words: if "reference:cve," or "reference:thisisabadreference," appear in the .rules files, then they are automatically added to reference.config just so the rules can be properly parsed.

boobletins

From suricata_check_for_rule_updates.php:

/******************************************************************/
        /* Build the classification.config and reference.config files     */
        /* using the ones from all the downloaded rules plus the default  */
        /* files installed with Suricata.                                 */
        /******************************************************************/
        $cfgs = glob("{$tmpfname}/*reference.config");
        $cfgs[] = "{$suricatadir}reference.config";
        suricata_merge_reference_configs($cfgs, "{$suricatadir}reference.config");
        $cfgs = glob("{$tmpfname}/*classification.config");
        $cfgs[] = "{$suricatadir}classification.config";
        suricata_merge_classification_configs($cfgs, "{$suricatadir}classification.config");

It doesn't look to me like this is actually using the default that is installed with Suricata (reference.config.sample) which does contain references like md5 and similar.

see: https://github.com/OISF/suricata/blob/master/reference.config

Could $cfgs be modified in your code base to either include a .local file in addition to the downloaded and sample files, or at least include the .sample file from the Suricata codebase? That would activate something like 2000 additional rules for users.

Here is the references.config file that is generated from a rule update without my modifications in the yaml.inc file (which you correctly point out aren't particularly maintainable):

config reference: arachNIDS    http://www.whitehats.com/info/IDS
config reference: bugtraq      http://www.securityfocus.com/bid/
config reference: cve          http://cve.mitre.org/cgi-bin/cvename.cgi?name=
config reference: McAfee       http://vil.nai.com/vil/content/v_
config reference: msb          http://technet.microsoft.com/en-us/security/bulletin/
config reference: nessus       http://cgi.nessus.org/plugins/dump.php3?id=
config reference: osvdb        http://osvdb.org/show/osvdb/
config reference: url          http://

Here's my reference.config.local which enables the rules by preventing the parsing errors:

# config reference: system URL

config reference: bugtraq   http://www.securityfocus.com/bid/
config reference: bid       http://www.securityfocus.com/bid/
config reference: cve       http://cve.mitre.org/cgi-bin/cvename.cgi?name=
#config reference: cve       http://cvedetails.com/cve/
config reference: secunia   http://www.secunia.com/advisories/

#whitehats is unfortunately gone
config reference: arachNIDS http://www.whitehats.com/info/IDS

config reference: McAfee    http://vil.nai.com/vil/content/v_
config reference: nessus    http://cgi.nessus.org/plugins/dump.php3?id=
config reference: url       http://
config reference: et        http://doc.emergingthreats.net/
config reference: etpro     http://doc.emergingthreatspro.com/
config reference: telus     http://
config reference: osvdb     http://osvdb.org/show/osvdb/
config reference: threatexpert http://www.threatexpert.com/report.aspx?md5=
config reference: md5       http://www.threatexpert.com/report.aspx?md5=
config reference: exploitdb http://www.exploit-db.com/exploits/
config reference: openpacket https://www.openpacket.org/capture/grab/
config reference: securitytracker http://securitytracker.com/id?
config reference: secunia   http://secunia.com/advisories/
config reference: xforce    http://xforce.iss.net/xforce/xfdb/
config reference: msft      http://technet.microsoft.com/security/bulletin/
config reference: msb       http://technet.microsoft.com/en-us/security/bulletin/

boobletins

In fact, it doesn't look like ET provides a reference.config file in each iteration of the rules at all. Unless I'm missing something?

https://rules.emergingthreats.net/open/suricata-4.0/

classification.config is in there, but I don't see a references.config?

This makes a certain amount of sense in that users may want to modify the reference links if they change over time without having to modify the signatures themselves.

If I want to use another CVE site in lieu of cve.mitre.org, for example.

I think the pfSense package needs to handle this differently and either allow for the file to be user managed or automatically generate the references based on a parsing of the .rules files -- the current system is broken if references.config isn't updated by ET.

bmeeks

@boobletins said in reference.config - overwritten after edit?:

In fact, it doesn't look like ET provides a reference.config file in each iteration of the rules at all. Unless I'm missing something?

https://rules.emergingthreats.net/open/suricata-4.0/

classification.config is in there, but I don't see a references.config?

This makes a certain amount of sense in that users may want to modify the reference links if they change over time without having to modify the signatures themselves.

If I want to use another CVE site in lieu of cve.mitre.org, for example.

I think the pfSense package needs to handle this differently and either allow for the file to be user managed or automatically generate the references based on a parsing of the .rules files -- the current system is broken if references.config isn't updated by ET.

I use Snort for my home system so I have not noticed this before. There is a references.config file included in at least the ET-Pro rules (or at least the last archive I have of that set from earlier this year contains one). A quick perusal of it seems to show it matches the default references.config file packaged with the Suricata source code from upstream.

I can add the ability to use a custom references.config to the package. I will look into what's going on with the default references.config. It should at least have the md5 reference in it. Maybe the one from the Snort package got accidentially included with the Suricata package on pfSense.

Give me a few days to look into it.

bmeeks

@boobletins
I'm having trouble duplicating what you reported. I fired up a virtual machine with the Suricata package installed and configured it to use the ET-Open rules on the LAN interface. I enable every single ET-Open category listed on the CATEGORIES tab. I then stopped and restarted Suricata on the interface. Here is the suricata.log file output from that exercise:

4/12/2018 -- 12:14:01 - <Notice> -- This is Suricata version 4.0.6 RELEASE
4/12/2018 -- 12:14:01 - <Info> -- CPUs/cores online: 2
4/12/2018 -- 12:14:01 - <Info> -- Netmap: Setting IPS mode
4/12/2018 -- 12:14:01 - <Info> -- HTTP memcap: 67108864
4/12/2018 -- 12:14:01 - <Notice> -- using flow hash instead of active packets
4/12/2018 -- 12:14:09 - <Info> -- 1 rule files processed. 18222 rules successfully loaded, 0 rules failed
4/12/2018 -- 12:14:09 - <Info> -- Threshold config parsed: 0 rule(s) found
4/12/2018 -- 12:14:09 - <Info> -- 18225 signatures processed. 1167 are IP-only rules, 5337 are inspecting packet payload, 13701 inspect application layer, 103 are decoder event only
4/12/2018 -- 12:14:14 - <Info> -- fast output device (regular) initialized: alerts.log
4/12/2018 -- 12:14:14 - <Info> -- http-log output device (regular) initialized: http.log
4/12/2018 -- 12:14:14 - <Info> -- Using 2 live device(s).
4/12/2018 -- 12:14:14 - <Notice> -- all 4 packet processing threads, 2 management threads initialized, engine started.

As you see, none of the 18,222 rules failed to load. What were you doing differently, and did you force enable a ton of other rules to reach your 26,848 number? Or is that including the Snort rules you also added?

And here is the reference.config file from the LAN interface on the virtual machine:

config reference: arachNIDS    http://www.whitehats.com/info/IDS
config reference: bid          http://www.securityfocus.com/bid/
config reference: bugtraq      http://www.securityfocus.com/bid/
config reference: cve          http://cve.mitre.org/cgi-bin/cvename.cgi?name=
config reference: et           http://doc.emergingthreats.net/
config reference: etpro        http://doc.emergingthreatspro.com/
config reference: exploitdb    http://www.exploit-db.com/exploits/
config reference: McAfee       http://vil.nai.com/vil/content/v_
config reference: md5          http://www.threatexpert.com/report.aspx?md5=
config reference: msb          http://technet.microsoft.com/en-us/security/bulletin/
config reference: msft         http://technet.microsoft.com/security/bulletin/
config reference: nessus       http://cgi.nessus.org/plugins/dump.php3?id=
config reference: openpacket   https://www.openpacket.org/capture/grab/
config reference: osvdb        http://osvdb.org/show/osvdb/
config reference: secunia      http://www.secunia.com/advisories/
config reference: securitytracker http://securitytracker.com/id?
config reference: telus        http://
config reference: threatexpert http://www.threatexpert.com/report.aspx?md5=
config reference: url          http://
config reference: xforce       http://xforce.iss.net/xforce/xfdb/

boobletins

As you see, none of the 18,222 rules failed to load. What were you doing differently, and did you force enable a ton of other rules to reach your 26,848 number? Or is that including the Snort rules you also added?

Yes, I also run the subscriber Snort v2 rules.

Other potential differences:

I believe I've upgraded pfSense twice since installing (without package removal)
I have run the Snort rules in several modes including IPS Maximum and IPS Security. The Snort rules are currently set to category mode and are all enabled.

I see a single reference.config file in /usr/local/etc/suricata -- it contains:

/usr/local/etc/suricata: cat reference.config
config reference: arachNIDS    http://www.whitehats.com/info/IDS
config reference: bugtraq      http://www.securityfocus.com/bid/
config reference: cve          http://cve.mitre.org/cgi-bin/cvename.cgi?name=
config reference: McAfee       http://vil.nai.com/vil/content/v_
config reference: msb          http://technet.microsoft.com/en-us/security/bulletin/
config reference: nessus       http://cgi.nessus.org/plugins/dump.php3?id=
config reference: osvdb        http://osvdb.org/show/osvdb/
config reference: url          http://

If I compare that to the reference.config available at the Snort website, it looks quite similar.

Is it possible the reference.config from Snort v2 rules is overwriting the ET Open version?

bmeeks

@boobletins said in reference.config - overwritten after edit?:

As you see, none of the 18,222 rules failed to load. What were you doing differently, and did you force enable a ton of other rules to reach your 26,848 number? Or is that including the Snort rules you also added?

Yes, I also run the subscriber Snort v2 rules.

Other potential differences:

I believe I've upgraded pfSense twice since installing (without package removal)

I have run the Snort rules in several modes including IPS Maximum and IPS Security. The Snort rules are currently set to category mode and are all enabled.

I see a single reference.config file in /usr/local/etc/suricata -- it contains:
/usr/local/etc/suricata: cat reference.config
config reference: arachNIDS    http://www.whitehats.com/info/IDS
config reference: bugtraq      http://www.securityfocus.com/bid/
config reference: cve          http://cve.mitre.org/cgi-bin/cvename.cgi?name=
config reference: McAfee       http://vil.nai.com/vil/content/v_
config reference: msb          http://technet.microsoft.com/en-us/security/bulletin/
config reference: nessus       http://cgi.nessus.org/plugins/dump.php3?id=
config reference: osvdb        http://osvdb.org/show/osvdb/
config reference: url          http://
If I compare that to the reference.config available at the Snort website, it looks quite similar.

Is it possible the reference.config from Snort v2 rules is overwriting the ET Open version?

No, it shouldn't. I also have the Snort rules enabled on the virtual machine I just tested with. What should happen is the reference.config packaged with the Snort v2 rules gets combined with the reference.config packaged with the ET rules, and then any duplicate lines are removed.

The only thing I can imagine at this point is for some reason your reference.config file from Snort is not getting removed when the package is deleted. That can happen if you modify a file. At that point the pkg utility in FreeBSD/pfSense will see the file's checksum is different from when it was installed and it will leave it in place even when removing the package. So manually delete that reference.config file from all locations, then force a rule update again. See if that produces a working file for you. You can keep a backup someplace like in /root if you wish before deleting reference.config.

boobletins

Yes, it looks like that's what's happening. The comments are stripped out, and then they are alphabetized (I assume you're merging them in code somewhere) -- but only the Snort v2 references survive. The ET / Suricata default references are clobbered?

bmeeks

Here is where in the rules update process the reference.config file is created for each interface. This is in the file /usr/local/pkg/suricata/suricata_check_for_rule_updates.php. The code starts on line 618.

	/******************************************************************/
	/* Build the classification.config and reference.config files     */
	/* using the ones from all the downloaded rules plus the default  */
	/* files installed with Suricata.                                 */
	/******************************************************************/
	$cfgs = glob("{$tmpfname}/*reference.config");
	$cfgs[] = "{$suricatadir}reference.config";
	suricata_merge_reference_configs($cfgs, "{$suricatadir}reference.config");

The function suricata_merge_reference_configs() can be found in the file /usr/local/pkg/suricata/suricata.inc.

boobletins

/usr/local/etc/suricata: find / -name reference.config

/usr/local/etc/suricata/suricata_9398_em0/reference.config
/usr/local/etc/suricata/suricata_43695_igb0/reference.config
/usr/local/etc/suricata/reference.config

I removed all of those and performed a force update to rules.

Here is the resulting reference.config:

config reference: arachNIDS    http://www.whitehats.com/info/IDS
config reference: bugtraq      http://www.securityfocus.com/bid/
config reference: cve          http://cve.mitre.org/cgi-bin/cvename.cgi?name=
config reference: McAfee       http://vil.nai.com/vil/content/v_
config reference: msb          http://technet.microsoft.com/en-us/security/bulletin/
config reference: nessus       http://cgi.nessus.org/plugins/dump.php3?id=
config reference: osvdb        http://osvdb.org/show/osvdb/
config reference: url          http://

Please forgive my nonexistent php skills.

It looks to me like the referenced input files are:

$cfgs = glob("{$tmpfname}/*reference.config");
$cfgs[] = "{$suricatadir}reference.config";

Presumably $cfgs[] is /usr/local/etc/suricata/reference.config plus whatever is in the $tmpfname?

The output from suricata_merge_reference_configs() is stored back into the same {$suricatadir}reference.config.

So it's an additive process? If that's the case, then if references.config is ever modified, data may be permanently lost?

Maybe what I need to do is replace /usr/local/etc/suricata/reference.config with the original from Suricata?

boobletins

Yes, that worked.

So it's possible that either I manually edited reference.config in the past or clobbered it by unzipping Snort rules there or something.

My apologies Bill -- looks like I managed to break it without knowing and assumed it was broken in the code.

bmeeks

@boobletins said in reference.config - overwritten after edit?:

Yes, that worked.

So it's possible that either I manually edited reference.config in the past or clobbered it by unzipping Snort rules there or something.

My apologies Bill -- looks like I managed to break it without knowing and assumed it was broken in the code.

Glad you got it sorted out.

The $cfgs variable is an array in PHP. It is loaded with the filenames of all the reference.config files it finds in the /tmp directory where the rules tarball is unpacked. Each vendor's rules unpack into their own sub-directory under /tmp. The $suricatadir variable does indeed point to /usr/local/etc/suricata where the original reference.config file that was installed with the binary portion of the package is located. So if you have both Snort and ET rules enabled, during a rules update three different reference.config files will be combined into a single reference.config with any duplicates removed. That file will be written to the interface sub-directory under /usr/local/etc/suricata/suricata_xxxx for each Suricata instance.