@bmeeks said in Short question about Suricata in cooperation with pfblocker:

@Trust9 said in Short question about Suricata in cooperation with pfblocker:

I have a lot of open ports.
Web server, Voip, and other services. Don't you think it makes sense to enable IPS on the wan interface as well?

No, I would not put Suricata on the WAN and here is why.

Nothing can reach your internal services (web server, VoIP box, etc.) without the traffic passing through either a LAN or DMZ interface on the firewall. pfSense is a router and a firewall. It takes incoming traffic on the WAN from the Internet, applies the firewall rules to the traffic, then routes the traffic to the appropriate internal interface.

You do not run an IDS/IPS to protect your firewall. If you have a firewall that needs an IDS/IPS in front of it for protection, then you need to find a new and more secure firewall 🙂. The firewall should be more than capable of protecting itself and should not need an IDS/IPS in front of it for protection.

In light of the above two points, what is the point of putting Suricata on the WAN? When running on the WAN, it will see and have to process all of the Internet "noise" on the WAN interface, but then your firewall rules are going to drop the vast majority of that traffic anyway. So, why waste CPU cycles having Suricata scan that stuff? Refer to the two diagrams I linked earlier. See how Suricata will always be the first thing traffic coming into an interface hits.

Suricata running on say a LAN or DMZ interface will still see and be able to scan all traffic traversing that interface (both inbound and outbound). That interface is the only way something from the Internet (via a port forward on the WAN, for example) can reach your internal host (a web server or VoIP box). Suricata sitting on the interface will see and police the traffic. But it will only have to deal with traffic that the firewall rules have already filtered thus saving CPU cycles.

In my view, the only time putting Suricata on the WAN makes any sense is if you have very limited RAM in the box, an anemic CPU, and you have several interfaces you want to protect with the same rules on all. Maybe then for overall conservation of RAM and CPU you compromise with a single WAN instance of Suricata. But at that point it may also be time to replace the firewall hardware with something with more capability.

Your explanations are really awesome. You really take a lot of time to share your knowledge. 🕶

Thank you very much. 👍
I will activate Suricata only on the lan interface and see how it goes.

@bmeeks I created a list that matches the current rule stub.

Attached here. It works with custom area.

Sorcerer's code file -->> textrules2.txt

Try updating to the latest pfSense version and the latest Suricata package. pfSense CE is now on version 2.7.0 and the Suricata package is at version 6.0.13. There have been several changes in Suricata since the version you have.

You can't update to the latest Suricata package without first updating pfSenses to 2.7.0.

Maybe (if this is still an issue for you) you want to join this discussion

The discussion is quite old but I've added my concerns today anyway.

Never does not work any longer it changed overnight to 5 mins for some reason, memory use I suspect.

Screenshot 2023-10-09 at 10.57.43 PM.png
Before

Screenshot 2023-10-09 at 10.58.08 PM.png
After

Here is a link to the generic pfSense documentation for the IDS/IPS packages (Snort and Suricata): https://docs.netgate.com/pfsense/en/latest/packages/snort/index.html.

Because those two packages share so much common GUI code, the way they operate is extraordinarily similar. That point is noted in the documentation linked above.

Just be aware that Suricata (and Snort) on pfSense runs a customized binary with a special output plugin compiled in for Legacy Mode Blocking. Also, the GUI in pfSense does everything "behind the scenes" that a user would normally do via command-line editing of configuration files on other Linux or FreeBSD distros. So, many of the online guides you might find for configuring Suricata have limited usefulness on pfSense (at least in terms of providing specific steps to achieve some particular configuration) because they refer you to direct file edits. Those don't work on pfSense because the GUI code rewrites all the local configuration files each time you save a change in the GUI or start the binary. Thus any hand-edits you may have made will be immediately lost.

At best these online generic Suricata guides can give you the overall concept, but then you need to find how some feature is implemented within the package GUI on pfSense. Posting specific questions back to this forum is a great way to get help and learn to use the package. There are quite a few Snort and Suricata users on pfSense. There are also some pinned Sticky Posts at the top of this sub-forum describing how to use certain features of both packages. Remember that anything you see posted for Snort operation likely applies about the same to Suricata. There are some differences, but the overall workflow of the GUI is the same in both IDS/IPS packages.

@bmeeks Verified that it's working perfectly on 4.1.6_11.

Thanks again!

@bmeeks

4.1.6_11 sorry I had a mix up.

I do not know if this has anything to do with the intermittent passlist block issue. I noticed this error shortly after the above screen shots. Thanks for all you do and also for sharing the code above.

Fatal error: Uncaught TypeError: fgetcsv(): Argument #1 ($stream) must be of type resource, bool given in /usr/local/www/snort/snort_alerts.php:858 Stack trace: #0 /usr/local/www/snort/snort_alerts.php(858): fgetcsv(false, 1000, ',', '"') #1 {main} thrown in /usr/local/www/snort/snort_alerts.php on line 858 PHP ERROR: Type: 1, File: /usr/local/www/snort/snort_alerts.php, Line: 858, Message: Uncaught TypeError: fgetcsv(): Argument #1 ($stream) must be of type resource, bool given in /usr/local/www/snort/snort_alerts.php:858 Stack trace: #0 /usr/local/www/snort/snort_alerts.php(858): fgetcsv(false, 1000, ',', '"') #1 {main} thrown

Screenshot 2023-10-06 at 5.49.45 PM.png

https://redmine.pfsense.org/issues/14850

@Bob-Dig said in [solved] Suricata resource consumption:

It looks like vtnet is supported and proxmox is using that. Will try now.

Had no luck with my weak VPS running pfSense in proxmox. I had random disconnects so I switched back to legacy mode.

@JonathanLee said in Snort LAN interface assignment:

So should I move it to wan side because of no access to inline mode?

No, not in my view. And Inline Mode or not Inline Mode has zero bearing on where you should run the IDS/IPS.

@JonathanLee said in Snort LAN interface assignment:

Do you know what official negate appliance supports inline mode?

Any of their non-Marvel switched ports appliances. Examples include SG-5100, SG-6100, SG-8200, and a few others. Look at the list of netmap compatible devices I posted earlier.

@michmoor @bmeeks

Here is, the fully converted appMapping.data to text file...

Screenshot 2023-10-04 at 5.58.46 PM.jpg

The pfSense Snort AppID de-cipher sorcerer's code file: --> textrules.txt

Sid range: 1000000 - 1003371

Total 3,371 AppID rules you can use with the custom option.

I converted it with a Java program I just made. The message is the same as the appid match it makes it easier.

Some of the ieee items are bigger but they seem to match.

@bmeeks said in Speed drops with snort in Inline Mode:

@Amodin said in Speed drops with snort in Inline Mode:

Snort is a multi-threaded application as of Snort3. Is that not running in the package?

No, the Snort package on pfSense is based on the 2.9.x binary which is single threaded only. There are currently no plans to implement Snort3 on pfSense.

Oof, good to know. I might have to try out some Suricata, as I was just reading about the differences in another thread you had posted about them.

Hello everyone,

I have just reinstalled PFSENSE again.

All the times I installed PFSENSE I used the UFS mode, I just redid an installation with the ZFS mode and I no longer have the problem ...

I'll keep my fingers crossed and continue my tests.

Thank you all for your participation.

I'll check out the difference between these 2 partitioning modes later.

Had a few in-between other activities and came up with this. RE: bonus on ideas - "..things that make you go Hmmmmm....." lol

Including if useful to anyone else - this appears to solve the riddle..... Not "pretty" but addresses the need with a few options that can be applied to satisfy some conditions. Could probably be augmented to enable specification of "interface" to further simplify.

Spoiler

#!/usr/local/bin/perl # use strict; use Getopt::Long; $| = 1; # GetOptions('debug'=>\$PROC::DEBUG,'include=s'=>\$PROC::INCLUDE,'severity=s'=>\$PROC::SEVERITY,'targetfile=s'=>\$PROC::TGTFILE,'mergefile=s'=>\$PROC::MRGFILE,); if (defined($PROC::DEBUG)) { $PROC::DEBUG=1; } else { $PROC::DEBUG=0; } %PROC::INCS=(); if (defined($PROC::INCLUDE)) { foreach(split(/,/,$PROC::INCLUDE)) { $PROC::INCS{$_}=0; } } # @PROC::DIRS=('/usr/local/share/suricata/rules','/usr/local/etc/snort/rules',); %PROC::HASH=(); # foreach(@PROC::DIRS) { my $RDIR=$_; opendir(DIR, "$RDIR"); rewinddir(DIR); while(my $FILE=readdir(DIR)) { if ($FILE=~/\.rules$/) { my $CHECK=$FILE; $CHECK=~s/\.rules$//; if ((keys(%PROC::INCS)>0) && (exists($PROC::INCS{$CHECK}))) { &procFile("$RDIR/$FILE"); } elsif (keys(%PROC::INCS)==0) { &procFile("$RDIR/$FILE"); } } } closedir(DIR); } # if (defined($PROC::MRGFILE)) { open(INF, "<$PROC::MRGFILE"); while(my $LINE=<INF>) { chomp($LINE); if (($LINE!~/^#/) && ($LINE!~/^[[:space:]]{0,}$/) && ($LINE=~/^suppress[[:space:]]{1,}/)) { my $GID=$LINE; $GID=~s/^.*gen_id[[:space:]]{1,}//; $GID=~s/,.*//; my $SID=$LINE; $SID=~s/^.*.sig_id[[:space:]]{1,}//; if ($SID=~/,/) { $SID=~s/,.*//; } if (exists($PROC::HASH{$GID}{$SID})) { delete($PROC::HASH{$GID}{$SID}); } } } close(INF); } my $FH; if (defined($PROC::TGTFILE)) { open $FH, ">", "$PROC::TGTFILE" || die("ERROR: $PROC::TGTFILE $!\n"); select($FH); } elsif (defined($PROC::MRGFILE)) { open $FH, ">>", "$PROC::MRGFILE" || die("ERROR: $PROC::MRGFILE $!\n"); select($FH); print $FH ("\n"); } foreach my $ID (keys %PROC::HASH) { foreach my $SID (sort {$a<=>$b} keys %{$PROC::HASH{$ID}}) { my $MSG=$PROC::HASH{$ID}{$SID}{msg}; my $FILE=$PROC::HASH{$ID}{$SID}{file}; print ("# ($FILE) $MSG\nsuppress gen_id $ID, sig_id $SID\n\n"); } } if (defined($PROC::TGTFILE) || defined($PROC::MRGFILE)) { close $FH; } # if (defined($PROC::MRGFILE)) { my $F = do { local $/ = undef; open my $FH, "<", "$PROC::MRGFILE"; <$FH>; }; $F=~s/\n//g; $F=~s/#/\n\n#/g; $F=~s/suppress[[:space:]]{1,}/\nsuppress /g; $F=~s/^\n{1,}//; open(OUF, ">$PROC::MRGFILE"); print OUF ("$F\n"); close(OUF); } # sub procFile { my ($FILE)=(shift); if ($PROC::DEBUG==1) { print ("\tFILE : $FILE\n"); } open(INF, "<$FILE"); while(my $LINE=<INF>) { chomp($LINE); if ($LINE=~/^alert ip \[/) { my $SID=$LINE; $SID=~s/^.*.sid://; $SID=~s/;.*//; my $MSG=$LINE; $MSG=~s/^.*.msg://; $MSG=~s/;.*//; $MSG=~s/"//g; my $SEV=$LINE; $SEV=~s/^.*.signature_severity //; $SEV=~s/,.*//; my $F=$FILE; $F=~s/\.rules$//; $F=~s/^.*.\///; if ($PROC::DEBUG==1) { print ("\t\t1 : $SID : $F : $SEV\n"); } if (defined($PROC::SEVERITY) && ($SEV eq $PROC::SEVERITY)) { $PROC::HASH{1}{$SID}{msg}=$MSG; $PROC::HASH{1}{$SID}{file}=$F; } elsif (!defined($PROC::SEVERITY)) { $PROC::HASH{1}{$SID}{msg}=$MSG; $PROC::HASH{1}{$SID}{file}=$F; } } } close(INF); return; } # __END__ ## ## Documentation ## =head1 NAME generate-suppress.pl =head1 SYNOPSIS generate-suppress.pl --debug --include=<include> --severity=<severity> --targetfile=<targetfile> --mergefile=<mergefile> =head1 DESCRIPTION Generates suppression data from source rules for "alert ip" style entries =head1 FUNCTION Insert or merge alert ip suppression data for SNORT/Suricata =head1 OPTIONS =over =item <debug> Enables debugging output. =item <include> Specifies which rule(s) to include in the resultant data. Comma separated - B<NO> spaces. =item <severity> Filter resultant rules to only those that match severity. (As of creation, appears to be either "Major" or "Minor") =item <targetfile> If the target file exists, it B<WILL> be overwritten with the results. =item <mergefile> If the mergefile already exists, it will be read and only [missing] deltas will be added. =item NOTE If neither targetfile nor mergefile are specified, the results are printed to STDOUT. =back =head1 COMMON USAGE perl generate-suppress.pl --mergefile=/usr/local/etc/suricata/suricata_<ID>_<interface>/threshold.config =cut

@NollipfSense
hi, that would be a way, or like we mentioned, put it in another VLAN. Well, VLAN does not work as well as DMZ like you suggested, but that was the idea if going that route. However, we are trying to assess if the app that is running on the particular host can be safely released to the "general population". Or what will be the minimum suppression rules we can get away with in order for it to run properly? Or what kind of compromise we need to take in order to run the app on this host?
The exercise will give us an idea on whether a new network/interface, like a DMZ, setup will be absolutely required.
Along the way, I also learnt quite a bit on how Snort works and what are good practices.
Cheers and thanks for the suggestion.
W

@andersondeda said in Wan Down:

I confess that I have difficulty finding supporting documentation, I feel the need to know the purpose of all these options.

I agree that the documentation for many of Suricata's internal settings is a bit opaque at times. But the upstream team does have a goal of improving the documentation. The official docs site has a sub-section for each Suricata version. Here is the link to the version for 6.0.13 (which is current in pfSense 2.7.0 right now): https://docs.suricata.io/en/suricata-6.0.13/.

Some details about the Stream Engine settings can be found here: https://docs.suricata.io/en/suricata-6.0.13/configuration/suricata-yaml.html#stream-engine. A quick and dirty explanation is TCP Stream Memory is used to buffer packets associated with a specific TCP flow and reassemble them into a contiguous block that the signature comparision engine can process to look for matches to rules. Because multiple CPU cores means multiple threads with each thread needing its own separate TCP stream reassembly area, the amount of needed RAM increases quickly as the CPU core count increases.

@bmeeks How did I miss that? Thanks very much.

You can't. Snort sits between the physical NIC and the kernel network stack before the firewall engine. So, when you run Snort (or Suricata for that matter) on the WAN, it only sees local traffic after NAT has been applied for outbound traffic, and before NAT is undone for inbound traffic. Here are two diagrams that show how the IDS/IPS packages are plumbed into the network. This is an operating system thing and not anything the packages can alter.

ids-ips-network-flow-legacy-mode.png

ids-ips-network-flow-ips-mode.png

This is why I have been recommending for the last few years that users put the IDS/IPS on internal interfaces. You should do the same. There is no point in having it on the WAN. IDS/IPS is not for protecting the firewall. It's for protecting the hosts behind the firewall. If you need IDS/IPS for your firewall itself, then you really need a new firewall 😀.

Running it on the LAN would eliminate your issue of NAT hiding local IP addresses. When running on the LAN, all traffic going to or coming from local hosts would have to pass through the IDS/IPS.

And one last note. Without MITM breaking of encryption, IDS/IPS on the firewall is severely limited in what it can accomplish these days because nearly 100% of network traffic is encrypted. The IDS/IPS can't peer into any of the payloads for SSL traffic. That means zero payload inspection of HTTPS, DoT, DoH, SMTPS, IMAPS, and POP3S for starters. That's nearly all of the web traffic, potentially all of the DNS traffic (if you use DoT), and pretty much all email traffic bypassing inspection. Intrusion Detection is rapidly becoming something best done on the local destination host itself and not on intermediate network devices.

Yup that^. Keep it civil please.

I will add that if you're running Snort on WAN you're going to see a lot of hits. Generally unless you are forwarding traffic to internal services they are not really useful.

Steve