Taming the beasts… aka suricata blueprint
-
Aside from extremely enjoying your funny writing style ( ;D ) I also think you can't get too much karma for all that you are doing with regards to helping people set up Suricata (and before that: Snort).
Thank you, secret man :P
-
I've read your post 3 times and I'm having a difficult time understanding the floating rule. The default nature of the firewall is to block incoming traffic unless you add a pass rule. As I understand it, floating rules are evaluated first. So wouldn't this rule always block incoming packets on the interface regardless of the interface rules?
The giant red warning under that rule should explain it. It's a rule that will ONLY apply to traffic destined for pfsense's ports. By default pfsense could open up the webgui to an undesired interface, which will not be covered by the default rule. Depending on how far away you sit from the fan, it leads to varying amounts of brown stuff raining down when "it" hits the fan. ;)
@Hollander: Finished today analyzing logs for 3,997,696 IP addresses. Those (almost) 4mil IPs were what tripped up our security systems in the first 6 months of this year. Needless to say they gained a magical place in my "Permanently Banned" Hall of Shame.
If pfsense/suricata/other logs can help identify 4mil malicious IPs, then sure as hell they deserve all the support we can give them.
@all: List has had a couple of updates don't forget to check it regularly. I'm trying to add descriptions when I edit the list, so it's obvious what I added/removed/changed, without needing to go through the entire list.
-
@jflsakfja:
I've read your post 3 times and I'm having a difficult time understanding the floating rule. The default nature of the firewall is to block incoming traffic unless you add a pass rule. As I understand it, floating rules are evaluated first. So wouldn't this rule always block incoming packets on the interface regardless of the interface rules?
The giant red warning under that rule should explain it. It's a rule that will ONLY apply to traffic destined for pfsense's ports. By default pfsense could open up the webgui to an undesired interface, which will not be covered by the default rule. Depending on how far away you sit from the fan, it leads to varying amounts of brown stuff raining down when "it" hits the fan. ;)
@Hollander: Finished today analyzing logs for 3,997,696 IP addresses. Those (almost) 4mil IPs were what tripped up our security systems in the first 6 months of this year. Needless to say they gained a magical place in my "Permanently Banned" Hall of Shame.
If pfsense/suricata/other logs can help identify 4mil malicious IPs, then sure as hell they deserve all the support we can give them.
@all: List has had a couple of updates don't forget to check it regularly. I'm trying to add descriptions when I edit the list, so it's obvious what I added/removed/changed, without needing to go through the entire list.
jflsakfja,
Thank you for the clarification! Makes sense now.
-
An intermezzo question: did anybody try to print this thread? I wanted to start working on this, and print it to study it thoroughly first. The printing leads to iny tiny small text on the paper, not readable. I tried this from three computers, 3 browsers, all the same.
Is this a forum software thing? Would an admin perhaps mind to verify?
(text in red so admin notices it)
Thank you ;D
-
Never tried printing anything from around here, but I've been getting weird errors when posting replies. It's time to abandon the clusterf*** that is the current forum software*. I believe it will also be the solution to the black hole creation problem as well, and who knows, maybe one day we too can edit our old posts. One can only hope.
Notes:
- This is my personal opinion and I'm allowed to say it based on provisions in my country's constitution, as well as international human rights treaties.
Disclaimer:
If you are in any way related to the current clusterf*** forum software, then you should not be offended by a single person's opinion of it. If it is the majority's opinion of it though, that means that the current forum software is indeed a clusterf***, and in that case you should seriously consider abandoning the project and letting it die the slow and horrible death it deserves. -
@jflsakfja:
Never tried printing anything from around here, but I've been getting weird errors when posting replies. It's time to abandon the clusterf*** that is the current forum software*. I believe it will also be the solution to the black hole creation problem as well, and who knows, maybe one day we too can edit our old posts. One can only hope.
Notes:
- This is my personal opinion and I'm allowed to say it based on provisions in my country's constitution, as well as international human rights treaties.
Disclaimer:
If you are in any way related to the current clusterf*** forum software, then you should not be offended by a single person's opinion of it. If it is the majority's opinion of it though, that means that the current forum software is indeed a clusterf***, and in that case you should seriously consider abandoning the project and letting it die the slow and horrible death it deserves.;D ;D ;D
(Love your funny writing style :P ).
Your royalness, I am currently clusterf*cking around on my box with your tuto. For one, the floating rule blocks everything out, so I had to disable that. I've noticed in a follow up post that you wrote the floating rule was meant to prevent access to the pfSense GUI, but I think that is not what your initial instruction does (but again, keep in mind I will be the eternal noob).
A more serious question for me is, while I am now currently looking at the script: which are the lines I need to comment out to select lists? I am looking but honestly have no clue :-[ Could you give an example of a line that contains a list? Is it the ones all the way at the bottom?
Thank you ;D
-
Sorry about that floating rule, I later added the giant red warning. The floating rule should only apply to the pfsense's ports, and that shouldn't block any other traffic.
As far as the lists go, it's near the end of the script. There are instructions in the script to enable/disable the lists. Enabling a list is usually removing the # in front of the line containing the list.
-
:'( :-[ ???
( >:( )
How on earth can this be possible? I even logged out and logged in again. Where on earth does it get this directory from?
It is not difficult to remain the eternal noob when this happens ( ;D >:( )
[b]EDIT: it appears it hadn't saved the first: userfolder=/home/badips
Probably because I had the file open in both WinSCP and via the Diagnostics/edit file. I saved it in the latter, but apparently since it was also open in WinSCP it didn't tell me it couldn't write but simply said nothing.
-
@jflsakfja:
Sorry about that floating rule, I later added the giant red warning. The floating rule should only apply to the pfsense's ports, and that shouldn't block any other traffic.
As far as the lists go, it's near the end of the script. There are instructions in the script to enable/disable the lists. Enabling a list is usually removing the # in front of the line containing the list.
Thanks Jflsakfja ;D
Next up Floating tab:
Set up a rule but make these changes:| Action | Block |
| Quick | TICKED!!! |
| Interface | Hold CTRL and click on all interfaces EXCEPT LAN(admin) and SYNC |
| Direction | any |
| Source | any |
| Destination | any |DON'T CHANGE DESTINATION PORT RANGE!!!
So when I add new floating rule, the above reads to block any source, any direction, to any destination (I left the ports to 'other', which is the default when I created a new floating rule, so I didn't change it per the red text), effectively blocking all LAN out (I think, at least when I disabled the rule I had internet access again).
Remember: eternal noobs will be eternal noobs ;D
-
Re-reading it does make sense on why it blocked out traffic. I meant to say create a new floating rule, based on the previous allow rule, but this time around change the pass to a block, keeping the destination ports the same.
A)1 normal pass rule for the ports active on the interface you want to administer pfsense from.
B)1 floating rule block rule for ALL interfaces EXCEPT the one you want to administer pfsense from.
Both rules should have their destination as the alias for pfsense's ports. The allow rule well, obviously allows traffic to those ports on your admin (LAN?) interface, but the floating rule should block all traffic for those ports, on each and every >other< interface.
I don't have access to a pfsense system since I'm out of town for the weekend, one can only post so much from memory :p.
-
I have been working on a script
As I am working my way down this thread on the instructions I arrived at your script: thank you very much for creating it ;D
I only understand 10% of what it does (given my eternal noob status), but I do know that this is quite some work. My hero-list on this board keeps on getting bigger, I just added you to it as well ;D
Thank you & bye,
-
@jflsakfja:
Re-reading it does make sense on why it blocked out traffic. I meant to say create a new floating rule, based on the previous allow rule, but this time around change the pass to a block, keeping the destination ports the same.
A)1 normal pass rule for the ports active on the interface you want to administer pfsense from.
B)1 floating rule block rule for ALL interfaces EXCEPT the one you want to administer pfsense from.
Both rules should have their destination as the alias for pfsense's ports. The allow rule well, obviously allows traffic to those ports on your admin (LAN?) interface, but the floating rule should block all traffic for those ports, on each and every >other< interface.
I don't have access to a pfsense system since I'm out of town for the weekend, one can only post so much from memory :p.
Thank you Jfl :P
-
I came across this site "infragard" https://www.infragard.org/node
InfraGard is a partnership between the FBI and the private sector.
It is an association of persons who represent businesses, academic institutions,
state and local law enforcement agencies, and other participants dedicated to
sharing information and intelligence to prevent hostile acts against the U.S.Unfortunately, you need to give them your first born to gain access to their Files.
However, I have come across their most recent data, which can be viewed with these links:
https://publicintelligence.net/fbi-cyber-targeting-gov-networks/
https://publicintelligence.net/siac-cryptowall/
https://publicintelligence.net/fbi-blackshades-bulletins/
http://www.eventtracker.com/support/knowledge-update-et75asig-001/As these are static Blocklists, I have added the option in the pfIP Rep Script to download
these files once only. (Setting the schedule to "$sch0), should the download fail, you can set
the schedule to "$sch1" and run it and set it back to "$sch0" after it completes.Two of the links are for domain blocking, this info could be used for Squid or dns sinkholes.
I have also updated the pfIP_Reputation.widget.php to include a "Ack" Acknowledge button to clear any previous "FAIL" Downloads. This will just edit the Daily.log from "FAIL" to "Fail", so you can still review the Daily.log for trending issues with downloading.
Here is a screenshot of the widget
And the link to my GIST for the pfIP_Reputation2.widget.php
https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfip_reputation2-widget-phpI Have updated the pf IP Reputation Manager Script to version 2.3.4
You can review the revisions in my GIST.
https://gist.github.com/BBcan17/67e8c456cb399fbe02ee
For pfiprep make the changes to your existing file or just overwrite and add your changes as required.
For pfiprepman, just backup the previous 2.3.2/3 version and replace with the latest 2.3.4 version.
Changes to pfiprep
Added the "FBI Suspicious Conus and Oconus Blocklists"
Added the "FBI Facebook FBUID Blocklist"
Added the "Suricata TOR Blocklist to the TOR Section"
INFO - OpenBL supports other Blocklist options that can be set.Changes to pfiprepman
The Script also now supports extracting IP Blocklists from .XLSX files.CountryCode Blocklists
With Cinos help, we have made some code improvements.
Added a "perl script - IPCALC" to convert the Ranges to CIDR
Found some other code changesI recommend running
[ [b]./pfiprep killdb ] with version changes or [ [b] ./pfiprep killdb dskip ]
If you find any Bugs please let me know and I will promptly fix them.
-
@Hollander:
I have been working on a script
As I am working my way down this thread on the instructions I arrived at your script: thank you very much for creating it ;D
I only understand 10% of what it does (given my eternal noob status), but I do know that this is quite some work. My hero-list on this board keeps on getting bigger, I just added you to it as well ;D
Thank you & bye,
This is what Open Source is all about. We've all caught the bug and that's why we enjoy spending time helping each other to advance of Network Security.
In regards to your comments, Thanks, its was lots of work but the best part is when people actually use it. If you have any questions, let us know or send me a PM when you need more help. :o :o
I would recommend leaving most of the settings as is, and then change things after you get it working. I would use the default Group/Tiers instead of adding all of the individual Blocklist aliases.
If anything, this is good practice to learn how to use the shell and other parts of FreeBSD that you never knew existed or maybe never wanted to know ! 8)
-
@jflsakfja:
EDIT!!!! MISSED THE QUICK CHECKBOX. TICK THAT!
Now I am confused. In your initial post on creating floating rules using the aliases you indicate NOT to tick Quick.
So just to be sure: To Tick Or Not To Tick? ;)
And does this go for both inbound and outbound?(BTW: thanks for all the time you put into this thread)
-
Tick the quick check box. This tells pfsense to process the rule right away. If its not ticked and let's say you have inbound port 80 opened,that rule would let traffic pass bypassing your floating block/reject rule
-
To explain the floating rules we need to examine the following examples:
A) 1 non-quick floating rule that blocks traffic to port 80 on interface wan
B) 1 normal (interface) rule that passes traffic to port 80 on interface wan- Floating rules get evaluated first from top to bottom
- If the floating rule is NOT quick, then proceed with further matching against other rules
- Interface rules get evaluated last from top to bottom
Going through the checklist above, tells us that a packet will first pass through the floating rule, where the rule does match, but is not the terminal match against it, so the packet will continue on until it reaches the terminal match, which is rule B. Although the floating rule says block, the packet will pass since that's the rule that matches it.
In the case that rule A is quick, the packet will immediately match that rule, which says block it, therefore it will be blocked.
-
We got it working ;D
Half of my normal sites are blocked now :P
( ;D )
(For example: www.geenstijl.nl, the links to the movies in there, the pfSense firewall says the new floating rules block them).
We = WIFE + me. WIFE, since I suffer from brain damage due to an accident, after which I can't concentrate on things suddenly. Combine that with the eternal noob status, and you will understand why WIFE, the love of my life, sometimes has to step in to help me out.
(The brain is a miraculous thing, a thing stupid economists like me will never understand. Because: not only do I have some, what some people might consider, 'advanced' post-academical degrees, but all that knowledge stays available to me. It is just that new knowledge doesn't seem to be allowed in. Perhaps I have a pfSense firewall in my brain now. It would make sense, because then nothing gets through ;D ;D ).
I forgot: Cino, thank you very much for your valuable contributions also: the list is getting longer ;D
Thank you & bye,
-
I've been setting up multiple pfsenses based on my recommendations in this topic and never had a problem with them blocking legitimate sites (aside from FPs now and then). In particular, a CARP cluster that provides connectivity to a small datacenter that houses web hosting servers, email servers and co-located servers, and internet access to remote clients, rarely gets any false positives. Those false positives are dealt with as soon as possible, which can be seen on the github list, since I took the enable all rules, then start removing rules as they are encountered approach, as recommended in this topic.
There is a reason those sites are blocked. Maybe it's a misconfiguration issue, a suricata false positive, or maybe they are not so legitimate after all.
-
@jflsakfja:
To explain the floating rules we need to examine the following examples:
A) 1 non-quick floating rule that blocks traffic to port 80 on interface wan
B) 1 normal (interface) rule that passes traffic to port 80 on interface wan- Floating rules get evaluated first from top to bottom
- If the floating rule is NOT quick, then proceed with further matching against other rules
- Interface rules get evaluated last from top to bottom
Going through the checklist above, tells us that a packet will first pass through the floating rule, where the rule does match, but is not the terminal match against it, so the packet will continue on until it reaches the terminal match, which is rule B. Although the floating rule says block, the packet will pass since that's the rule that matches it.
In the case that rule A is quick, the packet will immediately match that rule, which says block it, therefore it will be blocked.
One of my ever frustrating frustrations (yaah :P ) with the floating rules was that I never understood them. I think your explanation is very clear, so for the zillionth time: thank you, Jfl ;D
What remains is one little question: the source versus destination. You will have to set the rules twice, yes (?) because of the source versus destination difference: suppose you have multiple WAN and multiple LAN (as I do), in order to block connections from and to a specific IP you will have to set two floating rules: 1 for all WAN's where source is the aforementioned IP, and a second floating rule for all LAN's where the destination is the aforementioned IP.
I ask because in many posts (in this forum) as well as 'tutorials' on the internets it is said you only need one floating rule, and I never understood that since it depends on whether traffic is coming from the source or from the destination.
Brains: mysterious things ;D
-
@jflsakfja:
I've been setting up multiple pfsenses based on my recommendations in this topic and never had a problem with them blocking legitimate sites (aside from FPs now and then). In particular, a CARP cluster that provides connectivity to a small datacenter that houses web hosting servers, email servers and co-located servers, and internet access to remote clients, rarely gets any false positives. Those false positives are dealt with as soon as possible, which can be seen on the github list, since I took the enable all rules, then start removing rules as they are encountered approach, as recommended in this topic.
There is a reason those sites are blocked. Maybe it's a misconfiguration issue, a suricata false positive, or maybe they are not so legitimate after all.
I trust your judgements and knowledge completely, Jfl, you know that; more researching for me to do ;D
-
ping www.geenstijl.nl
PING www.geenstijl.nl (162.159.255.153): 56 data bytes
(Look at the Original Download Files)
grep "162.159.255.153" /home/USER/orig/*/home/USER/orig/ET_IPrep.txt:162.159.255.81,15,127
/home/USER/orig/ET_IPrep.txt:162.159.255.81,24,103
/home/USER/orig/ET_IPrep.txt:162.159.255.219,27,55
/home/USER/orig/ET_IPrep.txt:162.159.255.5,27,109(Look at the final pf Folder files)
grep "^162.159.255." /home/USER/pf/*/home/USER/pf/ET_IPrep.txt:162.159.255.81
/home/USER/pf/e_tfakeav:162.159.255.81(Look at the pfSense AliasTable Folder)
grep "^162.159.255." /usr/local/www/aliastables/*/usr/local/www/aliastables/IR_PRI1:162.159.255.81
I am using the Emerging Threats IQRisk Blocklist, so that is the only list that I see that has any IPs in that Range.
I don't believe you are using that list, so not sure which list had that IP. And there is ony 4 IPs listed which is below the threshold of "5" and so it didn't block the whole range.
Atleast you can see that this range has some malicious activity (FakeAv)
-
Yes, that's why I recommended setting up pairs for floating rules for the list's aliases.
A source rule says the source should be matched. Since it's "them" that send packets to "us" on our wan type interfaces, we should set up the wan side rules to perform their match against the source (list alias). These rules should be block (don't answer the door saying I will not talk to you).
A destination rule says the destination should be matched. Since it's "us" that send packets to "them" on our lan type interfaces, we should set up the lan side (or dmz, or any other internal interface) rules to match based on the destination. Matching against the source will never give a match, since the source is "us". These rules should be reject (answer the door to our internal client saying "You are not allowed to talk to that") so that browsing to a non-legitimate site doesn't take 2 minutes to time out.
-
ping www.geenstijl.nl
PING www.geenstijl.nl (162.159.255.153): 56 data bytes
(Look at the Original Download Files)
grep "162.159.255.153" /home/USER/orig/*/home/USER/orig/ET_IPrep.txt:162.159.255.81,15,127
/home/USER/orig/ET_IPrep.txt:162.159.255.81,24,103
/home/USER/orig/ET_IPrep.txt:162.159.255.219,27,55
/home/USER/orig/ET_IPrep.txt:162.159.255.5,27,109(Look at the final pf Folder files)
grep "^162.159.255." /home/USER/pf/*/home/USER/pf/ET_IPrep.txt:162.159.255.81
/home/USER/pf/e_tfakeav:162.159.255.81(Look at the pfSense AliasTable Folder)
grep "^162.159.255." /usr/local/www/aliastables/*/usr/local/www/aliastables/IR_PRI1:162.159.255.81
I am using the Emerging Threats IQRisk Blocklist, so that is the only list that I see that has any IPs in that Range.
I don't believe you are using that list, so not sure which list had that IP. And there is ony 4 IPs listed which is below the threshold of "5" and so it didn't block the whole range.
Atleast you can see that this range has some malicious activity (FakeAv)
IP 162.159.255.153 belongs to cloudflare. Since cloudflare is a CDN (Content Distribution Network) some other site might have used that IP to perform some nefarious act, therefore the IP got listed on one of the lists. Ideally cloudflare will deal with it and the IP will eventually be removed from the list. My only explanation as to why traffic to that IP is blocked.
-
F*** cloudflare… I have been blocking SPAM from their Network for two weeks now. Everyday its getting past all of the Blocklists, Snort, and my Mail Servers Mail Spam protection. FRESH Spam I guess. I send all of the mail that makes it past my Security Systems to SPAMCOP and hopefully they add it to the list of IPs to Ban.
-
Re-read my post, "ideally" is the key word ;)
Dealing with abuse (or spam) reports is an activity I don't look forward to. In particular my country's ISPs seem to think they invented the Internet, and all (without ANY exception) do NOT respond to abuse reports merely thinking out loud "Who the F**** are you to tell us our job!?!?!". Hell I've had to deal with a 15month DoS attack, until I had to scream at my upstream's "Head of Technical Department" (a complete and utter idiot, of the variety that no matter how many you seem to get rid of, a thousand more pop up for each one) to add the single IP to their ACL (a NON existent ACL until I forced them to implement it).
The attacker's upstream's response to my constant abuse reports was drum-roll…. they changed his IP. I got a brain aneurysm when they told me that. I thought my upstream's idiotic technical department was the lowest you could go, but apparently there are "innovators" all over the place.
Don't want to name and shame anyone, since there are "innovators" all over the place, even in the justice system, that would think those comments were "libelous" ;)
A small tip to ISPs. If I bother to write an official abuse report to you, citing logs, then that means I'm fed up with your client/system. Do something about it, or consider a career change and let us, that know how to do your job, do it.
-
Thanks Cino and jflsakfja, for clearing up why I should tick the Quick box.
If it were not for the risk of black hole and total collapse of the forum I might have suggested here to edit the original instructions in the #10 reply post accordingly ;) ;) ;D -
ping www.geenstijl.nl
PING www.geenstijl.nl (162.159.255.153): 56 data bytes
(Look at the Original Download Files)
grep "162.159.255.153" /home/USER/orig/*/home/USER/orig/ET_IPrep.txt:162.159.255.81,15,127
/home/USER/orig/ET_IPrep.txt:162.159.255.81,24,103
/home/USER/orig/ET_IPrep.txt:162.159.255.219,27,55
/home/USER/orig/ET_IPrep.txt:162.159.255.5,27,109(Look at the final pf Folder files)
grep "^162.159.255." /home/USER/pf/*/home/USER/pf/ET_IPrep.txt:162.159.255.81
/home/USER/pf/e_tfakeav:162.159.255.81(Look at the pfSense AliasTable Folder)
grep "^162.159.255." /usr/local/www/aliastables/*/usr/local/www/aliastables/IR_PRI1:162.159.255.81
I am using the Emerging Threats IQRisk Blocklist, so that is the only list that I see that has any IPs in that Range.
I don't believe you are using that list, so not sure which list had that IP. And there is ony 4 IPs listed which is below the threshold of "5" and so it didn't block the whole range.
Atleast you can see that this range has some malicious activity (FakeAv)
Amazing to see what you can do @ the CLI, BB ;D
I wasn't clear enough, since I could hit geenstijl.nl, but not the movies they embed there. They are hosted on one of their other sites, dumpert.nl. And that one was blocked. The IP lookup turned out this was cloudflare. So I added a floating rule IR_PASS (which, magically, also appears in the dashboard widget :P ) that will use the IR_PASS alias to contain hosts that I might consider 'false positives'.
Just curious, btw, BB: why is cloudflare spamming you (or, better said: how?) I thought this was a reputable service? Obviously not, but I don't know why? I mean, large sites use cloudflare(?)
-
Could I ask some of my most famous noob questions?
1. Does it make sense that if IR_PRI1 is blocking that Snort is also still blocking IP's based on Dshield (which is in IR_PRI1)?
2. While I will be moving over from Snort to Suricata after Bmeeks will have added new buttons for easy disabling rules, does it still make sense to buy the Snort VRT subscription for use in Suricata? Or won't these Snort rules work too well in Suricata?
3. How do you get rid of all these log lines about traffic being blocked:–- 255.255.255.255:10001 UDP
--- 239.255.255.250:1900 UDP
--- 224.0.0.252:5355 UDPThey were gone for months, but suddenly the log is being flooded with them again. Ever since I started with pfSense these lines appear to bug me every so many months.
Thank you ;D
-
@jflsakfja:
Yes, that's why I recommended setting up pairs for floating rules for the list's aliases.
A source rule says the source should be matched. Since it's "them" that send packets to "us" on our wan type interfaces, we should set up the wan side rules to perform their match against the source (list alias). These rules should be block (don't answer the door saying I will not talk to you).
A destination rule says the destination should be matched. Since it's "us" that send packets to "them" on our lan type interfaces, we should set up the lan side (or dmz, or any other internal interface) rules to match based on the destination. Matching against the source will never give a match, since the source is "us". These rules should be reject (answer the door to our internal client saying "You are not allowed to talk to that") so that browsing to a non-legitimate site doesn't take 2 minutes to time out.
Thank you Sir ;D
Whilst in the traffic jam I was thinking: what actually is the goal of floating rules as opposed to rules on Interface Groups? Both do the same as far as I can tell: they both work over different interfaces, they both come before the individual rules per interface. I am sort of wondering when you should use what ???
-
@Hollander:
1. Does it make sense that if IR_PRI1 is blocking that Snort is also still blocking IP's based on Dshield (which is in IR_PRI1)?
2. While I will be moving over from Snort to Suricata after Bmeeks will have added new buttons for easy disabling rules, does it still make sense to buy the Snort VRT subscription for use in Suricata? Or won't these Snort rules work too well in Suricata?
3. How do you get rid of all these log lines about traffic being blocked:1. Good idea on putting the Pass list into the IR_ category! Snort/Suricata processses a 'copy' of all the packets. So even if it was blocked by the Firewall, Snort/Suricata will still see it. Keep an eye on the Snort/Suricata alerts, and click on the "!" to see if the IP is listed or has the range blocked. I have noticed that over 90% of what Snort/Suricata Blocks is already being blocked by the Firewall Blocklists.
Hopefully when pfSense moves to 2.2 and they add NetMap api, it will allow a true-inline process for Snort/Suricata.
2. Suricata will load the Snort VRT ruleset except for about 600 rules. You can see that in the Suricata logs (which ones failed due to regex issues) I would still use the ruleset in Suricata.
(on another note, I still plan on staying with Snort for the time being)3. I get them occasionally. Maybe Bill has some suggestions.
-
@Hollander:
I wasn't clear enough, since I could hit geenstijl.nl, but not the movies they embed there. They are hosted on one of their other sites, dumpert.nl. And that one was blocked. The IP lookup turned out this was cloudflare. So I added a floating rule IR_PASS (which, magically, also appears in the dashboard widget :P ) that will use the IR_PASS alias to contain hosts that I might consider 'false positives'.
Just curious, btw, BB: why is cloudflare spamming you (or, better said: how?) I thought this was a reputable service? Obviously not, but I don't know why? I mean, large sites use cloudflare(?)
Here is what I show being blocked by that IP (IP may be different for you as cloudflare is a CDN and the IP could be different in your Country.
From these results, really makes you wonder if you would feel comfortable going to those websites? Note, the IPs below are from the ET Paid IQRisk Blocklist. No other lists show that Ip for me, so I am curious which list is blocking on your end?
grep "^141.101.116." pf/*
pf/ET_IPrep.txt:141.101.116.0/24
pf/e_tcnc:141.101.116.72
pf/e_tcnc:141.101.116.52
pf/e_tcnc:141.101.116.55
pf/e_tcnc:141.101.116.107
pf/e_tcnc:141.101.116.126
pf/e_tcnc:141.101.116.162
pf/e_tcnc:141.101.116.15
pf/e_tcnc:141.101.116.187
pf/e_tcnc:141.101.116.137
pf/e_tcnc:141.101.116.175
pf/e_tcompromised:141.101.116.174
pf/e_tddos:141.101.116.176
pf/e_tspywarecnc:141.101.116.55
pf/e_tspywarecnc:141.101.116.154
pf/e_tspywarecnc:141.101.116.95
pf/e_tspywarecnc:141.101.116.170
pf/e_tspywarecnc:141.101.116.112
pf/e_tspywarecnc:141.101.116.130
pf/e_tspywarecnc:141.101.116.17As an Economist, you should already know that answer ;) Money, Bones, Bread, Fins, Moola, Dinero
Why would they kicks spammers off of their network and lose revenue. Spread the word and block/boycott them all.
What happens is a Spammer sets up shop on a Hosting Service and Spams for as long as they can, after they get blocked, they move to another Hosting Service or just setup a new domain and new IPs. (Sadly, those IPs get recycled to some legit sites and people wonder why they are on a Blocklist when they just got these new IP addresses.)
I have had spamming issues from several Hosting Services HOSTNOC, OVH, Burstnet.
I am currently blocking the whole Burstnet network (3000 servers) as I was getting spammed for months without them doing anything about it. If you want to see how bad it is, take a look at this Google Group:
https://groups.google.com/forum/#!forum/news.admin.net-abuse.email
(and for a good laugh… ;D) -
@Hollander:
Whilst in the traffic jam I was thinking: what actually is the goal of floating rules as opposed to rules on Interface Groups? Both do the same as far as I can tell: they both work over different interfaces, they both come before the individual rules per interface. I am sort of wondering when you should use what ???
Floating Rules are good if you have multiple interfaces, so you can define these in the Floating Tab so that they are in effect for several Interfaces at one time.
-
1. Good idea on putting the Pass list into the IR_ category! Snort/Suricata processses a 'copy' of all the packets. So even if it was blocked by the Firewall, Snort/Suricata will still see it. Keep an eye on the Snort/Suricata alerts, and click on the "!" to see if the IP is listed or has the range blocked. I have noticed that over 90% of what Snort/Suricata Blocks is already being blocked by the Firewall Blocklists.
Hopefully when pfSense moves to 2.2 and they add NetMap api, it will allow a true-inline process for Snort/Suricata.
Thank you BB ;D
Ah, ok, so a package comes in, the firewall is the very first line of defense and in the millisecond it will take the firewall to block it, it will also send it to snort which will analyze it and will also block the IP - which is already blocked by the firewall.
So what we are doing right now is more or less a preparation for when 2.2. comes? As in: Jflsak sad he wants the firewall to do the work that the firewall can do, so Snort won't be wasting CPU (and speed) when not necessary. But if I understand you correctly currently that is still not the case, we will have to wait for 2.2, hence it is just 'preparing for'.
2. Suricata will load the Snort VRT ruleset except for about 600 rules. You can see that in the Suricata logs (which ones failed due to regex issues) I would still use the ruleset in Suricata.
(on another note, I still plan on staying with Snort for the time being)On that other note: why, if I may ask? I thought consensus was sort of Snort is no longer to be trusted?
3. I get them occasionally. Maybe Bill has some suggestions.
I hope ;D
-
Floating Rules are good if you have multiple interfaces, so you can define these in the Floating Tab so that they are in effect for several Interfaces at one time.
I am sure I am once again stupid, but: Interface Groups do exactly the same?
For example, I have MultiWAN, in which both my WAN's are. In the MultiWAN-tab I have set some rules that apply to both WAN's at the same time. Floating rules can do that too, so when would one use Interface Groups, and when would one use Floating Rules ???
-
@Hollander:
I wasn't clear enough, since I could hit geenstijl.nl, but not the movies they embed there. They are hosted on one of their other sites, dumpert.nl. And that one was blocked. The IP lookup turned out this was cloudflare. So I added a floating rule IR_PASS (which, magically, also appears in the dashboard widget :P ) that will use the IR_PASS alias to contain hosts that I might consider 'false positives'.
Just curious, btw, BB: why is cloudflare spamming you (or, better said: how?) I thought this was a reputable service? Obviously not, but I don't know why? I mean, large sites use cloudflare(?)
Here is what I show being blocked by that IP (IP may be different for you as cloudflare is a CDN and the IP could be different in your Country.
From these results, really makes you wonder if you would feel comfortable going to those websites? Note, the IPs below are from the ET Paid IQRisk Blocklist. No other lists show that Ip for me, so I am curious which list is blocking on your end?
grep "^141.101.116." pf/*
pf/ET_IPrep.txt:141.101.116.0/24
<snip>pf/e_tspywarecnc:141.101.116.17As an Economist, you should already know that answer ;) Money, Bones, Bread, Fins, Moola, Dinero
Why would they kicks spammers off of their network and lose revenue. Spread the word and block/boycott them all.
What happens is a Spammer sets up shop on a Hosting Service and Spams for as long as they can, after they get blocked, they move to another Hosting Service or just setup a new domain and new IPs. (Sadly, those IPs get recycled to some legit sites and people wonder why they are on a Blocklist when they just got these new IP addresses.)
I have had spamming issues from several Hosting Services HOSTNOC, OVH, Burstnet.
I am currently blocking the whole Burstnet network (3000 servers) as I was getting spammed for months without them doing anything about it. If you want to see how bad it is, take a look at this Google Group:
https://groups.google.com/forum/#!forum/news.admin.net-abuse.email
(and for a good laugh… ;D )</snip>Thank you ;D
I will use your CLI-examples to try and do that myself, and report back here what mine says for that site, dumpert.nl.
So if I understand you correctly the site dumpert.nl (part of a reputable news paper company group in The Netherlands) is not to be expected to be blamed, it is the fact that they are 'outsourcing' their hosting 'in the cloud' and the cloud provider is making a mess of it, hence risking dumpert.nl to be contaminated with crap ware.
-
I think both Snort or Suricata are both good options. The rules generally work for either, so the rules to disable are similar except for the Suricata Stream and other rules specific to Suricata. Suricata is a lot more involved and have a lot of new options not available in Snort.
I think the issue with Snort is when Cisco bought them, and the worry is that they close it to open-source at some time in the future. If that happens, everyone would either have to pay up, or switch to Suricata. As Suricata is harder to configure, getting it working now gets you ahead of the game.
So its all about choice and I think thats what Bill has said numerous times.
I think the Floating Tab also allows "Match" (instead of "pass", "Block" or "reject") these are necessary for things like Traffic Shaper to work. So if you use "Interface groups", I believe its probably the same in the end.
Yes lots of reputable sites do that. Take "Yahoo" as an example. Their site is laden with links to other sites and they are known to have issue for spyware excetera. But again, look at Yahoo's financial issues and they need the revenue.
Sometime, you visit a site, and it redirects to another IP just to popup an "Advert" or a drive-by installation of malicious software.
Also look at Dropbox, these malicious groups are using dropbox to push their malware as most people would assume anything from dropbox is safe? :o :o The next steps after IP blocking would be DNS Sinkholing or URL filtering, so that we can stop "known" URLs that lead to reputable sites but are still malicious.
-
@Hollander:
3. How do you get rid of all these log lines about traffic being blocked:
–- 255.255.255.255:10001 UDP
--- 239.255.255.250:1900 UDP
--- 224.0.0.252:5355 UDPThey were gone for months, but suddenly the log is being flooded with them again. Ever since I started with pfSense these lines appear to bug me every so many months.
Is snort/suricata blocking them or if it pfsense default block rule? if its snort or suricata, you can suppress them base on the rule that is being triggered:
#SURICATA IPv4 padding required
suppress gen_id 1, sig_id 2200007, track by_dst, ip 224.0.0.22 -
@Hollander:
I am sure I am once again stupid, but: Interface Groups do exactly the same?
For example, I have MultiWAN, in which both my WAN's are. In the MultiWAN-tab I have set some rules that apply to both WAN's at the same time. Floating rules can do that too, so when would one use Interface Groups, and when would one use Floating Rules ???
Floating rules are rules that can be active on multiple interfaces at the same time, by just merely CTRL+clicking multiple interfaces (extremely easy to add a new interface to an existing rule).
Not only that, floating rules are the first rules that are evaluated. Remember that rules are evaluated top to bottom (not only floating rules, ALL rules). pf(sense) starts from the top of the floating rules tab and works it's way down, then moves on to your interface's tab's top and works it's way down.
If you are going to block a packet, there is no need to pass that packet through the entire "firewall" stack. Passing it from rule to rule saying "does this match this rule?nope, move on to the next one" only to find out 100 rules later that "yes it does match this rule!hm, the rule says block, so I should throw this packet in the trash" will quickly guarantee that some networking "expert" out there will quickly tell you "spitWell there's your problem! You need an i3 for that", while in reality taking away bits and pieces from the work the system has to do makes even the slowest cpu you can afford, EXTREMELY capable. I've said it once, and I'll say it again: MOST people will NEVER need anything more than a p4/atom. You can't imagine the things that little atom box that could, well, could do (no pun intended).Floating rules help us in implementing these principles. The less work a system has to do, the more efficient it is at doing it.
A couple of years back I set up a couple of remote syslog servers at one on The Company's facilities. I was arguing to the networking/security "experts" (outside The Company) that I don't need a quad socket 12 cores per socket server to log a couple of damned lines. They kept saying "sysloging is one of the most difficult things you can do. Once you grow past a couple dozen of servers it will quickly bring down any server you throw at it", recommending about setting up at LEAST a single socket quad core CPU.
I was pointed at and laughed when I implemented the sysloging on a small ARM box, and curiously enough it kept up with the load. What they failed to see was that instead of requiring a dedicated power line to the sysloging servers, they each consumed less than 10W. A 24/7 10W load is NOTHING compared to what a recommended system would suck down. The trick to proving that industry leaders are complete and utter idiots is knowing your job and constantly thrive to be the best at it. The BEST! A "take no prisoners" approach to training/learning at your job, is the only secure way of achieving your goals.Oh, by the way, the trick to the ARM servers was filter the messages as early as possible (at the individual servers), then pass the actually needed logs on to the ARM servers WITHOUT passing them on to mysql. Ask yourself. Can I identify an attack/needed-to-identify-action using this line of syslog, merely by looking at it 5 years later? How about if I combine it with the lines above it and below it? If the answer is no, then that line should NOT be logged. It should be discarded as soon as possible (the start of syslog's directives is a great spot!). The EU Logging Standard (European Commission Information Systems Security Policy, Standard on Logging and Monitoring, page 12, technically page 11 section 8.4 Network Firewalls, continues to page 12) says to even log packets that are blocked by the default block rule. The multi-trillion $/€ question is: Why? There is no need to log those packets, since those packets will be discarded anyway. Log a single line per minute of those packets coming in. Then have the evidence that an attacker is still trying to penetrate/DoS your network. You keep the penpushers happy, and you keep your syslog servers happy. Logging 1 line is faster than logging 1,000 lines.
IF<<< (an infinitely huge IF,extremely likely this is NOT the case you are in) you need to analyze logs at the end of a time period for something particular, then import the logs in mysql all at once. If (an infinitely small if, almost a singularity, and most likely this IS the case you are in) you merely need logs to prove that IP 192.168.1.1 was poking at your ssh server, then you DON'T need to import the logs to mysql, EVER. Nope, not in a trillion years. A script grepping (zgrepping, why waste space?) your logs for a particular IP between 2 set time periods and spitting out the lines it's mentioned, is (educated guess based on my years of field experience) about 100 times faster than analyzing logs through a giant database containing billions and billions of logged entries. Yes a database system is designed for this. No you will NOT perform these lookups more than 3 times a day => you don't need the database system (and cluster, and load balancer that come with the contract).
Yet again we prove the industry leaders know nothing about the job they do. Yet again they will deny it, again and again. To keep a system performing at the top of it's potential, you need to KISS (not wanting to offend anyone, it's not directed at anyone in particular, well, maybe at the industry leaders, it's mentioned as in the KISS principle). Keep It Simple, Stupid. The KISS principle in the case of having to deal with multiple interface blocking type rules is:
- Evaluate those packets as soon as possible
- Simple to add/remove interfaces
- No need to duplicate rules to other interfaces.
In other words, floating quick rules :D
-
I'm over 25 years in the IT business as an engineer, consultant and many more functions, and I must say, you are so right.
