Taming the beasts… aka suricata blueprint
-
Isnt 2.2 FreeBSD 10 based??
Yes it is, but I couldn't find much info on this issue with my google FU…
I did find this link:
http://smyck.net/2014/01/22/freebsd-authentication-error/I don't have a 2.2 box, so I can't test it myself. If anyone else has 2.2, can you see if these two commands work? Don't need to be using my Script to test if fetch and https work on 2.2 Alpha.
cd /tmp
fetch -v -o honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"
wget –no-check-certificate -O honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1" -
My problem was a PEBKAC, I had a self signed cert installed in pfSense. I removed it and it works just fine. Thanks for your hard work. I can report that I'm not having any issues with the script or the widget in 2.2, all seems to be working fine except the patches for dns look up report that they can not be installed cleanly. Not important.
Thanks,
Bill -
all seems to be working fine except the patches for dns look up report that they can not be installed cleanly. Not important.
Good to hear!
Is this the 2.2 diag_dns.php file that matches what you have on your box?
https://github.com/pfsense/pfsense/blob/master/usr/local/www/diag_dns.phpIf its the same, I will post an updated version of the Patch.
-
I believe it is. On the current snapshot.
Patch can NOT be applied cleanly (detail)
Patch can NOT be reverted cleanly (detail)Thanks,
Bill -
Also neither widget displays list count or correctly determines failed downloads. It is correctly displaying other correct information. This is on Today's snapshot of 2.2. I didn't look into it because I can read logs… :)
PfIP Reputation2
minimize
close
(All Downloads Successful)List Count
Alias CIDRs Packets Updated Status
IR_IB 60359 0 Jul 29 22:42
IR_PRI1 2418 0 Jul 29 20:42
IR_PRI2 24652 40 Jul 30 15:52 -
I want to think jflsakfja and BBcan177 for this thread. Fantastic work.
-
I want to think jflsakfja and BBcan177 for this thread. Fantastic work.
As long as you are not thinking of us while taking a bath ;)
-
@jflsakfja:
I want to think jflsakfja and BBcan177 for this thread. Fantastic work.
As long as you are not thinking of us while taking a bath ;)
;D ;D ;D
-
JFL, could I ask: what is your opinion on this:
Why block an attacker that's already been blocked by your firewall? Because they will continue to beat on your door till they find something they can brute force or exploit. So lets use their initial scanning to identify hostile intent and block.
http://doc.emergingthreats.net/bin/view/Main/WhatEveryIDSUserShouldDo
It would make sense to this noob ;D
-
@Hollander:
JFL, could I ask: what is your opinion on this:
Why block an attacker that's already been blocked by your firewall? Because they will continue to beat on your door till they find something they can brute force or exploit. So lets use their initial scanning to identify hostile intent and block.
http://doc.emergingthreats.net/bin/view/Main/WhatEveryIDSUserShouldDo
It would make sense to this noob ;D
Nice to see that most things in this topic are also mirrored in that page. Too closely mirrored, if you catch my drift ;). Now if they had the same spirit about deleting their outdated rules, then we would all be happy ;D
And yes, that's what the recommended unused ports rule (in this topic and that page) does. It identifies people that are port scanning you, but maybe managed to get a list of open ports before they were banned. Cutting off their access before them noticing that you have open ports and try to connect to those, is a sane way of preventing them from getting onto your network. If you know that they are malicious, why even let them in the network in the first place?
My recommended rule was detecting ALL traffic to unused ports. His rule detects traffic that is attemping to be established. What if the "attacker" decides to scan in an unconventional way? You should never see traffic to those ports, end of story. If even a single packet arrives at those ports, then it is malicious, since you did not ask for it.
Didn't have time to read the entire page, but from quickly going through it it looks like he is recommending a similar set as to what is described in this topic. Suricata on pfsense belongs in the network gateway category, and as such should be responsible for protecting the network proactively if it can, or reactively if it cannot (encrypted connections terminated to internal servers for example). From suricata's/pfsense's point of view, it should try to make the network conform to specific requirements, like the requirement to contact specific DNS servers.
What I will comment on though is the section "Systems That Should Never Surf the Web". This is the typical wrong approach to using an IDS/IPS system. If you don't want those systems to surf the web, use what the firewall is designed to do: Be a glorrified router that routes according to specific rules, and block those hosts using a solution as close to the network as possible. If I were responsible for systems that shouldn't surf the web, then I would just air-gap them instead. It's the closest to the network you can be, there isn't actually a network (connecting them to the Internet), a separate physical intranet is acceptable, since not everybody has ninjas sworn in blood oaths to get into their networks.
But shouldn't the firewall be used for unused ports as well? From the inside of the network, absolutely yes. From the outside of the network you need the sentry (suricata) as high on the walls as possible. Bonus points if you can give him binoculars as well. The more he can see, the faster he will alert the rest of the town (pfsense) that danger is coming. The faster the town is warned, the faster the gates will be locked (host added to banned hosts). Ask yourself, what use is the sentry on top of the wall 1km away, monitoring the market stall thieves in the center of the town? Why the medieval thinking? If you haven't yet read the Art of War, you don't belong in the network security field.
-
I was looking at the Suricata Package and I don't see where the PORT Scanning Pre-Processor is configured? Is this option available in Suricata? or is it expected to be released in the 2.0x releases?
Along with what jflsakfja and Matt Jonkman (ET) have said, here is some more good advice:
http://dcid.me/notes/2013-jul-08
-
Quotes taken from http://dcid.me/notes/2013-jul-08 unless otherwise stated. Don't sue me to death :P
"And that’s a hard spot to be on. If you read all recent cases of APT, 100% of them started with zero-days sent via very targeted phishing emails. How do you protect against that?"
With the only way you can defend against that. Take a baseball bat with you when you are going on your tour around the office educating people not to click on every link/download every file sent to them via email. I got this right up to the point where (certain) people don't even answer phones from unknown numbers. Yes users can be educated, it's only a matter of which finger will be broken first. And yes, I'm being 100% serious."Unfortunately, very few companies have that level of monitoring and security enabled. Very few would be able to detect a user trying to increase their privileges or even detect any anomaly from where he is logging in from. They can have a firewall and an IDS, but nobody looks at them. They are just so noisy that it is very easy to miss the important activity."
More work for security "experts", more people to pick on for me ;D. A person (key word) should never be expected to watch logs for intrusion. EVER. If you are an industry leader and recommend you watch your logs closely, then all hope is lost for you. Please consider a career selling hotdogs instead. It's a lot easier than trying to convince the world you know anything about network security.
I do NOT care what the security industry is recommending about logs. It's a case of the right way (watch the logs), the wrong way (don't watch the logs) and my way. My way is setting up automatic monitoring of the logs and alert the sysadmins automatically when the monitoring detects anomalies. Did the webserver restart in the middle of night? Which user requested the restart? From where were they connected to the system? What other commands were ran in the last 10 minutes of them being logged in?
Watch the IDS? Who in their right mind watches an IDS? Have you seen the volume of data it logs in realtime? The key to keeping your fire-breathing dragon (suricata, snort is dead, stop feeding it) on the leash is to look at the blocked hosts every now and then. Sort their alerts alphabetically, take out the common suspects (unused ports rule for example, should be the largest volume) then take a close look at what the other blocks say. Make this a habit instead of wasting hours of company time reading what others did last night on fb/twitter/other + "thing". My way? If the IDS bans a host with an alert other than the usual suspects, drop me an email alerting me to this.
Sidenote to the above couple of paragraphs. You can safely skip this.
Got a call today from a client of mine. He told me that the router must have blocked him again, because he tried to ping the webhosting server to find out its IP. The conversation went something like this:Client: "Hi, how are you?"
Me: "Hello, I'm good, how about you?"
Client: "Fine. Hey the router must have blocked me."
Me: "Why?"
Client: "I tried to ping the webserver that X is hosted on, and after a single reply all traffic was cut off."
Me: "That means the security system did it's job correctly. Doesn't windows have a command similar to the "host" command on linux? Why did you ping it?"
Client: "It's the easiest way to get the IP. My IP is XX.XX…"
Me: "Wait, hold on, I'm on the second password, 1 more to go"
Client: "You should allow 4 pings, that would make it easier"
Me: "No. I get a few hundred thousand pings per day from China,Romania,Russia, I'm good thank you. Use the proper command to get the IP. On linux it's host, on windows I dunno, don't have any experience with "that"" (exact phrase removed, this is a public forum, even visited by minors)
Client: Doing other things while I try to find the password to log on to the router. We said something about this in the beginning of the guide (I think, I'm getting old :p). If you know the passwords to your systems, you are doing it wrong (I should trademark that phrase).
Me: "ok, what's your IP?"
Client: "XX.XXX.XXX.XXX"
Me: "Hm... yea, you pinged it using windows" BOFH like clickety-click
Client: "Ok thanks, it's working now"
Me: "Happy to help, if you have any other problems give me a call."See? It's only a matter of which finger you'll break first, trust me.
"I will certainly do a full post about it, but what I learned is that you can’t really block all attacks. You can’t even detect all of them. Your software will have vulnerabilities. You will make mistakes. You will get owned one day!"
Started securing networks/systems when I was 10 years old. I'm 27 now, and that day has not yet come. It's a record I'm doing my best to keep, thank you very much."What you need is a system to alert and raise red flags when that does happen, so you can respond. The “you just got owned” alerting system. And those are not hard to setup, but requires a different mindset."
My points exactly ;DComputers have become extremely good at one thing: Doing the same thing over and over without making a mistake. Set up your network/systems so that EVERYTHING is logged to syslog. From temperatures, to disk performance change (I get a couple of seek performance changed a day, must really replace that disk, the head actuator is getting weak), to users logging in, to users logging out, commands being ran by users, scheduled tasks, EVERYTHING.
Filter through the noise by discarding unneeded messages. For example, a host's resync with a mariadb galera cluster causes (about) 30 lines of syslog messages. It should only generate a single line from that particular host, and a line from each of the cluster members seeing the host joining.Keep the log messages to a manageable volume, then set up automatic monitoring for those logs. If something is suspicious, contact a sysadmin to have it looked at. If one particular sysadmin logs from networks X and Y, and suddenly logs in from network Z in the middle of the sahara desert, then something fishy is going on.
And as stupid as this may sound, it will actually save you when a judge asks you "WHAT?". Keep >forensic< evidence trails. Yes, which pc connected to what, under what user, what commands were given and all that, BUT make sure those can actually STAND in a court of law. If I was a consultant for the prosecution, I would provide evidence showing that the authenticity of the logs submitted cannot be questioned. If i was a consultant for the defense, I would provide evidence showing that those logs that were submitted earlier as unquestionable evidence showing that X is guilty, cannot be actually checked to be sure that they weren't manually typed up and submitted.
We already got pretty far with this guide and the help of the pfsense devs, package maintainers, other forum members. If we keep this up, not one of the security industry "leaders" will have a job. Think of the children/cyber/terrorism/Russia ;D
I should really finish that internal book for The Company: "Building sentient AI security systems. Their applications and limitations." aka the "How to build the Matrix guide".
An example of this is an automated system we use at The Company. It takes a list of the servers and their uses. It then decides what each system should run, sets it up accordingly, and starts monitoring it. Good events are slowly ignored, bad events are evaluated and future systems are set up to resist them (if possible). We had a few "near employee firing" experiences, but it slowly learned to behave ;D
To the disbelievers, you can also design the same system. You think the technology still isn't there? puppet/chef/nagios (or others)/fail2ban/logwatch/syslog/ssh/network-booting/not booting servers unless their specific usb key is plugged in (HDD encrypted, of course). VPN the logs to another location and print them on a dot-matrix, just for the hell of it. Connect the dots and stop questioning everything I say ;D. If you are really creative, you can even set up your systems to be as simple as a message popping up "Hey, go plug USB key XXXXXX into server YYYYYY, it needs to restart". Or spend the $200 it takes to do it using an actuator ;) Taking care of the systems then becomes a mob the floor, stare at the screen for pretty graphs to dip/raise.
-
I think it was obvious that you wouldn't look at every single log and every single IDS alert. It is true thou that people can setup these systems and get drowned out by the minutia.
However, this is not what he is saying. Its important to filter those logs and those alerts from the IDS so you are only seeing the important ones. Setting up tripwires so that if and when someone does get in, or someone on the inside does something, it will trip an alarm.
It all great to put up blocking system to stop maliciousness from getting in, but as the article articulated, you will make a mistake or someone will, or a zero-day and something gets past your security. Nothing is impenetrable.
Tools like OSSEC are good to have running on servers so that it can alert on file changes or brute forces inside your network. I run Security Onion running full packet capture immediately after my Firewall. So when you have an issue, you can atleast have a bread crumb trail so you can see what was accessed and infiltrated or attempted to be. These logs and pcaps can determine whether someone just snooped around or if they actually downloaded/uploaded anything.
If you look at most network intrusions, its not the first hack that made any damage. Most likely a single event won't be catastrophic. So if you have detection on the inside, they will most likely trip an alarm that will allow you to root out an intruder before they do damage.
As with age, I never judge the length of time someone has been doing something as a sign of wisdom. People do jobs for their entire life; unfortunately some of them never had it right in the first place.
-
I was looking at the Suricata Package and I don't see where the PORT Scanning Pre-Processor is configured? Is this option available in Suricata? or is it expected to be released in the 2.0x releases?
Along with what jflsakfja and Matt Jonkman (ET) have said, here is some more good advice:
http://dcid.me/notes/2013-jul-08
I'm not a Suricata expert yet, but to the best of my knowledge there is no equivalent of Snort's sf_portscan preprocessor in Suricata. There are text rules (ET Scan rules come to mind) that can detect most port scans, though.
Bill
-
Give me a ping, Vasili. One ping only.
-
Hi,
Trying to code a custom rule and getting an error. The rule is basically to block the traffic to closed ports, something like:
alert tcp $EXTERNAL_NET any -> any [1:1024,![XX,XX,XX,XXX]]
However, I'm getting an error:
[ERRCODE: SC_ERR_NEGATED_VALUE_IN_PORT_RANGE(56)] - Can't have a negated value in a range.I thought this was a valid syntax. What am I missing here?
Thanks for your help!
-
I thought this was a valid syntax. What am I missing here?
This is from an older manual, but I believe its still the same format.
2.2.4 Port Numbers
Port numbers may be specified in a number of ways, including "any" ports, static port definitions, ranges, and by negation. "Any" ports are a wildcard value, meaning literally any port. Static ports are indicated by a single port number, such as 111 for port mapper, 23 for telnet, or 80 for http, etc. Port ranges are indicated with the range operator ":". The range operator may be applied in a number of ways to take on different meanings, such as in Figure 2.6.
log udp any any -> 192.168.1.0/24 1:1024 log udp
traffic coming from any port and destination ports ranging from 1 to 1024
log tcp any any -> 192.168.1.0/24 :6000log tcp traffic from any port going to ports less than or equal to 6000
log tcp any :1024 -> 192.168.1.0/24 500:
log tcp traffic from privileged ports less than or equal to 1024 going to ports greater than or equal to 500
Port negation is indicated by using the negation operator "!". The negation operator may be applied against any of the other rule types (except any, which would translate to none, how Zen…). For example, if for some twisted reason you wanted to log everything except the X Windows ports, you could do something like the rule in Figure 2.7.
log tcp any any -> 192.168.1.0/24 !6000:6010
-
Hi,
Trying to code a custom rule and getting an error. The rule is basically to block the traffic to closed ports, something like:
alert tcp $EXTERNAL_NET any -> any [1:1024,![XX,XX,XX,XXX]]
However, I'm getting an error:
[ERRCODE: SC_ERR_NEGATED_VALUE_IN_PORT_RANGE(56)] - Can't have a negated value in a range.I thought this was a valid syntax. What am I missing here?
Thanks for your help!
You need to remove the regular ports from the rule and only select the negated range. It wouldn't be any use anyway to include 1:1024. If you don't allow that range, the rule will still alert for those ports, since that's what you told the IDS to do. Alert on any port other than the open ports (used ports). Any port you don't specifically allow, will generate the alert.
@G.D. Wusser Esq.: It's not a matter of one ping only Vasili. It's a matter of not using a screwdriver and a hammer to remove a 1/2" bolt. Yes it can be done, yes it's extremely useful if the head of the bolt is broken off for any reason, but it's not the right tool for the job. Use the 1/2" wrench to remove the 1/2" bolt.
To ping a host you first need to resolve the host, then ping it.
To find out the IP of a host, you just need to resolve the host. -
@jflsakfja next time you speak to that user, tell them to use nslookup on windows. you can look up by hostname or ip…
-
@jflsakfja:
You need to remove the regular ports from the rule and only select the negated range. It wouldn't be any use anyway to include 1:1024. If you don't allow that range, the rule will still alert for those ports, since that's what you told the IDS to do. Alert on any port other than the open ports (used ports). Any port you don't specifically allow, will generate the alert.
Thanks jflsakfja. That makes sense. However, the reason I stated 1:1024 is because I want ports 1024: onwards to remain open as well. Basically I would like to block all destination privileged port, except a few (IPsec, OpenVPN, etc.), but also leave non-privileged ports open as well (as you suggested :) ). What would be the appropriate syntax for the port part of the rule? Based on the documentation I found online, my proposed syntax should work, but it doesn't and I get the error that I mentioned.
Maybe I'm over-thinking this. Can I use pfsense port alias here?
Also, I've been using snort for a while before this, and it seems that suricata is utilizing more CPU than snort. I got a 50/10 line at home, and during speedtests while with snort my CPU utilization would spike to 30-40% (I have an Atom D2550). However, with suricata during the same test the CPU spikes to 85-95%. And during regular Netflix/youtube streaming suricata seems to use 2x more CPU (snort: ~2-4% avg, suricata: ~5-7%). Anyone else notice that?
Again, thanks for the help!
-
Thanks jflsakfja. That makes sense. However, the reason I stated 1:1024 is because I want ports 1024: onwards to remain open as well. Basically I would like to block all destination privileged port, except a few (IPsec, OpenVPN, etc.), but also leave non-privileged ports open as well (as you suggested :) ). What would be the appropriate syntax for the port part of the rule? Based on the documentation I found online, my proposed syntax should work, but it doesn't and I get the error that I mentioned.
As suggested, keep the rule alerting on all unused ports: ![port1,port2,port3,port4:port25,port1024:port65535]
You just need to set up the ports you use inside that [ ]. I highly suggest to include all the unprivileged ports in there, unless you manually go into every program you use and tell it to use a specific unprivileged range.Maybe I'm over-thinking this. Can I use pfsense port alias here?
Nope. I used to be able to declare the variables at the start of the custom rules tab (eg USED_PORTS) but last time I tried it, it didn't work. Didn't fiddle with it anymore, since it's not that many ports you need to open up anyway.
Also, I've been using snort for a while before this, and it seems that suricata is utilizing more CPU than snort. I got a 50/10 line at home, and during speedtests while with snort my CPU utilization would spike to 30-40% (I have an Atom D2550). However, with suricata during the same test the CPU spikes to 85-95%. And during regular Netflix/youtube streaming suricata seems to use 2x more CPU (snort: ~2-4% avg, suricata: ~5-7%). Anyone else notice that?
Again, thanks for the help!
Yeap seen that too. I'm putting my money on the old version of suricata as being the culprit for this.
@Cino: Will do, thanks.
-
Thanks, jflsakfja. That rule syntax worked.
I see that the latest Suricata release is 2.0.3. Any ideas when pfsense package will be upgraded to that version? Is there a way to manually update?
I have noticed that as of late I'm being pinged from multiple hosts for several minutes at a time. Feels like an attack from hijacked hosts. I put a rule to block those, and I was able to block almost 500 hosts in 24 hours via suricata.
Thanks again!
-
Thanks, jflsakfja. That rule syntax worked.
I see that the latest Suricata release is 2.0.3. Any ideas when pfsense package will be upgraded to that version? Is there a way to manually update?
I have noticed that as of late I'm being pinged from multiple hosts for several minutes at a time. Feels like an attack from hijacked hosts. I put a rule to block those, and I was able to block almost 500 hosts in 24 hours via suricata.
Thanks again!
I think 2.x is coming with the next release of the suricata package. How long that takes, dunno though.
The pings are regular internet noise. Don't worry about it. As long as you are not responding back, then you are still flying under the radar.
-
After doing some testing of Snort vs Suricata, I've decided to go back to Snort.
For whatever reason, looks like Comcast upgraded my line to 100/10 tier in the last couple of days. Now, with my D2550 Atom CPU Suricata was maxing out my CPU cycles and my max throughput was 95 mbps (@100% CPU load). I've tried snort and @45% CPU load I'm getting about 108 mbps (plus it's a more stable/smooth download vs suricata, which was more "jumpy"). I've ran the test 2x between the two, and same result. Suricata came to be the bottleneck for me. And Suricata couldn't download Snort VRT rule set, so, snort had a larger rule set running as well. (although I've never seen a single VRT rule triggered, only the custom rules and the ET rules).
I'll try Suricata again once the 2.0 comes to pfsense. Hopefully that'll perform better.
Just my 2 cents. Thanks for the help!
-
It would be interesting to see more details about your setup. Did you disable the rules I recommended in this topic? Even the amazon one (yes that single rule does matter)? How much RAM was used? Nice to see that a dual core atom @ 1.86Ghz can (nearly) max out 100Mbps. I'm sure with a bit of tuning it could get there, unless you have already removed suricata and installed snort.
Don't worry about the VRT rules.
-
I have TWC. I'm currently 100/5. I have both snort and suricata running on my D510 Atom with no issue. Running speed test, I can max out at 107-110mbps. CPU% anywhere from 45% to 100%. If I download torrents, cpu will peg at 100% but i'm still able to browse with no issues.
-
Respect for the little atoms that could ;D. The newer 4 core models (technically a celeron, or is it the other way around?) are interesting, thinking about getting a couple for testing. A fully loaded psfsense system based on those should be close to 30W (cpu+cards+hdd).
-
I want to say my D510 box is running around 20-25watts. Have to find my build notes to confirm. off topic but I have a few interfaces, traffic shaping, snort, pfblocker, suricta, squid, ntop, vnstat… she runs good... Need to change her it 64bit so I can use all of the 4gb of memory but I think i'll wait for 2.2 to be release then do a fresh install and rebuild the config for fun :o
-
I want to say my D510 box is running around 20-25watts. Have to find my build notes to confirm. off topic but I have a few interfaces, traffic shaping, snort, pfblocker, suricta, squid, ntop, vnstat… she runs good... Need to change her it 64bit so I can use all of the 4gb of memory but I think i'll wait for 2.2 to be release then do a fresh install and rebuild the config for fun :o
Ah, the Debian bug. Nothing happens to it, to the point where you want to upgrade to testing just for the hope of something breaking? :p
Atoms are perfect for personal use, IMHO.
-
@jflsakfja:
Thanks, jflsakfja. That rule syntax worked.
I see that the latest Suricata release is 2.0.3. Any ideas when pfsense package will be upgraded to that version? Is there a way to manually update?
I have noticed that as of late I'm being pinged from multiple hosts for several minutes at a time. Feels like an attack from hijacked hosts. I put a rule to block those, and I was able to block almost 500 hosts in 24 hours via suricata.
Thanks again!
I think 2.x is coming with the next release of the suricata package. How long that takes, dunno though.
The pings are regular internet noise. Don't worry about it. As long as you are not responding back, then you are still flying under the radar.
I am hopefully just a week or so away from posting the Pull Request for 2.0.x Suricata. I ran into a small snag compiling the new package for 2.2 of pfSense, but I think I have a solution for that now. I have been developing/testing with 2.0.2, but if it's not too big of a change I will bump it to 2.0.3 before I post the Pull Request.
Bill
-
I am hopefully just a week or so away from posting the Pull Request for 2.0.x Suricata. I ran into a small snag compiling the new package for 2.2 of pfSense, but I think I have a solution for that now. I have been developing/testing with 2.0.2, but if it's not too big of a change I will bump it to 2.0.3 before I post the Pull Request.
Bill
Hi Bill ;D
Would that also include the suggestion from one of the biggest noobs on this board to have an easy way to multi-enable/disable the rules per category (the same check boxes you see in the left side of the firewall rules screens)?
That would be quite lovely, so to speak :P
-
A most stupid question, for which I am by now famous: wat is the OpenVPN-interface; WAN or LAN?
Virtual Private Network would suggest LAN, but on the other hand: it is connected to the WAN ???
-
@jflsakfja:
It would be interesting to see more details about your setup. Did you disable the rules I recommended in this topic? Even the amazon one (yes that single rule does matter)? How much RAM was used? Nice to see that a dual core atom @ 1.86Ghz can (nearly) max out 100Mbps. I'm sure with a bit of tuning it could get there, unless you have already removed suricata and installed snort.
Don't worry about the VRT rules.
My setup is pretty simple, bough off the newegg:
** OEM Production 2550L2D-MxPC Intel NM10 Black Mini / Booksize Barebone System - OEM (http://www.newegg.com/Product/Product.aspx?Item=N82E16856205007)
** 4GB of RAM
** 32GB SSD
** Latest version of pfsense
** 1 LAN+ 1 WAN + IPsec + OpenVPN
** Bind, snort/suricata (not at the same time), pfblockerIt has dual Broadcom nics, which is not too bad. If I disable the snort/suricata IPS, then @108 mbps down the CPU load is only 33% or so. So, theoretically this thing should be able to push 250 mbps easily. Not too shabby.
On the suricata setup, I followed your instructions for the rules. So, I did turn off the ones you mentioned in the posts here. However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there. For comparison sake, I have the same rules enabled for snort (just keeping it apples-to-apples).
I'm a believer in suricata, based on what I read, but probably not quite prime-time ready (at least in my setup, based on my limited testing). I have not uninstalled it, just disabled it at this point. I'll try again once 2.x.x comes out, hopefully soon.
-
…
The High Level function of the script:Download Individual List
Extract IPs
Save copy to /orig Folder
Check for Ranges that have 255 IPs and mark a single /24 Range
Process /24 (Which looks for repeat Offenders in a /24 Range) (max variable) Individual Blocklist Only.
Duplication CheckOnce all of the Downloads are completed that were scheduled to run:
The Following is performed Globally on ALL Lists, except for the ones that were marked as "p24=no" on the Collect Line.
p-Deduplication - Looks for Repeat Offenders that are over the pmax variable regardless of Country Code.
d-DeDuplication - Looks for Repeat Offenders that are over the dmax variable but uses the Country Code Whitelist function.
If the Sanity Checks passes, it will create the TIER (Group) lists and perform the "pfctl" commands to update the pfSense Alias Tables.
If you decide to remove a list, you need to add "remove" after the collect line. When the script runs at its next scheduled run, it will remove the list from the database properly. Don't try to do this manually.
If you follow the High level steps, when you use the p24 process in d-deduplicaton, it will look for a repeat range of malicious IPs and find all of the Blocklists that have this IP listed.
The FIRST blocklists get a single x.x.x.0/24 Block and all of the other Lists that have the range are deleted.
So if a List is removed, and it happens to be a list that had the p24 process and was the first list processed as above, then you have no Blocklists for that range. This will correct itself on when the Lists are re-downloaded but that could be 1-4hrs depending on when the Lists are scheduled to run.
To get back into Sync, you can run this function:
[ [b]./pfiprep killdb ]
Which will wipe the Database (Settings are not touched) and it will resync the database.
Out of Curiosity, which Lists did you disable?
Another Function is to use the "IR_Match" Alias in the Floating Rules as a "Match" Rule. This will show you activity for the IP Ranges that passed the Country Code Whitelist process. Because its a "Match" rule, it will not block, but just log the activity.
Since I have been running the script, I have not found too many False Positives, but I always recommend not to disable a list but to create a "SAFE Alias" Rule that is defined above the "Block/Reject" Rules. And just add the IPs that you want to allow.
The Patch for diag_dns.php will also work when looking at the Snort/Suricata Alert Logs.
If you are running Snort/Suricata, when you click on the "!" ICON to Resolve an IP, you will find that most of the IPs are already listed in the BlockLists. You will also see over time that it will pickup an Alert for an IP but the Blocklists do not have the specific IP but there are several IPs within the same Range that are being Blocked.
Also in diag_dns.php, there are several IP Reputation Links that can help you determine the Reputation of any Blocked IP before you remove a list, or Add an IP to the SAFE Alias list.
Let me know if you need any clarification or any other help.
Hi BBcan177, thanks for all your help setting up the scripts. I managed to get everything setup, including the widgets & the DNS patch.
Could you please shed some more light on the max, dmax, and pmax variables? Not completely clear on the differences between them and how they operate.
For example, upon first run (using max=5, dmax=5, pmax=50), I had 7 unique addresses in the 104.28.7.x range in the IR_Match alias file. This also created a 104.28.7.0/24 entry as well. I tried accessing a site that was not one of the unique addresses (ie. not blocked by the lists), but it was blocked by the /24 range. When I changed max=10 and dmax=10, not only does the /24 range not appear (good) but the 7 unique blocked addresses don't appear either (bad).
Thus, just need a better understanding of how max, dmax, and pmax work, and what happens when you change the values. -
@Double:
Hi BBcan177, thanks for all your help setting up the scripts. I managed to get everything setup, including the widgets & the DNS patch.
Good job!
Could you please shed some more light on the max, dmax, and pmax variables?
-
Using a "max" variable, if it finds over the Max variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries on an individual Blocklist Basis.
-
Using a "dmax" variable if it finds over the dmax variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries at the end of the download process on all of the Blocklists together.
-
Using a "pmax" variable, if it finds over the dmax variable it will process a /24 Block excluding Country Code whitelist at the end of the download process on all of the Blocklists together.
I had 7 unique addresses in the 104.28.7.x range in the IR_Match alias file. This also created a 104.28.7.0/24 entry as well. I tried accessing a site that was not one of the unique addresses (ie. not blocked by the lists), but it was blocked by the /24 range.
You are referencing "Match" aliases here.
Here is a snipet from the pfiprep script-
# country Code p24 Process (pass/match) ccwhite=match # Define what to do with IP Ranges found that are in the # Country Code p24 Process (block/match) ccblack=block # For pfSense, the "Match" IPs can be "Monitored" with # "Floating Rules" which can log packets from these IP Ranges, # but still allow the Blocking of the Individual IPs found in the # same /24 Range.
So the script will Block a whole /24 range depending if you select ccblack=block.
ccwhite=match will put the IP ranges that are in the Safe Country list into a match alias.So the match file has all of the IPs that are being blocked with a "!" at the start of the IP to tell pfSense not to match the "!" excluded IPs, and match on anything else in the /24 range.
I would suggest leaving the "match" alone until you get everything else working. Change the ccwhite=match to ccwhite=pass
So at a high level, the max,dmax and pmax variables look at how many IPs are repeat offenders in all of the blocklists. And then depending on these settings block/match as required.
-
-
However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there.
I don't believe that this is correct.
In pfBlocker or in my pfiprep script, you can use the following dShield URL:
https://feeds.dshield.org/block.txt
and for Spamhaus:
http://www.spamhaus.org/drop/drop.txt
http://www.spamhaus.org/drop/edrop.txtYou don't need to have them enabled in Snort/Suricata if its in pfBlocker/pfIPrep
You are probably seeing Snort/Suricata Alerting on these alerts because Snort/Suricata is not a true-inline IPS. These packages are inspecting a "copy" of each packet, so even thou the pf filter blocked the IP, Snort/Suricata will still alert because it is seeing a 'copy' of the packets.
I posted here :
https://forum.pfsense.org/index.php?topic=78062.msg432804#msg432804
with what Rules can be disabled as these can be implemented in pfBlocker/pfIPrep.
-
However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there.
I don't believe that this is correct.
In pfBlocker or in my pfiprep script, you can use the following dShield URL:
https://feeds.dshield.org/block.txt
and for Spamhaus:
http://www.spamhaus.org/drop/drop.txt
http://www.spamhaus.org/drop/edrop.txtYou don't need to have them enabled in Snort/Suricata if its in pfBlocker/pfIPrep
You are probably seeing Snort/Suricata Alerting on these alerts because Snort/Suricata is not a true-inline IPS. These packages are inspecting a "copy" of each packet, so even thou the pf filter blocked the IP, Snort/Suricata will still alert because it is seeing a 'copy' of the packets.
I posted here :
https://forum.pfsense.org/index.php?topic=78062.msg432804#msg432804
with what Rules can be disabled as these can be implemented in pfBlocker/pfIPrep.
Thanks BB. I didn't have those lists added. I will do so and disable the categories on snort/suricata.
-
Thanks BB. I didn't have those lists added. I will do so and disable the categories on snort/suricata.
Anytime.
Take a look at my script. It has a lot of Threat Sources. Not all of them can be used in pfBlocker as it can't recognize the different formats like my script can.
-
Thanks to those that have spent their time to make document so much and distribute solid advice.
Its been a long time since I administered or needed to administer a firewall and was about to setup SNORT and such to build a better kid trap for my kids to have access to the internet. The efforts here convinced me to use Suricata and I'm happy I did. While a lot of the initial setup is redundant to me, it does cover the general theme for my multii-WAN-LAN FW.Thanks for all the efforts!!
-
@Hollander:
A most stupid question, for which I am by now famous: wat is the OpenVPN-interface; WAN or LAN?
Virtual Private Network would suggest LAN, but on the other hand: it is connected to the WAN ???
Think of it like a dedicated ethernet line connecting two adjacent houses, without the dedicated ethernet line. That's why it's called a "private" network, because the tunnel is really a direct connection between two points, and (supposedly) it actively tries to be secure at it, for example by tearing the tunnel down and establishing a new one when attackers interfere with it. Like digging up the cable and moving it a couple feet over, when detecting that someone is tampering with it.
There are a couple of ways to set up openvpn:
- Terminating the tunnel on an internal (separate) interface, then use rules to direct traffic where it's supposed to go.
or - An even safer way is terminating the tunnel on a separate host connected to a separate LAN-type interface. A raspberrypi/cubox-i/other-cheap-ARM-thingy is perfect for this, unless trying to route loads and loads of bandwidth through it. If it's just for remotely administering a pfsense (I'll save my don't do it speech ;D) then go for it. Don't forget the interface rules.
On the suricata setup, I followed your instructions for the rules. So, I did turn off the ones you mentioned in the posts here. However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there. For comparison sake, I have the same rules enabled for snort (just keeping it apples-to-apples).
I'm a believer in suricata, based on what I read, but probably not quite prime-time ready (at least in my setup, based on my limited testing). I have not uninstalled it, just disabled it at this point. I'll try again once 2.x.x comes out, hopefully soon.
Dshield/drop is a lot of work for an IDS. Try enabling suricata without those categories and test again, if it's not too much trouble.
You've demonstrated the reason this topic was created. Using the IDS part of a gateway is using 3 times the power the firewall part is using, which is exactly the reason I started this and the snort topics.
@all:
WRT the low power, a particular asrock mobo caught my eye: http://www.asrock.com/mb/Intel/Q1900M/. It has space for 6 intels (2x1 port + 1x4 ports) + extra onboard nic. 2 uplinks, 2 downlinks, 2 to play with, 1 spare. It's all you want in a small cute package ;D. Anyone out there care to try it/has tried it and want to share experiences? I'm sure for a top-of-rack firewall it should be perfect. - Terminating the tunnel on an internal (separate) interface, then use rules to direct traffic where it's supposed to go.