Taming the beasts… aka suricata blueprint

Good to hear the thread was so much help.

Thanks to all the guys that made it happen :)

BBcan177

In conclusion I'd say you guys have given some super costly consulting away. I appreciate the efforts!

I think the general concensus is when people help you in this forum, that you pass that information down to those that need it… That's why this forum is so good!

:) :)

Since wetspark's post, I've been meaning to ask all how things are going if you followed this guide.

@ALL Is everything working like it should? Does it get the job done? How stable was the resulting system?

If anyone has anything to add, please do so. The more discussion that goes into pfsense/suricata/lists the better for us all ;D

EDIT: In other words, was the guide successful in taming the beasts? :D

And this is what comes out of the guide:

suricata_record.jpg_thumb

Mr. Jingles

@jflsakfja:

Since wetspark's post, I've been meaning to ask all how things are going if you followed this guide.

@ALL Is everything working like it should? Does it get the job done? How stable was the resulting system?

If anyone has anything to add, please do so. The more discussion that goes into pfsense/suricata/lists the better for us all ;D

EDIT: In other words, was the guide successful in taming the beasts? :D

I was away for a while, so sorry for not responding sooner :-*

It is very stable. Your guide about the firewall rules shows a lot of things being blocked albeit the internet experience doesn't suffer from it (e.g. websites can still be accessed, WIFE doesn't complain although I see numerous blocks from her LAN-IP out to the internets).

BB's lists also works very nice, although there are quite some false positives in it –- although BB can't be blamed for that, of course not.

I can not yet comment on the Suricata matters, as I currently have Snort:
1. I wanted to move over to Suricata but your rules instructions are massive, and the Suricate GUI doesn't allow me to fix the rule instructions relatively fast.
2. The great Bill anounced some modifications to the GUI, I don't know where that currently is at, because:
2A. I had to uninstall Suricata and some other packages since one of them was making my box crash - and I don't know which one it was nor did I have the time to find out;
2B. I was away, busy, playing 'Zorro' for some people out there ;D

So:
1. Firewall rules seem very useful;
2. BB's script is an artwork that performs very well too;
3. I am dying to dive into Suricata but I need to catch up to where it is at, avoiding the repeated crashed of some weeks ago.

In the end:
Great JFL;
Great BB;
Great Bill;
Great mysterious man who refuses me to buy him a coffee;
Great pfSense team;
Great many other helpful members;
Not so great few members ( ;D ;D ;D ;D ;D ).

wcrowder

@jflsakfja

Can that beast ever really be tamed? But your posts have went along way taming them for my meager home network. With kids and all my toys, having this king of control over that beast is assume. Thank you for your time putting this all to together.

foetus

I just found an issue and would like some input (Really need a vacation soon..).

So, in regards to the block-lists, I use floating rules instead of interface specific rules. (pfblocker)
All good and well, be it I use a couple of white-lists also (generated by a whois cron job, or just plain manual).

For example : I have a Google white-list to allow both gmail, google dns, the bunch. General floating rules for both directions.
BUT, if I apply the "Apply the action immediately on match." my NAT rules are completely ignored. So forget forwarding a match in the whitelist floating rule to an exchange server internally.

Causing the very nice result of not a single Gmail message getting delivered anymore.

Using the white-list in the specific interfaces will have no use, since the block lists floating rules will always take priority.
So, any other solution except for changing all floating rules to interface specifics? Be it manually or pfblocker.

*edit
to be specific. I just want to whitelist the whole list I create. Not start with specific port ranges.

@foetus:

I just found an issue and would like some input (Really need a vacation soon..).

So, in regards to the block-lists, I use floating rules instead of interface specific rules. (pfblocker)
All good and well, be it I use a couple of white-lists also (generated by a whois cron job, or just plain manual).

For example : I have a Google white-list to allow both gmail, google dns, the bunch. General floating rules for both directions.
BUT, if I apply the "Apply the action immediately on match." my NAT rules are completely ignored. So forget forwarding a match in the whitelist floating rule to an exchange server internally.

Causing the very nice result of not a single Gmail message getting delivered anymore.

Using the white-list in the specific interfaces will have no use, since the block lists floating rules will always take priority.
So, any other solution except for changing all floating rules to interface specifics? Be it manually or pfblocker.

*edit
to be specific. I just want to whitelist the whole list I create. Not start with specific port ranges.

Can you please post your rules? It's not very clear what's happening. Are you using pfblocker as aliases + floating rules?

NAT rules shouldn't be just ignored. Think of NAT as the "final" stepping stone. NAT should be the last thing that a packet sees coming out of an interface. Even if it is explicitly passed, it only goes up to the interface if it's going to be NATed. If the interface has any NATing applied to it, that packet must follow that.

wetspark

jflsakfja,

I see the pull request has gone in for Suricata 2. Plan to start a new thread and refresh the beast taming blueprint with updates or just update this thread?

I'm excited to get v2 for sure. Besides the core areas of improvement, the sig management and log output abilities are going to be great!

A lot of changes are coming, and that will take a while to properly test and document them. What I'm planning to do is a separate thread where we all discuss the useless rules, and come up with bare minimum disablesid files that (hopefully) everybody that installs suricata/snort will use. Imagine not having to go through my list and disable all those rules ;)

I also plan to post instructions on tweaking suricata preprocessor settings, and maybe get The Company to release our permanently banned list. We just recently peaked at nearly 15K suricata banned hosts, that gives us a whole lot of intelligence on certain IP ranges (interesting fact: Microsoft (yes THAT Microsoft) has a subnet dedicated to remotely scanning hosts). This list already contains a whole lot of hosts, and I'm currently working on figuring out a way to get suricata to ignore the packets from that list, so that we don't waste processing on those (the hosts are already blocked by pfsense, but the copy of the packets still has to pass through suricata and get processed). There is a way to set the packet "forwarder" to ignore certain packets, but I didn't have the time to mess with it. Keeping track of a 4 million IP list takes a lot of head scratching :o (we have a /11 in that list).

There is also an extremely thin chance of The Company actually releasing our custom rules. The only thing stopping us from doing so isn't that they are proprietary, it's the fact that the vast majority of so called "security experts" will be out of job if we do release them. We had to clear up our blocked suricata hosts recently (IPv6 bug) and within an hour they had already added 200 blocked hosts.

So many things to do, so few years to do them in. I envy elves for living a few centuries.

A copy+paste from the list:
emerging-ftp > all except:
2101377 GPL FTP wu-ftp bad file completion attempt <<< breaks filezilla
2101378 GPL FTP wu-ftp bad file completion attempt with brace <<< breaks filezilla
A copy paste from ET's RSS:

[–-] Removed rules: [–-]
…snipped...
2101377 – GPL FTP wu-ftp bad file completion attempt (ftp.rules)
2101378 – GPL FTP wu-ftp bad file completion attempt with brace (ftp.rules)

Someone actually listened to me? It brings tears to my eyes :'(. Now go do the same for the rest of the rules ;D

EDIT: Updated the list and removed those rules to keep it clean

wetspark

While I don't have anything to do with MS, I have done some work on infrastructure for such probe systems for other companies. They are all "Whitehat" but I don't know that I agree with all of it. MS has a significant whitehat probe deployment and they often make the news with botnet take downs due to it.

Realistically, if there was a way to get a consumer friendly version of what's explained in this thread in a box folks could just install at home, the probe networks would have nothing to do. I have wondered if a decent ISP couldn't make a fair profit by deploying managed consumer firewalls based on stuff like this. They all want to give away anti-virus but that's a crap bandaide that has little real impact.

Then I talk to those that have MACs or something and think they are all secure because those companies told them they were since they don't run Windows.

Back on topic. I hope to spend some more quality time with the 2 release and certainly will be following this thread and maybe offering more than peanut gallery commentary.

@wetspark:

While I don't have anything to do with MS, I have done some work on infrastructure for such probe systems for other companies. They are all "Whitehat" but I don't know that I agree with all of it. MS has a significant whitehat probe deployment and they often make the news with botnet take downs due to it.

Realistically, if there was a way to get a consumer friendly version of what's explained in this thread in a box folks could just install at home, the probe networks would have nothing to do. I have wondered if a decent ISP couldn't make a fair profit by deploying managed consumer firewalls based on stuff like this. They all want to give away anti-virus but that's a crap bandaide that has little real impact.

Then I talk to those that have MACs or something and think they are all secure because those companies told them they were since they don't run Windows.

Back on topic. I hope to spend some more quality time with the 2 release and certainly will be following this thread and maybe offering more than peanut gallery commentary.

Whitehat has its limits, as far as I'm concerned. For example you don't go around scanning every single webserver out there if its vulnerable to a certain exploit. There is nothing whitehat about that. Would you want a stalker documenting your daily life in public for the purpose of "observing how people live their daily lives"? With detailed public notes about when you come home and leave for work? No, what you would want is that your neighbor called you up and said that car with number plates X has parked in front of your house, people got out and had a look peeking through windows taking notes, then took a couple of pictures before getting in the car and going away.

The traffic we've seen from Microsoft is remotely scanning for RDP, among other more "interesting" stuff (eg. privileged>privileged traffic, technically bad traffic). Other than to exploit a vulnerability in RDP, there is absolutely no reason to remotely scan for that. What, is someone trying to tell me Microsoft wants to know how many are running RDP? Or will they contact me if they find I am running RDP on the Internet and tell me I'm a bad boy and slap me on the wrist? Come no guys, defending the moon landing is one thing, defending observed traffic coming from a (barring NSA ties) respectable corporation is another.

I'm sure those hosts have nothing to do with Microsoft and due to certain cosmic events have ended up in one of Microsoft's subnets. I also understand that nobody on the Internet has observed this traffic originating from them, and no person (dead or alive) has alerted them to it. In the end, it's likely due to a number of misconfigured hosts. There is also the high likelyhood that it's spoofed traffic and the attacker has chosen that particular subnet, without an endorsment, support and/or otherwise help from Microsoft Corporation and/or any of its partners.

cough

Whitehat, spoofed or otherwise, if I don't have a use for that particular subnet (no services I need, no services they need) it's getting blocked. And the best thing is I didn't do it. The machine told me to :D

wetspark

I wasn't defending the activity, just mentioning that I'm aware that certain probe networks and infrastructures exists. The stuff you've seen from MS subnets is nearly certain to not be from their probe systems. All of these that I've had experience with are buried in a ton of shell companies and fake registrations to hide the originating sources.

I'd guess what you've seen is either due to hosted/cloud systems, spoofed IPs, or even compromised systems within their networks.

In that case they are violating IANA and/or regional registry IP assignment rules. The subnet was DA (Directly Allocated) to Microsoft. That means that ONLY Microsoft can use that subnet, NOT assign it to their customers. For further reassignments, the IPs need to be AP (Assigned Portable). Can I have that IP now that they have a reason to take it away from Microsoft? :P Pretty please with cherry on top?

wetspark

Good luck with that. I've seen large chunks of IPv4 rotting away in allocations for a long time and no one has ever been abel to pry them lose.

As far as I can tell, there have been recent cases where regionals did exactly that. They reclaimed wasted IPv4 space, since it's far easier to waste a couple of years legally pursuing a decision in a court to reclaim them, than to go ahead and enable IPv6. Not saying that I'll reclaim IPs from Microsoft, I'm saying that it's not impossible to do it.

wetspark

I'm ready for IPv4 to die really. I took my first IPv6 class in the late 90s and I've been pushing for adoption every since. It's been my experience that FUD has prevented more IPv6 than any technical reason.

@wetspark:

I'm ready for IPv4 to die really. I took my first IPv6 class in the late 90s and I've been pushing for adoption every since. It's been my experience that FUD has prevented more IPv6 than any technical reason.

What, no +1000 button on this forum? ;)

Exactly. Personally I'm in a position where I'm dealing with the provider side of things. If I don't enable it, ISPs have no reason to enable it. I enable IPv6 for all my clients (where applicable) so at least I'm doing my part for it. It's exactly like you said, there is no technical reason not to enable it, it's just 60 year old sysadmins have a fear for large numbers. It's exactly the same thing as IPv4, but with more characters in an address, nothing more.

Queue the "but what about ISP routers not supporting IPv6, switches, ETHERNET CABLES NOT SUPPORTING IPv6!" crowd. Every single device out there in production use that is lvl2 or lower, can handle IPv6. If anyone disagrees, please file your resignation first thing tomorrow morning. There are a lot of new people out there just dieing to get your job.

wetspark

I do provider infrastructure and lots of them are still on the "Wait and see" fence for IPv6. I've shown them how easy it is to run dual stack and they still just don't get it. I'm a gray beard myself and numbers are my friends. :)