RFC (make up a number not in use) - Blueprint for setting up snort + pfblocker
-
@jflsakfja:
Saw you writing somewhere that you should reduce your lenghty texts. Please don't, I enjoy reading your curious thoughts very much ;D ;D And as they say, a laugh a day keeps the things away.
-
Hey, thanks for replying!
You really should condense everything you said, and make a tutorial out of it. You explain extrelemy well and it seems that you know what you are talking about, which are two good qualities to have at the same time ;)
That means you did not select alias only in the list.
FYI, I did. I however uninstalled pfblocker 2 or 3 times in the last few weeks and upgraded pfsense to latest release in between. It appeatrs to me as if my install was more or less stuck with the old rules created by the initial install…
If I remember chronologically:
I installed pfblocker
Added my custom lists
Selectrd Deny Both in ALL lists
Confirmed pfblocker had created rules based on my selections and lists
...
Uninstalled pfblocker
Upgraded pfsense
Reinstalled pfblocker
Re-created my lists from your initital post on this thread
Selected Alias only for ALL listsThen I discovered that it had re-created rules on the WAN & LAN IF while also creating aliases in the Aliases menu..
This is where I am as of now.
Question: Would it be safe to dlete all the rules (except the ones automatically created by pfsense) and re-create them based on your instructions?
Also I wanted to tell you: Something's wrong in the lists you are suggesting (at least from my geographical location) or maybe its the fact that I am blocking entire countries but immediately after I enabled pfblocker, my internet ceased immediately to function, I cannot even connect to Google.com from Canada.
That bothers me a lot to say the least. I thought (and really honestly thought) that the whole point of pfblocker was to allow you to block countries by IP ranges.... Apparently, its not the case.
Tonight, I will try to make this work.
-
OK as I said, I tried to make it work.
I did a lot of house cleaning… Removed pretty much every custom rules, made sure EVERYTHING in pfblocker was set to "Alias Only" (and you were right jflsakfja, some of them were still at Deny both).. My mistake, I guess trying to make this stuff work while being tired is useless...
Then pfblocker created the Aliases properly, and I added the custom firewall rules to WAN & LAN as you suggested (WAN with Block, LAN with reject).
And I applied the new settings. Waited for a few minutes (that pfsense box is slow like sh**) and then I picked a few random IP's in the custom pfblocker lists that are supposedly being blocked (or rejected) by my custom firewall rules. I tried pinging these IP's from my LAN, and I can successfully ping all of them.
Shouldn't I normally find that ping gets rejected by the firewall and issue something like 100% packet lost????
I dont think its working after all.
What have I done wrong?!
-
@lpallard:
OK as I said, I tried to make it work.
I did a lot of house cleaning… Removed pretty much every custom rules, made sure EVERYTHING in pfblocker was set to "Alias Only" (and you were right jflsakfja, some of them were still at Deny both).. My mistake, I guess trying to make this stuff work while being tired is useless...
Then pfblocker created the Aliases properly, and I added the custom firewall rules to WAN & LAN as you suggested (WAN with Block, LAN with reject).
And I applied the new settings. Waited for a few minutes (that pfsense box is slow like sh**) and then I picked a few random IP's in the custom pfblocker lists that are supposedly being blocked (or rejected) by my custom firewall rules. I tried pinging these IP's from my LAN, and I can successfully ping all of them.
Shouldn't I normally find that ping gets rejected by the firewall and issue something like 100% packet lost????
I dont think its working after all.
What have I done wrong?!
Your story is my story; I must have tried at least 100 times to get pfBlocker to work doing the same as you did; for me also it has never blocked any IP's at all, even though the tables are filled with IP's. There's something buggy somewhere I guess.
-
Hollander: It's just that sometimes I feel I'm going off in lengthy tangents and sometimes (most times actually) people miss my point.
lpallard: are you sure the alias selected in the rules you created starts with pfBlocker?
For example, I create the list BadPeers and put those lists in it, alias only for the list BadPeers. pfBlocker then creates that alias, but appends (prepends?) pfBlocker in front of it. So in pfsense's rules I have to select pfBlockerBadPeers as the alias, if I type BadPeers, then it will not function, since that alias doesn't exist.
Go into your firewall rules, and hover over the alias. WAN side, source should be your alias. Hover over it with the mouse (it would be akward to hover yourself over a screen :o) and see if a tooltip pops up with IPs in it.
A newly created blocking(or rejecting) rule could allow packets to flow through it, unless you clear the firewall states (Diagnostics>States>Reset) but I don't think this is the case since you selected random IPs to ping.If it allows traffic then the rule is not set up properly, pfblocker is currently updating the lists (so the table is empty, just to make sure go Diagnostics>Tables>select the pfblocker table (could take a while) and check that it is populated), source/destination/protocol do not match. Or a horrific bug exists somewhere, but I haven't found anything to support this.
I'll attach a couple of screenshots to help explanation.
-
Hi there! I have found several anomalies in my setup.
First of all, yes, the aliases are all properly created, and I also created the rule properly. As a matter of fact, I am also selecting the pfblocker**** alias in the rules as you said (see screenshots).
The story got uglier when I went to the Diagnositc > Table:
1. All tables are populated except the iblocklist custom pfblocker list. I dont know why. I went to the list settings in pfblocker, and tried removing a few list entries that I had suspicions about their validity… Then I hit Save but my router takes ages to return to the homepage, sometimes completely dies, sometimes, freezes, its a POS.. If you (or someone else feels like helping me pick up a new box, plz feel free to weight in on my thread (http://forum.pfsense.org/index.php/topic,70534.msg384951.html#msg384951)
2. There is a bug with my "alienvault" list. When I select it in the Diagnostics > Tables, the pages disappears and I end up on a totally blank page, kinda like if the pfsense web server died..
So in a nutshell:
-list alienvault doesnt display in Diagnostics > Tables
-iblocklist is empty
-iblocklist takes 10-20 minutes+ to save when modified, usually requires a hard firewall reset. Quite frankly, every time I modify tables, rules, aliases or whatever else that touches the firewall core, the entire box dies. Posting this reply took me over an hour for Heaven's Sake!!!! Sorry... I had to reboot (hard reset by unplugging the power cord) the box twice.I have seen a LOT of messages such as these in the "Filter Reload " page: Most of the time, they hang up indefinitely....
Processing early nat rules for package /usr/local/pkg/squid.inc...
Processing early nat rules for package /usr/local/pkg/tftp.inc...
Then the pfsense box stops responding from the web browser, the internet goes down, the phone goes down, the SSH session no longer works
...Do I have a more general firewall issue you figure??
Thanks!!
-
Blank pages if I remember correctly (this is a non-transferable license granted to the reader of this post, to correct me) is php running out of memory.
The aliases look ok, rules look ok, but the memory usage on that page is insane! I'm using 25% with all snort rules, those lists, on 2GB of RAM. Just saw squid on the services, never used it, can't comment on that. (this is also a non-transferable license granted to the reader of this post, to correct me)If the router dies at random times,freezes etc.. etc.. then something is definately wrong. I'm suspecting a hardware issue. Can you check Diagnostics>SMART status > Information/Tests select Attributes, and make sure that:
Reallocated sector count = 0
Current pending sector = 0
Offline Uncorrectable = 0 ?If that comes out positive (no other value than those) check your RAM with memtest86.
If that checks out ok, check PSU voltages (NOT responsible for anything etc…etc....)
If that checks out ok, visually inspect the motherboard for blown caps (capacitors,you'll know how they look when their top has a hole with metal bulging outwards).
An atom box is not that old, shouldn't have given up the spirit yet.
The lists taking a long time to populate means that the list is huge and is being downloaded, the download was OK but the box runs out of memory populating them, or the download failed.
Re-reading your reply, I'm now thinking it's more a out of memory problem than anything else. But you could perform the tests I showed above, just to make sure. Is that 80something% with the lists populated? -
is php running out of memory.
Make sense with so much RAM usage…
Snort is using about 80% of the RAM.... Before snort is started, my memory usage is around 10% (with Squid, SG, HAVP, etc) all running then I start Snort and it goes up to 85-88%...
SMART returns no errors, I had already checked this one often.
RAM could be faulty, never checked it. Out of 4 sticks, 2 were brand new at the time I purchased this Foxconn barebone machine, then I added another older stick, and finally another stick salvaged from an old machine... RAM could (and probably) is faulty
Lets talk about the PSU shall we ;)
About 6 months after I bought the machine, the PSU fan started to make faint grinding noises. At that time I thought it was out of alignment due to wear (since running 24/7) and because of the PSU's quality.. I decided to cut off the steel mesh protecting the PSU's fan thinking it was hitting it. Didnt't help at all... Fast forward 1 year later, the grinding noise is so intense it sounds almost like a real grinder.
Then one night I was woken up by a strong burning electronic smell... The fan had stopped turning completely and the PSU was probably in the 200degrees range (seriously I burnt my finger touching the PSU case..
No doubt the PSU is dying and I am borrowing time here...
Also, I need a rackmount enclosure... Quite frankly, I'd keep the CPU and Mobo and would only add more RAM, put it in a 1U enclosure and change the PSU and be done with it, but this Atom platform doesnt allow adding more RAM so is it really worthwhile to spend money to change the PSU and be stuck with this CPU/mobo?
Other than that, mobo seems OK. I also blame the Realtek NIC for some anomalies (cant restore WAN public IP after Power outage, Squid gets hung up on the WAN interface and requires rebooting the box)... You can search for my name on this forum, you'll see how much I had problems to make this thing run smoothly or reliably.
To a certain extent, I wonder how much the Realtek NIC and RAM are responsible for my misadventures!?
How can you use less than 2GB RAM with all the rules you suggested on this thread!?
Oh & I am forgeting! THis box didnt come with a dual NIC (surprise surprise!) and has only a PCI slot, so I added a cheap second hand PCI ethernet adapter (cant remember the brand/model). Maybe this compoennt is also defective..
I think the main thing to remember is that I built this box when I had no idea what I was doing and I was looking only at Watts (hence the choice for an Atom). Now with my better knowledge of pfsense and hardware reliability, if I had to restart fresh (which is what I am thinking to do), I'd start with making sure the box comes with 2 Intel NICs, supports more than 4GB RAM, and has at least 1 PCIE for future expansion, and finally has a reliable 24/7 rated PSU..
This box has none of these features.
-
When I try to load this list: http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt I get a php error from pfblocker.inc line 262. Increasing max table entries doesn't seem to help. Is there another setting I need to tweak?
Did anyone get this list to build? All the others are fine, this one just won't take. Ideas? fixes?
Thanks,
Rick -
When I try to load this list: http://doc.emergingthreats.net/pub/Main/RussianBusinessNetwork/RussianBusinessNetworkIPs.txt I get a php error from pfblocker.inc line 262. Increasing max table entries doesn't seem to help. Is there another setting I need to tweak?
Did anyone get this list to build? All the others are fine, this one just won't take. Ideas? fixes?
Thanks,
RickYou did select the txt Format and not the gz Format?
-
You did select the txt Format and not the gz Format?
Yep,
deleted it, rebuilt it. Didn't trust the <url>copy so I went to the url, saw the list, copied the known working url and still can't get this list to build.I have 7 others, all done the same way… some GZ, some TXT, this one just does not want to populate.
Rick</url>
-
lpallard: saw your other thread about hardware recommendations. Like I said on this thread, your problems sound like a hardware problem. I'll be replying to that thread for hardware, to keep this one somewhat on topic.Dunno when, since I'm really busy these days, but I promise I will.
As for the memory usage, I'm using right now…24% of 2026 MB, which comes out to...486.24. So, snort with all rules enabled CAN run on a machine with less than a gig of ram. Highest I've seen it rise was to 30% (don't forget the machine does all the other pfsense duties). Make sure you select AC-BNFA-NQ under snort interface settings> Detection performance > Search method. I could write a 500 page essay on why, but to Hollander's disappointment, I will not :D. What did we say about correcting me? (rhetorical question)
Ramosel: see if this helps: http://forum.pfsense.org/index.php?topic=42543.300. Can't remember if I had to do that on my systems, but it was definately not needed going 2.0 to 2.1. The list does download on mine (just re-checked, since all lists are in a single table, took 5 random IPs from that list and searched for them in the table, they were there.
-
@jflsakfja:
Ramosel: see if this helps: http://forum.pfsense.org/index.php?topic=42543.300. Can't remember if I had to do that on my systems, but it was definately not needed going 2.0 to 2.1. The list does download on mine (just re-checked, since all lists are in a single table, took 5 random IPs from that list and searched for them in the table, they were there.
Thanks, saw that earlier and left alone as I am running a 2.1 64bit pfsense. I've built all the pfblocker lists using the links your provided at the end of the first run. All built successfully except this RBN one. I don't understand it?
Sent you PM, don't know if you wanted to respond or not??
Rick
-
The bigger the problem, the bigger the hammer? :o
Replied to your PM.
-
As was privately requested, here's detailed instructions on how to set up an alias only list and use that in your rules.
Setting up the alias list
First you need to create the alias, and then populate it with IPs. This process is handled by pfblocker. Install the pfblocker package, go to Firewall>Pfblocker.
General tab:
Enable: ticked
Logging: ticked
Inbound interfaces: $your_wan_interface ctrl + click for multi selection
Inbound deny action: block
Outbound interfaces: $your_lan_dmz_interfaces, ctrl + click for multi selection
Outbound deny action: block
Don't actually know if this is needed since we will be creating an alias list, but doesn't hurt to try it.
Lists tab:
Press the + button
Alias name: your alias name CARE!!! the alias will be pfBlockerYourName
Description: blah blah
lists: add the lists I posted making sure you select the proper type (txt or gz). To add more, there's a + button below the box you enter the lists.
IMPORTANT! List action: Alias only
Update frequency: Once a day
scroll down, hit save, wait for the lists to be populated. A coffee break is recommended.Firewall rule with your alias list:
from a previous post (self-quoting will be the new rage in 2014):"You need a block rule on WAN:
Action: Block
Disabled: NOT ticked
Interface: $wan_interface (select your interface)
TCP/IP Version: VERSION 4!!!! Remember, IPs in those lists are v4 IPs. Don't go creating another rule for v6.
Protocol: Any
Source: not NOT ticked (I hope that makes sence).
In the box under source start typing pfblocker. A tooltip should pop up with an entry in the form pfBlockerCustomAliasName. Select that
Destination; Any
Log: NOT ticked. (there are occasions when you do need to log attempted connections from banned sources, general public use is not one of them)
Description: A description to help you identify this rule.
Hit save.No go into your LAN (or DMZ,OPT1,SATELLITECONNECTIONTOTHEMOON1 etc..etc..) interface and repeat the above, this time with:
Action: Reject (I don't have to go into why you shouldn't wait for timeouts for the LAN side, you need your applications (browser, remote control exploit etc…etc....) to know they can't connect to a destination immediately. Hitting a bad website will get you an immediate couldn't open the website for example). Oh wait, I did go into.....)
Interface: $lan_interface"The thing I missed in that post was under WAN source (or LAN destination), right bellow the NOT tickbox is a dropdown menu. MAKE SURE YOU SELECT "Single host or alias" THEN start typing pfblocker…. it should show a popup with all the pfblocker aliases, select the one you created above
I posted screenshots of the rules, and pfblocker lists a couple posts up, please refer to them.
-
@jflsakfja:
The bigger the problem, the bigger the hammer? :o
Replied to your PM.
I'm about to hammer this thing…
I've moved my Firewall Max Tables and Firewall Max Table Entries to 10,000,000 per the discussion
I've built separate lists in pfBlocker for all the recommended lists and they all populate... Except the Russian Business Networks.
I've torn down all the separate list and used the single alias model and threw multiple lists together... they all populate except the Russian Business Networks.What am I missing? I've looked at my logs and saw nothing (not saying I looked in the right place).
Rick
-
OK, no responses and no luck fixing this. Something with this list and my system just don't jibe and I'm not the only one. Doing a search here, at least 2 other postings from folks having trouble with this specific list.
Did some research and it looks like this list is almost 2 years old anyway… (Feb 10, 2012)
http://doc.emergingthreats.net/bin/view/Main/RussianBusinessNetworkSo that being the case is is best to abandon this list and just use the "emerging-rbn.rules" from within Snort?
Rick
-
Abandoning the list is not better, since you want to block all packets coming from those sources, not let through until snort decides that it should block.
Have you tried posting in the pfblocker thread? Maybe the guys over there can help.Technically RBN was dismantled, but the "bad guys" are still on a large number of those IPs.
-
Just as an update; it appears the lists were updated last night. Blocking cloudflare and perfectly legitime sites. I've been creating exceptions (dump them in an alias and allow that one before the block rule) for 1 hour :-[
-
Sometimes the pfblocker lists I use are updated and block legitimate sites. The reason is that the host that website is on, could be compromised and start throwing scans here and there and sooner or later it's detected. From my point of view, that host shouldn't be allowed, but considered as a hostile compromised host. It's the host's admin responsibility to check how the compromise happened, why it happened, what was compromised, correct all those, then notify the list maintainers to request removal from those lists.
An example is a host that scanned one of the networks I'm responsible for. That scan was detected by my rules (which, to be completely honest account for about 90% of all hosts snort blocks). Investigating it shows that it's supposed to be a mail gateway host, but it's scanning for webservers. Interesting behaviour for a mail gateway. I could notify the server admins, but why bother? Last time I bothered was after some of my hosts were hit with persistent scans. The hosting company's response was:
"Thank you for your report.This client has assured us that they are performing benign research scans at a non-intrusive and non-destructive rate. We do not believe that they are in violation of any relevant laws or our terms of service at this time…." snipped
That's after I provided log snippets showing multiple scans to multiple hosts for well over half an hour.
Knowing most server admins come from the "hell bent to f*** up the user interface" company's environment, then I'm guessing one of 2 things will happen:- Not bother at all with their compromise
- Not bother with their compromise for the next 5 years, at which point their whole infrastructure will be updated.
There are the exceptions to server admins, the best of the best, coming from the Linux/Unix environment. In that case only one thing will happen: - Bother with it after 6 months.
If anyone reading this thinks I'm overly trigger happy, I'll have you know I identified 2 things that need fixing in the infrastructure of one of the 3 largest banks in my country. I notified their "IT department" which I'm sure it's manned by the absolute best server admins, with multiple PhDs in "Computer Security" and regularly being inspected by the world's top white hat hackers to find ways in. 2 years later and both things identified to them are still there. 1 of those is using password storing (hashing) technology from even before the abacus was invented, and the other is a simple 1 line fix in their web interface to download your statements, which as is, gives more details than I would be happy to give to my server admins, let alone the public (hint:it's IIS running on an unpatched version of windows).
Disclaimer: This is not a libelus comment with regards to server administrators worldwide. It's a riducule attempt to the idiots pretending to be server admins. And the idiots that hire them.
To recap, there's a reason those hosts ended up on the lists I use. It's the server admin's responsibility to determine why. Then there's the hosting companies that cover up for their clients. Personally, I don't want any traffic from said hosts.
