PfBlocker

LinuxTracker

I'd like to offer up a copy of the Spam List I've been compiling for over a year.
I began compiling it for use in IPBlocklist (used with Country IP Blocks). It's now optimized for pfBlocker.

Here's the details/disclaimer:

505 CIDRs as of today.
Based on spams received at the mail servers I care for.
It's focused on US Spammers but includes CIDRs from some countries I couldn't block outright. Non-US CIDRs are noted.
3a) A few countries I eventually gave up on and just country blocked outright (ie Poland, Peru); so there may be some inconsistency.
I converted it to CIDR format 2 weeks ago. That took a long time (orig PG ranges were 1-254).
For each spam IP, I carefully examined it's host to determine the appropriate range.
5a) Criteria includes bot spams, dynIP ranges and scummy hosting companies.
5b) For a single IP, it may take 15+ minutes of careful research before I can decide what range to block.
I generally do 2-4 update sessions every month.
Use At Your Own Risk. I'd review it first for possible editing, if I were you.

I've broken the list up into 3 because it became unwieldy.
I've recently broken off corporate spam (ie: Linkedin, Constant Contact, exacttarget) into a fourth list. I prob still need to shift some IPs into it.

I thought Pastebin (Private link - 1 Month expiration) would be the most transparent option for publishing this.

SpamIPs_0-69 http://pastebin.com/MTds2fik
SpamIPs_70-179 http://pastebin.com/w0ZDtMym
SpamIPs_180-255 http://pastebin.com/QPi4PtMN
CorporateSpam http://pastebin.com/95xvHnk9

MODS: If this violates forum protocol, please delete the post and forgive me.
If a mod wants the constantly updated live url (.gz format), please PM me.
The update URL is under my personal domain so I can't otherwise distribute it. Sorry.

Thanks.

edit: added screencap - 8 Hours of spam hits - domain w/ ~10 email accounts.

pfCustomSpamList = 3 SpamIP lists above.

squirreldip

First off fantastic package!

I'm using pfSense as a VM in ESXi 5.0 (Host is Xeon X3320 with 8Gb RAM - VM is 2Gb RAM and 20Gb Storage). This VM is likely overkill but didn't want issues with RAM or disk space (I'm new to pfSense).

Two problems:

1. Everything works well using smaller Lists - Currently using Level 3 from I-BlockList (about 35,000 CIDRs). When I try Level 1 (over 400,000) I get no system errors but it won't block and listing the pfBlocker tables produces a blank page (after a fairly long wait). Also using Level 1 pfSense will not block the 'bad' IPs (and is very slow).

Is anyone else experiencing this issue? I've read through this entire thread but haven't seen a resolution… Is this something that is a known issue and will be fixed in future release? If not, is the fix a manual edit?

2. Not really a problem but an annoyance... This may be by design but when pfBlocker is updated it will move it's automatically created rules to the top of their respective lists. In order to bypass certain IPs I've created rules that pass the IPs and place these manual rules above automatic rule. I've created a custom internal alias for IPs I do not want to protect and another custom alias for external IPs that I need connection to. As long as these rules are above the pfBlocker rule everything is good - any time I reset/update pfBlocker and I have to manually place the auto rule back where I want it.

Could a possible update be to leave the rule where it sits if it already exists (and place it at the top if it's a new rule)?

Cheers and Thanks!

fsavoir

Thanks a lot. It's really generous from you for sharing that….
Really appreciated.

All the best,

Fred

squirreldip

Okay - Answered my own for '2.' above:

Redefine the list to be alias only and then manually add rules. (guess I had an extended blonde moment…)

'1.' still has issue even if it's alias only.

marcelloc

@squirreldip:

1. Everything works well using smaller Lists - Currently using Level 3 from I-BlockList (about 35,000 CIDRs). When I try Level 1 (over 400,000) I get no system errors but it won't block and listing the pfBlocker tables produces a blank page (after a fairly long wait). Also using Level 1 pfSense will not block the 'bad' IPs (and is very slow).

I've read through this entire thread but haven't seen a resolution…

Did you saw posts about memory limit?

read again page 21 of this topic and see if it helps

There is also a wiki contributed by XIII

http://doc.pfsense.org/index.php/Pfblocker

@squirreldip:

2. Not really a problem but an annoyance… This may be by design but when pfBlocker is updated it will move it's automatically created rules to the top of their respective lists. In order to bypass certain IPs I've created rules that pass the IPs and place these manual rules above automatic rule. I've created a custom internal alias for IPs I do not want to protect and another custom alias for external IPs that I need connection to. As long as these rules are above the pfBlocker rule everything is good - any time I reset/update pfBlocker and I have to manually place the auto rule back where I want it.

Okay - Answered my own for '2.' above:

One way you found by yourself, the other way is to create another list on pfBlocker, paste your whitelisted ips on custom field and define action as Allow inbound or Allow outbound.

marcelloc

@LinuxTracker:

I'd like to offer up a copy of the Spam List I've been compiling for over a year.
I began compiling it for use in IPBlocklist (used with Country IP Blocks). It's now optimized for pfBlocker.

I thought Pastebin (Private link - 1 Month expiration) would be the most transparent option for publishing this.

Nice contribution, thanks. :)

If you want, I can publish this list for you.

You can send me in pv from time to time.

squirreldip

@marcelloc:

Did you saw posts about memory limit?

Thanks for the post - I'll give it a shot when I get home tonight.

Is this fix something that will eventually be added to pfBlocker? Or fixed in pfSense? Will this fix have to be reapplied if pfSense is updated (obviously yes for a new install)?

If not then maybe something should be added to the wiki speaking to list limits and possibly something pointing to a "use at your own risk" fix.

taryezveb

@squirreldip:

Is this fix something that will eventually be added to pfBlocker? Or fixed in pfSense? Will this fix have to be reapplied if pfSense is updated (obviously yes for a new install)?

marcelloc, said yes for x64:
@marcelloc:

The ini_set("memory_limit", "250M") will be merged to next release and used when x64 instalation is detected.

When pfBlocker package is updated. All that should be needed is to reinstall pfBlocker package.

squirreldip

@squirreldip:

@marcelloc:

Did you saw posts about memory limit?

Thanks for the post - I'll give it a shot when I get home tonight.

Is this fix something that will eventually be added to pfBlocker? Or fixed in pfSense? Will this fix have to be reapplied if pfSense is updated (obviously yes for a new install)?

If not then maybe something should be added to the wiki speaking to list limits and possibly something pointing to a "use at your own risk" fix.

I made the changes from Page 21 and tested: I found an IP in the list that returned a ping. When using Level 3 list the blocker worked (would not return ping) but doing the same with the Level 1 list did not work (the ping returned).

I also increased my "Firewall Maximum Table Entries" from 1,000,000 to 100,000,000 and still no luck.

marcelloc

@squirreldip:

I found an IP in the list that returned a ping. When using Level 3 list the blocker worked (would not return ping) but doing the same with the Level 1 list did not work (the ping returned).

what action did you choose to level1 list and what ip did you tested?

marcelloc

@LinuxTracker:

SpamIPs_0-69 http://pastebin.com/MTds2fik
SpamIPs_70-179 http://pastebin.com/w0ZDtMym
SpamIPs_180-255 http://pastebin.com/QPi4PtMN
CorporateSpam http://pastebin.com/95xvHnk9

I've published your lists on my packages repo

http://e-sac.siteseguro.ws/pfBlocker/lists/

Thanks again LinuxTracker for your collaboration.

Note:

For those who will use these lists, config update to at least once a day as it's manually uploaded.

tommyboy180

@LinuxTracker:

I'd like to offer up a copy of the Spam List I've been compiling for over a year.
I began compiling it for use in IPBlocklist (used with Country IP Blocks). It's now optimized for pfBlocker.

Here's the details/disclaimer:

505 CIDRs as of today.

Based on spams received at the mail servers I care for.

It's focused on US Spammers but includes CIDRs from some countries I couldn't block outright. Non-US CIDRs are noted.
3a) A few countries I eventually gave up on and just country blocked outright (ie Poland, Peru); so there may be some inconsistency.

I converted it to CIDR format 2 weeks ago. That took a long time (orig PG ranges were 1-254).

For each spam IP, I carefully examined it's host to determine the appropriate range.
5a) Criteria includes bot spams, dynIP ranges and scummy hosting companies.
5b) For a single IP, it may take 15+ minutes of careful research before I can decide what range to block.

I generally do 2-4 update sessions every month.

Use At Your Own Risk. I'd review it first for possible editing, if I were you.

I've broken the list up into 3 because it became unwieldy.
I've recently broken off corporate spam (ie: Linkedin, Constant Contact, exacttarget) into a fourth list. I prob still need to shift some IPs into it.

I thought Pastebin (Private link - 1 Month expiration) would be the most transparent option for publishing this.

SpamIPs_0-69 http://pastebin.com/MTds2fik
SpamIPs_70-179 http://pastebin.com/w0ZDtMym
SpamIPs_180-255 http://pastebin.com/QPi4PtMN
CorporateSpam http://pastebin.com/95xvHnk9

MODS: If this violates forum protocol, please delete the post and forgive me.
If a mod wants the constantly updated live url (.gz format), please PM me.
The update URL is under my personal domain so I can't otherwise distribute it. Sorry.

Thanks.

edit: added screencap - 8 Hours of spam hits - domain w/ ~10 email accounts.

pfCustomSpamList = 3 SpamIP lists above.

Great SPAM list!

LinuxTracker

@tommyboy180:

Great SPAM list!

Thank you. I really owe you guys. I'm not in a position to pay back properly, but I'm glad to contribute something.

I put a lot of time into keeping the list up to date. It's rewarding to know a wider audience can make use of it.

I have this one email account I keep alive as a honeypot.
The former employee must have left their address everywhere, seems like every spammer in the US hits on it.

This is 12p-9p today on the same server.

After 5p the numbers ramped way up.
(I think 70%-80% were SMTP attempts, so I adjusted the rules tonight. My future numbers will all be SMTP.)

And again - Thanks.
Especially to you TB180, because I was well served by CountryBlock and IPBlocklist for 18 months.
But also to marcelloc, JimP and other devs who've spent scads of their own time developing an open Enterprise Class Security Suite; that I couldn't possibly afford.

LinuxTracker

First up:
I noticed a couple of interesting things happen at the top of the hour.

During the pfBlocker list update; my filters drop offline for a few seconds.
There's often a brief spike in spam activity at the top of the hour. Just from certain hard core spammers (ie: whoever is at 91.205.234.240-91.205.234.245)
That spike lasts from about a minute before the hour till a minute after.

The result is I'll have 2-8 spams get through.

Any changes to the lists in pfBlocker also drops my filters for 3sec-5sec.
It happens at the change, not when I navigate to General Tab and press save.

Notes:
The box is a P4 w/ HT, 1GB and an SATA drive. It runs pfBlocker, squid/squidGuard and Unbound.
It runs really well. RAM use averages 40% and CPU is mostly under 5%.

I did put in NTOP last night. It seemed to weigh heavy on the system, so I uninstalled it tonight.
I'll see if that makes the difference.

Secondly - about the Level1 Blocklist:
After pfBlocker converts it and loads into the table, it seems overly huge to me.
I've thought about streamlining the post-conversion CIDR ranges.
But doing that manually would be tedious. I also wouldn't know how to update it afterward.
I'm not sure anything can be done, but I thought I'd toss that out there.

And yes. I know my P4 is old and slow.
However:
To me, all code Must Be Efficient. It's never the hardware's fault; till it is.

squirreldip

@marcelloc:

@squirreldip:

I found an IP in the list that returned a ping. When using Level 3 list the blocker worked (would not return ping) but doing the same with the Level 1 list did not work (the ping returned).

what action did you choose to level1 list and what ip did you tested?

Thanks for your help!

I've backed up my VM and re-installed a fresh copy - modified according to your posts. The large lists now block.

Interesting is that trying to list the list the pfBlocker table still replies a blank page.

marcelloc

@squirreldip:

Interesting is that trying to list the list the pfBlocker table still replies a blank page.

The diagnostics -> table does not apply the 256 MB php memory limit. You may need to apply the DO AT YOUR OWN RISK patch there too.

marcelloc

@LinuxTracker:

During the pfBlocker list update; my filters drop offline for a few seconds.

There's often a brief spike in spam activity at the top of the hour. Just from certain hard core spammers (ie: whoever is at 91.205.234.240-91.205.234.245)
That spike lasts from about a minute before the hour till a minute after.

The result is I'll have 2-8 spams get through.

Any changes to the lists in pfBlocker also drops my filters for 3sec-5sec.
It happens at the change, not when I navigate to General Tab and press save.

Both are related to same thing.

To prevent max table memory limit errors, pflBlocker cleans it's tables before reapply, that's why you have some seconds without "protection"

If you choose to update list every hour, then every hour you will have some seconds off.

biogoon

I think I've found an issue with the password field in the XMLRPC Sync pane. The field seems to truncate some passwords as it does not escape at least a subset of symbols. As a result, configs don't get synced properly across systems.

Thanks again for such a great package!

LinuxTracker

@marcelloc:

To prevent max table memory limit errors, pflBlocker cleans it's tables before reapply, that's why you have some seconds without "protection"

If you choose to update list every hour, then every hour you will have some seconds off.

OK. That's helpful to know. I'll normally have the lists set for daily updates. It looks like that happens at 23:00.
I'll write a script to cycle the mail server service during that time.

Thanks.

marcelloc

@biogoon:

I think I've found an issue with the password field in the XMLRPC Sync pane. The field seems to truncate some passwords as it does not escape at least a subset of symbols. As a result, configs don't get synced properly across systems.

Thanks again for such a great package!

Sync problem you did was related to old package's version or password symbols?