Alias Bulk loader and huge aliases
-
This isn't really a major bug nor a feature, but a caveat I thought I'd point out.
I have set up in a test pf2.0 installation all UK and US subnets, which amounts to 70,308 CIDRs. In order to import them I needed to create multiple aliases with 3,000 lines in each and multiple rules that effectively do the same thing. This is no bad thing, as probably having one alias with 70k CIDRs is probably a bit much. However I think the limitation of 3,000 lines is a ridiculously simple thing: lighttpd seems to have a a limit of 64k in the POST buffer that roughly equates to ~3,000 CIDR lines in the bulk importer, and that's the caveat I thought I'd point out.
For the record, if pfSense allowed simple IP ranges, the number would have been reduced to 28,263 ranges using a lot less space.
Also for the record, I am pretty amazed that pfSense can handle this amount of ranges in an alias and not really have a massive knock on effect in terms of RAM usage either. In fact the only thing I've noticed is the boot time when it says 'Bringing up firewall…..' takes about 5 seconds to load instead of near instant - I don't notice any other issue in performance, though I haven't yet created a situation where I really stress pfSense with a huge state table that connects to a vast array of IP addresses (ie like downloading a torrent), not would I expect this to be able to handle any serious routing in a core hub either, but for home usage it's fine.
On a somewhat related note, GeoIP routing could be possibly supported in pfSense and simply referenced by the 2 letter ISO code. This is distinctly a 'nice to have', not a demand, but some Debian dude connected MaxMind's GeoIP db to iptables (http://www.debian-administration.org/articles/518), and GeoIP is already a package in the portstree.
-
Breaking them up is necessary anyhow because pf won't accept an alias line that long. You can have aliases which reference other aliases in 2.0, so you could still have just one rule.
e.g.
cidralias0 … cidralias999
And then:
goodips alias which includes cidralias0 through cidralias999Then you just have one rule that uses goodips.
I'm also trying to look into the feasibility of using persist files like bogons but I'm not sure that would be possible in an easy manner.
-
You can have aliases which reference other aliases in 2.0, so you could still have just one rule.
You can't do that with such large aliases though - I get the following errors:
There were error(s) loading the rules: /tmp/rules.debug:644: cannot define table US_Networks_14: Cannot allocate memory/tmp/rules.debug:659:This is on a 1GB RAM / 1 CPU VM. As you can see though it's breaking on the 14th alias, so that means the limit for aliases must be around 3000*13 = 39,000 networks (ie probably 40k), even with aliases-of-aliases.
So if you have huge aliases, you still need to create multiple rules. It's no biggie though it's 73000 network aliases for UK and 173000 aliases for the US - meaning I have to add 24 firewall rules instead of 2 - if it's efficient despite the additional labour required, I think it's fine.
-
Another small point is on the firewall rules page where you get the summary table and an aliases has the underline and an AJAX popup showing the aliases, obviously it's taking a fair while to open that page - can you concatenate the contents of those popups to say the first 50 entries?
-
I thought that popup had a limit, but maybe when working with such a large alias it still has issues. In order to pick up the first <x>entries out of the list it probably still has to load the whole list. Not sure how PHP works with that.
I've got an idea cooking though that may be a better fix.</x>
-
howdy,
http://forum.pfsense.org/index.php/topic,23431.0.html
like i said, persist file is needed for that many ranges. i gave up on using bulk importer unfortunately and modded filter.inc to create my own rules using <persist>directive to assign network range file(s), yes, it's just like for bogon. and create small script to update range file(s) which runs on cron so that your range file would always up-to-date so does pf rules.
cheers,</persist>
-
I checked in some code this afternoon to add a "url table" type alias. You give it a URL to the list of IPs/CIDRs and it will pull it down and put it into a persist table. I have a script checked in to periodically update the list, but I don't have it actually in a cron job yet.
You can't edit the list once it has been downloaded (not sure if that's something we'd want to support in the GUI yet or not) but I was able to load up the whole US CIDR list, all 38,000 entries of it.
-
I checked in some code this afternoon to add a "url table" type alias. You give it a URL to the list of IPs/CIDRs and it will pull it down and put it into a persist table. I have a script checked in to periodically update the list, but I don't have it actually in a cron job yet.
You can't edit the list once it has been downloaded (not sure if that's something we'd want to support in the GUI yet or not) but I was able to load up the whole US CIDR list, all 38,000 entries of it.
This is wonderful! Works as intended using http://www.countryipblocks.net/e_country_data/US_cidr.txt
Just one small thing - I understand the cron job isn't actually working yet, but since you're using the CIDR subnet mask dropdown as the Update Frequency (ie 0-32) - how would this translate to how often the list is refreshed?
-
It's intended to be the # of days between refreshes.
The code will check the age of the file, if it's more than <x>days old, it downloads a new copy. (or it will once I put the cron job in)
For now to force an update just edit the alias and save it without making changes, that should always download a fresh copy.</x>
