Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Alias Bulk loader and huge aliases

    Scheduled Pinned Locked Moved 2.0-RC Snapshot Feedback and Problems - RETIRED
    9 Posts 3 Posters 7.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      MrHorizontal
      last edited by

      This isn't really a major bug nor a feature, but a caveat I thought I'd point out.

      I have set up in a test pf2.0 installation all UK and US subnets, which amounts to 70,308 CIDRs. In order to import them I needed to create multiple aliases with 3,000 lines in each and multiple rules that effectively do the same thing. This is no bad thing, as probably having one alias with 70k CIDRs is probably a bit much. However I think the limitation of 3,000 lines is a ridiculously simple thing: lighttpd seems to have a a limit of 64k in the POST buffer that roughly equates to ~3,000 CIDR lines in the bulk importer, and that's the caveat I thought I'd point out.

      For the record, if pfSense allowed simple IP ranges, the number would have been reduced to 28,263 ranges using a lot less space.

      Also for the record, I am pretty amazed that pfSense can handle this amount of ranges in an alias and not really have a massive knock on effect in terms of RAM usage either. In fact the only thing I've noticed is the boot time when it says 'Bringing up firewall…..' takes about 5 seconds to load instead of near instant - I don't notice any other issue in performance, though I haven't yet created a situation where I really stress pfSense with a huge state table that connects to a vast array of IP addresses (ie like downloading a torrent), not would I expect this to be able to handle any serious routing in a core hub either, but for home usage it's fine.

      On a somewhat related note, GeoIP routing could be possibly supported in pfSense and simply referenced by the 2 letter ISO code. This is distinctly a 'nice to have', not a demand, but some Debian dude connected MaxMind's GeoIP db to iptables (http://www.debian-administration.org/articles/518), and GeoIP is already a package in the portstree.

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Breaking them up is necessary anyhow because pf won't accept an alias line that long. You can have aliases which reference other aliases in 2.0, so you could still have just one rule.

        e.g.

        cidralias0 … cidralias999

        And then:
        goodips alias which includes cidralias0 through cidralias999

        Then you just have one rule that uses goodips.

        I'm also trying to look into the feasibility of using persist files like bogons but I'm not sure that would be possible in an easy manner.

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • M
          MrHorizontal
          last edited by

          @jimp:

          You can have aliases which reference other aliases in 2.0, so you could still have just one rule.

          You can't do that with such large aliases though - I get the following errors:

          There were error(s) loading the rules: /tmp/rules.debug:644: cannot define table US_Networks_14: Cannot allocate memory/tmp/rules.debug:659:
          

          This is on a 1GB RAM / 1 CPU VM. As you can see though it's breaking on the 14th alias, so that means the limit for aliases must be around 3000*13 = 39,000 networks (ie probably 40k), even with aliases-of-aliases.

          So if you have huge aliases, you still need to create multiple rules. It's no biggie though it's 73000 network aliases for UK and 173000 aliases for the US - meaning I have to add 24 firewall rules instead of 2 - if it's efficient despite the additional labour required, I think it's fine.

          1 Reply Last reply Reply Quote 0
          • M
            MrHorizontal
            last edited by

            Another small point is on the firewall rules page where you get the summary table and an aliases has the underline and an AJAX popup showing the aliases, obviously it's taking a fair while to open that page - can you concatenate the contents of those popups to say the first 50 entries?

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              I thought that popup had a limit, but maybe when working with such a large alias it still has issues. In order to pick up the first <x>entries out of the list it probably still has to load the whole list. Not sure how PHP works with that.

              I've got an idea cooking though that may be a better fix.</x>

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • N
                nocer
                last edited by

                howdy,

                http://forum.pfsense.org/index.php/topic,23431.0.html

                like i said, persist file is needed for that many ranges. i gave up on using bulk importer unfortunately and modded filter.inc to create my own rules using <persist>directive to assign network range file(s), yes, it's just like for bogon. and create small script to update range file(s) which runs on cron so that your range file would always up-to-date so does pf rules.

                cheers,</persist>

                1 Reply Last reply Reply Quote 0
                • jimpJ
                  jimp Rebel Alliance Developer Netgate
                  last edited by

                  I checked in some code this afternoon to add a "url table" type alias. You give it a URL to the list of IPs/CIDRs and it will pull it down and put it into a persist table. I have a script checked in to periodically update the list, but I don't have it actually in a cron job yet.

                  You can't edit the list once it has been downloaded (not sure if that's something we'd want to support in the GUI yet or not) but I was able to load up the whole US CIDR list, all 38,000 entries of it.

                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                  Need help fast? Netgate Global Support!

                  Do not Chat/PM for help!

                  1 Reply Last reply Reply Quote 0
                  • M
                    MrHorizontal
                    last edited by

                    @jimp:

                    I checked in some code this afternoon to add a "url table" type alias. You give it a URL to the list of IPs/CIDRs and it will pull it down and put it into a persist table. I have a script checked in to periodically update the list, but I don't have it actually in a cron job yet.

                    You can't edit the list once it has been downloaded (not sure if that's something we'd want to support in the GUI yet or not) but I was able to load up the whole US CIDR list, all 38,000 entries of it.

                    This is wonderful! Works as intended using  http://www.countryipblocks.net/e_country_data/US_cidr.txt

                    Just one small thing - I understand the cron job isn't actually working yet, but since you're using the CIDR subnet mask dropdown as the Update Frequency (ie 0-32) - how would this translate to how often the list is refreshed?

                    1 Reply Last reply Reply Quote 0
                    • jimpJ
                      jimp Rebel Alliance Developer Netgate
                      last edited by

                      It's intended to be the # of days between refreshes.

                      The code will check the age of the file, if it's more than <x>days old, it downloads a new copy. (or it will once I put the cron job in)

                      For now to force an update just edit the alias and save it without making changes, that should always download a fresh copy.</x>

                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.