Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Bogon Lists Mirror Outdated

    Scheduled Pinned Locked Moved Firewalling
    8 Posts 2 Posters 2.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      NOYB
      last edited by

      pfSense mirror IPv6 full bogons list hasn't been updated in nearly 2 months (Oct. 12th).

      mirror: http://files.pfsense.org/mirrors/fullbogons-ipv6.txt
      master: http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

      What is the expected update schedule / frequency?

      Also pfSense mirror does not have the IPv4 full bogons list.  Can it be added?

      mirror: http://files.pfsense.org/mirrors/fullbogons-ipv4.txt
      master: http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt

      Thanks

      1 Reply Last reply Reply Quote 0
      • C
        cmb
        last edited by

        We had to make an emergency switch to a backup server at another datacenter on that host, expecting it to be temporary but it's going to be a bit longer. New server didn't have the update scheduled. Fixed so it checks daily as the old server did, and I added the full v4 bogons here:
        http://files.pfsense.org/lists/fullbogons-ipv4.txt

        I also changed 2.1 so it now uses the fullbogons-ipv4.txt list instead of bogon-bn-nonagg.txt. If that proves not to break anything, I'll likely change the update script to write fullbogons-ipv4.txt to bogon-bn-nonagg.txt so existing installs switch to that list as well.

        1 Reply Last reply Reply Quote 0
        • N
          NOYB
          last edited by

          Thanks!

          What about the pfSense cron job update schedule.  Is once a month frequent enough for full bogons?  I've been running it at once a day.  Overkill?

          Once a month for a home environment might be okay, but a business might what a more frequent update cycle.

          Also I'd like to suggest / request not removing the comments from bogonsv6 list.

          
          rc.update_bogons.sh: 58
          current: egrep -v "^#" /tmp/bogonsv6 > /etc/bogonsv6
          suguest: egrep -v "^\\" /tmp/bogonsv6 > /etc/bogonsv6
          
          
          1 Reply Last reply Reply Quote 0
          • C
            cmb
            last edited by

            That's why I haven't replaced the main bogons list every current install fetches, I'm not sure what a reasonable update frequency would be, and there's no changing that on 150,000+ existing installs that fetch it every month. Historically, back when the list we were using was actually changing, it took well over a month between when something was assigned, removed from the bogons list, and when it was actually in use. The fullbogon list may be significantly different from that. Ditto potentially for the v6 list.

            Updating it daily is almost certainly overkill unless Cymru lags on updating the list and removes something after it's in active use. Monthly may or may not be frequent enough.

            There are also bandwidth and server usage considerations on how frequent to make the updates, which we haven't really taken into account for 2.1 yet. The v6 list is vastly larger than the v4 list, and the fullbogon v4 list is over 400 times the size of the former v4 list. Multiply that by 150,000+, with growth of about 50% every year at current rates, and you have a big number. Historically the only scalability concern has been a web server that performs well under significant load (trying to scale Apache for that proved to be a serious pain, thttpd just works), but changing v4 to fullbogons and adding v6 is going to create bandwidth concerns where they never previously existed. Not a problem as long as we have good colo sponsors like NYI and bluegrass.net, but if we had to shift all that to the colo we're paying market rate for in Austin, it could bring our 95th percentile up enough to cost upwards of $1000/month more. We're already paying more for colo than the general donations the project receives.

            I contacted Team Cymru to see if they could provide any guidance on how quickly IP space is in use once removed from their v4 and v6 fullbogon lists. Hopefully they'll respond and provide some guidance, and we can take that into consideration for update frequency for 2.1.

            1 Reply Last reply Reply Quote 0
            • N
              NOYB
              last edited by

              It will be interesting to hear what Team Cymru has to say.

              Would a CDN (Content Delivery Network) service, such as Akamai, be in order to distribute the load?

              1 Reply Last reply Reply Quote 0
              • C
                cmb
                last edited by

                It'll never be the kind of scale for something like Akamai (and that would cost more). Round robin DNS load balancing is the next step.

                1 Reply Last reply Reply Quote 0
                • N
                  NOYB
                  last edited by

                  Growth rates typically slow as demand is filled and becomes saturated.

                  Also as IPv6 becomes more widely implemented the list should shrink.  Though with such a large address space the percentage change will probably be slight for quite some time.

                  
                  	Bytes			Bits		
                               76,527	              612,216		IPv4 Bogons List Size
                           1,011,532	          8,092,256		IPv6 Bogons List Size
                           1,088,059	          8,704,472		IPv4+6 Bogons List Size
                  
                  						 500,000		~# pfSense Instalations + 3 Years of 50% Growth
                  544,029,500,000	4,352,236,000,000		Bandwidth to Update all Installations
                  
                  				50.4	mbps for Daily Updates *
                  				  7.2	mbps for Weekly Updates *
                  				  1.7	mbps for Monthly Updates *
                  
                  					* Even Distribution for Period Duration (average)
                  
                  

                  Significant amount of bandwidth, but certainly not insurmountable.

                  1 Reply Last reply Reply Quote 0
                  • C
                    cmb
                    last edited by

                    The Cymru guys replied back that they update daily based on what they're pulling from the RIRs, but they have no info unfortunately on how quickly that IP space is actually in use.

                    The issue with the even distribution for the period is you probably don't want to put a random sleep in there that's a week long as that introduces its own challenges. Updating it every, say, Saturday or Sunday with enough of a random sleep to spread the load over a day, would be the way to keep things. Doing that 4 times a month is enough to blow your 95th percentile to ~50 Mbps, which is potentially $1000/month worth of bandwidth if we had to pay for it at some point. Granted, we can spread that out enough now and probably in the foreseeable future that it won't be an issue.

                    Frequency of updates is going to have to be a wait and see thing for now. Hopefully as many 2.1 boxes as are out there (upwards of 3000 systems fetched v6 bogons this month), we'll get some feedback from the community, as well as with our own installs like the one all our sites are running behind, and have a reasonable frequency by release.

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.