Bogon Lists Mirror Outdated

NOYB

pfSense mirror IPv6 full bogons list hasn't been updated in nearly 2 months (Oct. 12th).

mirror: http://files.pfsense.org/mirrors/fullbogons-ipv6.txt
master: http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

What is the expected update schedule / frequency?

Also pfSense mirror does not have the IPv4 full bogons list. Can it be added?

mirror: http://files.pfsense.org/mirrors/fullbogons-ipv4.txt
master: http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt

Thanks

cmb

We had to make an emergency switch to a backup server at another datacenter on that host, expecting it to be temporary but it's going to be a bit longer. New server didn't have the update scheduled. Fixed so it checks daily as the old server did, and I added the full v4 bogons here:
http://files.pfsense.org/lists/fullbogons-ipv4.txt

I also changed 2.1 so it now uses the fullbogons-ipv4.txt list instead of bogon-bn-nonagg.txt. If that proves not to break anything, I'll likely change the update script to write fullbogons-ipv4.txt to bogon-bn-nonagg.txt so existing installs switch to that list as well.

NOYB

Thanks!

What about the pfSense cron job update schedule. Is once a month frequent enough for full bogons? I've been running it at once a day. Overkill?

Once a month for a home environment might be okay, but a business might what a more frequent update cycle.

Also I'd like to suggest / request not removing the comments from bogonsv6 list.


rc.update_bogons.sh: 58
current: egrep -v "^#" /tmp/bogonsv6 > /etc/bogonsv6
suguest: egrep -v "^\\" /tmp/bogonsv6 > /etc/bogonsv6

cmb

That's why I haven't replaced the main bogons list every current install fetches, I'm not sure what a reasonable update frequency would be, and there's no changing that on 150,000+ existing installs that fetch it every month. Historically, back when the list we were using was actually changing, it took well over a month between when something was assigned, removed from the bogons list, and when it was actually in use. The fullbogon list may be significantly different from that. Ditto potentially for the v6 list.

Updating it daily is almost certainly overkill unless Cymru lags on updating the list and removes something after it's in active use. Monthly may or may not be frequent enough.

There are also bandwidth and server usage considerations on how frequent to make the updates, which we haven't really taken into account for 2.1 yet. The v6 list is vastly larger than the v4 list, and the fullbogon v4 list is over 400 times the size of the former v4 list. Multiply that by 150,000+, with growth of about 50% every year at current rates, and you have a big number. Historically the only scalability concern has been a web server that performs well under significant load (trying to scale Apache for that proved to be a serious pain, thttpd just works), but changing v4 to fullbogons and adding v6 is going to create bandwidth concerns where they never previously existed. Not a problem as long as we have good colo sponsors like NYI and bluegrass.net, but if we had to shift all that to the colo we're paying market rate for in Austin, it could bring our 95th percentile up enough to cost upwards of $1000/month more. We're already paying more for colo than the general donations the project receives.

I contacted Team Cymru to see if they could provide any guidance on how quickly IP space is in use once removed from their v4 and v6 fullbogon lists. Hopefully they'll respond and provide some guidance, and we can take that into consideration for update frequency for 2.1.

NOYB

It will be interesting to hear what Team Cymru has to say.

Would a CDN (Content Delivery Network) service, such as Akamai, be in order to distribute the load?

cmb

It'll never be the kind of scale for something like Akamai (and that would cost more). Round robin DNS load balancing is the next step.

NOYB

Growth rates typically slow as demand is filled and becomes saturated.

Also as IPv6 becomes more widely implemented the list should shrink. Though with such a large address space the percentage change will probably be slight for quite some time.


	Bytes			Bits		
             76,527	              612,216		IPv4 Bogons List Size
         1,011,532	          8,092,256		IPv6 Bogons List Size
         1,088,059	          8,704,472		IPv4+6 Bogons List Size

						 500,000		~# pfSense Instalations + 3 Years of 50% Growth
544,029,500,000	4,352,236,000,000		Bandwidth to Update all Installations

				50.4	mbps for Daily Updates *
				  7.2	mbps for Weekly Updates *
				  1.7	mbps for Monthly Updates *

					* Even Distribution for Period Duration (average)

Significant amount of bandwidth, but certainly not insurmountable.

cmb

The Cymru guys replied back that they update daily based on what they're pulling from the RIRs, but they have no info unfortunately on how quickly that IP space is actually in use.

The issue with the even distribution for the period is you probably don't want to put a random sleep in there that's a week long as that introduces its own challenges. Updating it every, say, Saturday or Sunday with enough of a random sleep to spread the load over a day, would be the way to keep things. Doing that 4 times a month is enough to blow your 95th percentile to ~50 Mbps, which is potentially $1000/month worth of bandwidth if we had to pay for it at some point. Granted, we can spread that out enough now and probably in the foreseeable future that it won't be an issue.

Frequency of updates is going to have to be a wait and see thing for now. Hopefully as many 2.1 boxes as are out there (upwards of 3000 systems fetched v6 bogons this month), we'll get some feedback from the community, as well as with our own installs like the one all our sites are running behind, and have a reasonable frequency by release.