Cannot define table bogonsv6

Guest

It has 4GB of RAM, running the i386 version of pfsense. It said the default was set to 100,000.

dhatz

Recently the IPv4 bogons has changed from the non-aggregated list to the full list which currently contains 4000+ more records.

What was the reason behind this change? And will the monthly update schedule used by pfsense be sufficient to keep the fullbogons table up-to-date ?

It is important to note that fullbogons change every day, and absolutely must be kept up-to-date, because prefixes are being distributed all the time. If you just download a fullbogons list once and use it to block access to systems, it WILL become out of date very quickly, and you WILL wind up blocking legitimate traffic.
http://www.team-cymru.org/Services/Bogons/

Not to mention any possible firewall performance hit, due to the bogons table now having thousands instead of just a handful entries …

bardelot

There exists no traditional IPv6 bogons list only the full version, so I assume this was the reason to transition the IPv4 list to the full one as well. Here's the previously mentioned commit.

The IPv6 list currently contains around 65'000 and the IPv4 list around 5000 entries. Lookups against tables are very fast and the lookup time on a table with 50'000 entries is only slightly more than for one holding 50 entries. If you still prefer the old IPv4 list you can disable bogons filtering on the interface and instead add a custom firewall rule on top using the old URL with an URL table alias.

The update interval of the bogons list should indeed be decreased as a monthly schedule for a full bogons list, which may get updated multiple times a day, is too low.

dhatz

It would seem to me that having a basic version of the bogons list should be the pfsense default, and those who wish to to use the full version (and update it every few days) can do so via a url-alias/cron job. And probably offer gzipped versions in addition to .txt.

Otherwise bogon downloads from files.pfsense.org will soon consume over 1 TeraByte/month (rough estimate based on the number of pfsense "live installs" and ~1MB per bogons/bogonsv6) for little to none apparent benefits …

NOYB

My understanding is that an appropriate update interval is under investigation. There is typically a significant lag between addresses being assigned and removed from the bogons list and becoming an active site. Probably doesn't need updated as frequently as it may seem. Don't panic.

If you want to use one of the other bogon lists, perhaps the easiest way would be to edit the /etc/rc.update_bogons.sh script.

If you want a different update schedule that can be changed in /etc/crontab and cron section of /cf/conf/config.xml

I've seen an increase in the bogon blocks with the full list. I think it is overall a positive change.

NOYB

@bardelot:

How much memory does your system have? If I remember correctly the default tables-entries should be 200'000 unless you have less than 100MB of RAM in which case it is 100'000.

I've also seen this issue on system with more than 4GB RAM. Seems like there may be a variable somewhere that needs to be of a larger type.

http://php.net/manual/en/language.types.integer.php

"
The size of an integer is platform-dependent, although a maximum value of about two billion is the usual value (that's 32 bits signed). 64-bit platforms usually have a maximum value of about 9E18. PHP does not support unsigned integers. Integer size can be determined using the constant PHP_INT_SIZE, and maximum value using the constant PHP_INT_MAX since PHP 4.4.0 and PHP 5.0.5.
"

bardelot

@NOYB:

My understanding is that an appropriate update interval is under investigation. There is typically a significant lag between addresses being assigned and removed from the bogons list and becoming an active site. Probably doesn't need updated as frequently as it may seem. Don't panic.

True, an advanced dropdown option to change the bogons update interval from monthly, weekly (default) to daily might be appropriate for most users. I also see the transition to the full bogons list as inevitable now that all IPv4 addresses have been assigned.

@NOYB:

I've also seen this issue on system with more than 4GB RAM. Seems like there may be a variable somewhere that needs to be of a larger type.

If 200'000 entries is the required minimum on all systems now, one could change the following code in filter.inc

	/* User defined maximum table entries in Advanced menu. */
	if ($config['system']['maximumtableentries'] <> "" && is_numeric($config['system']['maximumtableentries']))
		$limitrules .= "set limit table-entries {$config['system']['maximumtableentries']}\n";

to

	/* User defined maximum table entries in Advanced menu. */
	$maxentries = is_numeric($config['system']['maximumtableentries']) ? $config['system']['maximumtableentries'] : "200000";
	$limitrules .= "set limit table-entries {$maxentries}\n";

NOYB

As mentioned by cmb in another thread, the bandwidth required to support more frequent updates needs to be considered as well. Going from once a month to once a day is x30. However once a day is almost certainly overkill. Major sites just don't come and go that quickly. Even once addresses have ben assigned.

I'm confident they will figure it out and have an appropriate update schedule.

dhatz

@NOYB:

Major sites just don't come and go that quickly. Even once addresses have ben assigned.

It's not 'major sites' that I'm worried about. I can imagine a case where a roadwarrior has been assigned a dynamic IP from a newly acquired IP range (formerly bogon) by his ISP, and is trying to connect to a pfsense VPN but is blocked by his office's pfSense default block bogon WAN rule.

Edit: Also with the latest changes the uptime / reliability of files.pfsense.org server has become a (greater) issue; that server will need to distribute TeraBytes worth of bogons/bogonsv6 data every month to 150K+ "live installs" and that functionality can't be easily delegated to dozens of mirrors due to security concerns.

NOYB

I would consider an ISP that's getting directly assigned IP's to be a major site. And they are not typically going to be moving that quickly.

dhatz

Well it seems to me that unless the bogons update frequency is increased to at least weekly updates (or default is switched back to the basic version of the bogon files), it could result to infrequent and hard-to-troubleshoot connectivity issues for some people.

bardelot

I've also seen this issue on system with more than 4GB RAM. Seems like there may be a variable somewhere that needs to be of a larger type.

Indeed, FreeBSD uses a 32-bit integer to determine the physical memory size in bytes 1 and thus pf fails to correctly set the table-entries limit 2. Submitted the previous patch to bump the default value to 200000.