No bogonsv6 in tables?

databeestje

I will need to verify. I do see we ship with a empty table but the file it fetches should hold some 50k networks.

Cino

i manually adding it a while ago and it hit a pf-filter limit. Can't remember what it was but its the same limit that IPBlocker and pfBlocker has when large list.

databeestje

It could be that the import is going wrong. Is there contents i. /Etc/bogonsv6 ?

johnpoz

-rw-r–r-- 1 root wheel 0 Mar 6 2011 bogonsv6

File is empty.

databeestje

Arr. That is suspect. Since my install does have a filled file you need to check if your dns works.

There is a script in /etc rc.update_bogons.sh that normally runs. Does that work?

johnpoz

I can assure you my dns works ;)

ok I ran that script by hand, it first picked a random number of 53000 something – that is seconds? So it was going to sleep for 14 hours? WTF

Nov 24 13:03:23 root: rc.update_bogons.sh is sleeping for 53124

So I killed it and ran it by hand with 10 seconds And it clearly updated the files.

[2.1-DEVELOPMENT][root@pfsense.local.lan]/etc(18): ls -la bogons*
-rw-r–r-- 1 root wheel 132 Nov 24 13:10 bogons
-rw-r--r-- 1 root wheel 761384 Nov 24 13:10 bogonsv6

Nov 24 13:10:11 root: rc.update_bogons.sh is ending the update cycle.
Nov 24 13:10:11 root: Bogons files downloaded: 49475 addresses added.
Nov 24 13:10:10 root: Bogons file downloaded: no changes.
Nov 24 13:10:08 root: rc.update_bogons.sh is beginning the update cycle.
Nov 24 13:10:08 root: rc.update_bogons.sh is starting up.

So something wrong with cron? Why has that script not kicked off? After I kicked it off by hand, the tables loads bogonsv6 -- and yeah its huge and does take a while to load the table ;)

johnpoz

ok did an update last night to
2.1-DEVELOPMENT (i386)
built on Fri Nov 25 17:45:38 EST 2011
FreeBSD 8.1-RELEASE-p6

and now bogonsv6 is empty again?
No entries exist in this table.

looked at /etc
-rw-r–r-- 1 root wheel 132 Apr 28 2011 bogons
-rw-r--r-- 1 root wheel 0 Mar 6 2011 bogonsv6

So is there something wrong in the cron that should update these?

So I manually ran script rc.update_bogons.sh 10
Nov 30 14:50:29 admin: Bogons files downloaded: 49621 addresses added.

[2.1-DEVELOPMENT][admin@pfsense.local.lan]/etc(19): ls -la bo*
-rw-r–r-- 1 root wheel 132 Nov 30 14:50 bogons
-rw-r--r-- 1 root wheel 763666 Nov 30 14:50 bogonsv6

So what is suppose to update these, do you have to manually run the script - should something kick it off after an update? On some sort of schedule?

iamzam

I am running the IPv6 2.1-DEVELOPMENT (amd64) branch, and I see tons of these in the system log:

Jan 3 09:03:07 pfs root: rc.update_bogons.sh is beginning the update cycle.
Jan 3 09:03:09 pfs root: Could not download http://files.pfsense.org/mirrors/bogon-bn-nonagg.txt.md5 (md5 mismatch)
Jan 3 09:03:09 pfs root: rc.update_bogons.sh is starting up.
Jan 3 09:03:09 pfs root: rc.update_bogons.sh is sleeping for 35036
Jan 3 09:03:09 pfs root: Could not download http://files.pfsense.org/mirrors/fullbogons-ipv6.txt.md5 (md5 mismatch)
Jan 3 09:03:09 pfs root: rc.update_bogons.sh is ending the update cycle.
Jan 3 09:03:09 pfs root: rc.update_bogons.sh is starting up.
Jan 3 09:03:09 pfs root: rc.update_bogons.sh is sleeping for 37978
Jan 3 09:03:52 pfs root: rc.update_bogons.sh is beginning the update cycle.

I logged in via ssh and there are 64 "sh /etc/rc.update_bogons.sh" processes running.

I could kill all the processes and manually download the file and put it in place but if the md5 sums don't match there is possibly a bigger problem than that?

jimp

Looks fine from here:

[jimp@files mirrors]$ md5 bogon-bn-nonagg.txt
MD5 (bogon-bn-nonagg.txt) = 82bf1e457c3cbb64021067e2a2230051
[jimp@files mirrors]$ cat bogon-bn-nonagg.txt.md5
MD5 (bogon-bn-nonagg.txt) = 82bf1e457c3cbb64021067e2a2230051
[jimp@files mirrors]$ md5 fullbogons-ipv6.txt
MD5 (fullbogons-ipv6.txt) = 0c56fda817b9933477c576a36da0e5b6
[jimp@files mirrors]$ cat fullbogons-ipv6.txt.md5 
MD5 (fullbogons-ipv6.txt) = 0c56fda817b9933477c576a36da0e5b6

iamzam

I'm not sure what was happening with the md5 sums not matching. I killed all the sleeping processes (and all the sleep processes too) and i still couldn't run it manually. I finally downloaded the http://files.pfsense.org/mirrors/fullbogons-ipv6.txt file (and removed the first and last comment lines) and put it into place at /etc/bogonsv6 but the web interface still showed the bogonsv6 table empty.

I then went into the IPv4 bogons table and clicked the Download the latest bogon data button. After that the bogonsv6 table was populated.

I do notice one thing with the rc.update_bogons.sh script:

in /etc/rc.update_bogons.sh there is a problem with the last part:

BOGON_MD5=/usr/bin/fetch -q -o - "http://files.pfsense.org/mirrors/fullbogons-ipv6.txt.md5" | awk '{ print $4 }'
ON_DISK_MD5=md5 /tmp/bogonsv6 | awk '{ print $4 }'
if [ "$BOGON_MD5" = "$ON_DISK_MD5" ]; then
egrep -v "^#" /tmp/bogonsv6 > /etc/bogonsv6
/etc/rc.conf_mount_ro
RESULT=/sbin/pfctl -t bogonsv6 -T replace -f /etc/bogonsv6 2>&1
rm /tmp/bogons
echo "Bogons files downloaded: $RESULT" | logger
else
echo "Could not download http://files.pfsense.org/mirrors/fullbogons-ipv6.txt.md5 (md5 mismatch)" | logger

Relaunch and sleep

sh /etc/rc.update_bogons.sh &
fi

It writes the downloaded file (minus the '^#' lines) to /tmp/bogonsv6 but after replacing /etc/bogonsv6 it deletes /tmp/bogons instead of /tmp/bogonsv6

It is possible, i suppose, that if the script was run multiple times simultaneously the IPv6 part would delete the /tmp/bogons file while the other script was attempting to use it to update /etc/bogons

jimp

I fixed that line in the script just now.

Not sure how you could have had so many of them sitting there unfinished though, that is a bit perplexing.

iamzam

Thanks jimp for fixing that!

It is strange since my crontab has the rc.update_bogons.sh running once a day,

…although each time the rc.update_bogons.sh script is run, it has the initial sleep plus each section has an additional relaunch and sleep in it for a total of 4 relaunch and 5 sleeps if it has major problems...
Such as if the WAN interface is down, or some other problem such as md5 (weird i know).

maybe an exit 1 should be called after the first relaunch so that it doesn't relaunch up to 4 times/script and start a relaunch cascade!