"Could not extract fullbogons-ipv6.txt"

artenpie

Running 2.8.0 beta, upgraded from a fresh install of 2.7.2.

2.8.0-BETA (amd64)
built on Mon Mar 31 22:29:00 EDT 2025

When I go to Diagnostics/Tables/bogonsv6, the table is empty (this is normal). Then I click on "Update", nothing happens for a minute or so, and then the table remains empty. In the system logs, I see:

Apr 6 12:07:21 	root 	63685 	rc.update_bogons.sh is sleeping for 84898
Apr 6 12:07:21 	root 	62960 	rc.update_bogons.sh is starting up.
Apr 6 12:07:21 	root 	61493 	Could not extract fullbogons-ipv6.txt
Apr 6 12:07:15 	root 	47839 	Could not extract fullbogons-ipv4.txt
Apr 6 12:07:14 	root 	44347 	rc.update_bogons.sh is beginning the update cycle.
Apr 6 12:07:14 	root 	43138 	rc.update_bogons.sh is starting up.

I see this same error in the latest (2nd) beta of 25.3.

I do not see this error in 24.11 or 2.7.2 (i.e., the bogonsv6 table properly updates in both those releases).

Gertjan

@artenpie said in "Could not extract fullbogons-ipv6.txt":

I see this same error in the latest (2nd) beta of 25.3.

I'm using that same 25.3-2 beta version ...

The shell script loads : https://files.netgate.com/lists/fullbogons-ipv6.txt
Btw : it also loads https://files.netgate.com/lists/fullbogons-ipv4.txtt just before that.

Coming from the same server, so f the first worked out fine, and the second didn't, see it as a temporary glitch.
I presume you've checked disk space left, it's a 2,6 Mbytes file.

This file : rc.update_bogons.sh is pretty straight forward, and probably something's off :

Click on the links above, and the files will download in your browser.
Then do what the script does (console or SSH command line please) :

/usr/bin/fetch -a -w 600 -T 30 -q -o /tmp/bogonsv6 https://files.netgate.com/lists/fullbogons-ipv6.txt

and you'll have a file : /tmp/bogonsv6

Then then script does this : (line 49 ....) :

	/usr/bin/tar -xf "$file.tmp" -O > "$file" 2> /dev/null

It extracts from an already plain text file ???

So : let not tar extract a file that isn't tarred ^^ :

and now, when running manually :

/etc/rc.update_bogons.sh 2

Looks better ...

artenpie

@Gertjan Thank you for your work - it made me do some digging :-)

Looking at my 2.8 beta, I see what you see. I also see it in /etc/rc.update_bogons.sh on github

https://github.com/pfsense/pfsense/blob/master/src/etc/rc.update_bogons.sh

process_url() {
	local file="$1"
	local url="$2"
	local filename="${url##*/}"

	/usr/bin/fetch -a -w 600 -T 30 -q -o "$file" "${url}"

	if [ ! -f "$file" ]; then
		echo "Could not download ${url}" | logger
		proc_error="true"
	fi

	/usr/bin/tar -xf "$file.tmp" -O > "$file" 2> /dev/null

	if [ -f "$file.tmp" ]; then
		rm "$file.tmp"
	else
		echo "Could not extract ${filename}" | logger
		proc_error="true"
	fi
}

However, on my (working) 24.11 installation, I see something different:

process_url() {
	local file=$1
	local url=$2
	local filename=${url##*/}
	local ext=${filename#*.}

	/usr/bin/fetch -a -w 600 -T 30 -q -o $file "${url}"

	if [ ! -f $file ]; then
		echo "Could not download ${url}" | logger
		proc_error="true"
	fi

	case "$ext" in
		tar)
			mv $file $file.tmp
			/usr/bin/tar -xf $file.tmp -O > $file 2> /dev/null
			;;
		tar.gz)
			mv $file $file.tmp
			/usr/bin/tar -xzf $file.tmp -O > $file 2> /dev/null
			;;
		tgz)
			mv $file $file.tmp
			/usr/bin/tar -xzf $file.tmp -O > $file 2> /dev/null
			;;
		tar.bz2)
			mv $file $file.tmp
			/usr/bin/tar -xjf $file.tmp -O > $file 2> /dev/null
			;;
		*)
			;;
	esac

	if [ -f $file.tmp ]; then
		rm $file.tmp
	fi

	if [ ! -f $file ]; then
		echo "Could not extract ${filename}" | logger
		proc_error="true"
	fi
}

It looks like the working code (in 24.11) tests for the file extension and untars if/as needed, but the code in github (and 2.8 beta) doesn't.

Gertjan

That explains for 2.7.2 :

@artenpie said in "Could not extract fullbogons-ipv6.txt":

case "$ext" in

When the file(s) was (were) downloaded, there is an test - the "if" statement :

If it was an 'tar','tar.gz' or 'tar.bz2', then the file is "dezipped" with the correct "unzipper" command.

I'm using 24.11 - actually, I was, right now I'm on 25.03B2, and the file to be downloaded is a .txt file.
And it's still "dezipped" - and that fails for me.

@artenpie said in "Could not extract fullbogons-ipv6.txt":

local url="$2"

What is the URL for these two file for 2.7.2 ?

artenpie

I don't have 2.7.2 running at the moment, but in 24.11, those two files are:

v4url=${v4url:-"https://files.netgate.com/lists/fullbogons-ipv4.txt"}
v6url=${v6url:-"https://files.netgate.com/lists/fullbogons-ipv6.txt"}

Gertjan

@artenpie

So you know now why it fails.
And what to do about it ^^

marcosm

Thanks for testing! For reference:
https://redmine.pfsense.org/issues/16129

artenpie

@marcosm Thanks, Marcos.

artenpie

@marcosm I manually edited rc.update_bogons.sh to apply the two changesets shown in the redmine. (I first manually applied the first changeset, then I manually applied the one with "additional improvements", which was clearly an edit on top of the first one.)

Both the bogonsv4 and bogonsv6 now update correctly. Here are my logs:

Apr 8 17:33:40 	root 	62500 	rc.update_bogons.sh is ending the update cycle.
Apr 8 17:33:40 	root 	61824 	Bogons V6 file downloaded: 150898 addresses added.
Apr 8 17:33:38 	root 	55652 	Bogons V4 file downloaded: 2807 addresses added.
Apr 8 17:33:27 	root 	42097 	rc.update_bogons.sh is beginning the update cycle.
Apr 8 17:33:27 	root 	40810 	rc.update_bogons.sh is starting up.

Thanks!