Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    "Could not extract fullbogons-ipv6.txt"

    Scheduled Pinned Locked Moved CE 2.8.0 Development Snapshots (Retired)
    9 Posts 3 Posters 458 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      artenpie
      last edited by

      Running 2.8.0 beta, upgraded from a fresh install of 2.7.2.

      2.8.0-BETA (amd64)
      built on Mon Mar 31 22:29:00 EDT 2025
      

      When I go to Diagnostics/Tables/bogonsv6, the table is empty (this is normal). Then I click on "Update", nothing happens for a minute or so, and then the table remains empty. In the system logs, I see:

      Apr 6 12:07:21 	root 	63685 	rc.update_bogons.sh is sleeping for 84898
      Apr 6 12:07:21 	root 	62960 	rc.update_bogons.sh is starting up.
      Apr 6 12:07:21 	root 	61493 	Could not extract fullbogons-ipv6.txt
      Apr 6 12:07:15 	root 	47839 	Could not extract fullbogons-ipv4.txt
      Apr 6 12:07:14 	root 	44347 	rc.update_bogons.sh is beginning the update cycle.
      Apr 6 12:07:14 	root 	43138 	rc.update_bogons.sh is starting up. 
      

      I see this same error in the latest (2nd) beta of 25.3.

      I do not see this error in 24.11 or 2.7.2 (i.e., the bogonsv6 table properly updates in both those releases).

      GertjanG 1 Reply Last reply Reply Quote 0
      • GertjanG
        Gertjan @artenpie
        last edited by Gertjan

        @artenpie said in "Could not extract fullbogons-ipv6.txt":

        I see this same error in the latest (2nd) beta of 25.3.

        I'm using that same 25.3-2 beta version ...

        The shell script loads : https://files.netgate.com/lists/fullbogons-ipv6.txt
        Btw : it also loads https://files.netgate.com/lists/fullbogons-ipv4.txtt just before that.

        Coming from the same server, so f the first worked out fine, and the second didn't, see it as a temporary glitch.
        I presume you've checked disk space left, it's a 2,6 Mbytes file.

        This file : rc.update_bogons.sh is pretty straight forward, and probably something's off :

        Click on the links above, and the files will download in your browser.
        Then do what the script does (console or SSH command line please) :

        /usr/bin/fetch -a -w 600 -T 30 -q -o /tmp/bogonsv6 https://files.netgate.com/lists/fullbogons-ipv6.txt
        

        and you'll have a file : /tmp/bogonsv6

        Then then script does this : (line 49 ....) :

        	/usr/bin/tar -xf "$file.tmp" -O > "$file" 2> /dev/null
        

        It extracts from an already plain text file ???

        So : let not tar extract a file that isn't tarred ^^ :

        d1d5196b-8288-40b3-a46f-069198eda31c-image.png

        and now, when running manually :

        /etc/rc.update_bogons.sh 2
        

        01d06599-230f-4727-a5f5-316a34664817-image.png

        Looks better ...

        No "help me" PM's please. Use the forum, the community will thank you.
        Edit : and where are the logs ??

        A 1 Reply Last reply Reply Quote 0
        • A
          artenpie @Gertjan
          last edited by

          @Gertjan Thank you for your work - it made me do some digging :-)

          Looking at my 2.8 beta, I see what you see. I also see it in /etc/rc.update_bogons.sh on github

          https://github.com/pfsense/pfsense/blob/master/src/etc/rc.update_bogons.sh

          process_url() {
          	local file="$1"
          	local url="$2"
          	local filename="${url##*/}"
          
          	/usr/bin/fetch -a -w 600 -T 30 -q -o "$file" "${url}"
          
          	if [ ! -f "$file" ]; then
          		echo "Could not download ${url}" | logger
          		proc_error="true"
          	fi
          
          	/usr/bin/tar -xf "$file.tmp" -O > "$file" 2> /dev/null
          
          	if [ -f "$file.tmp" ]; then
          		rm "$file.tmp"
          	else
          		echo "Could not extract ${filename}" | logger
          		proc_error="true"
          	fi
          }
          

          However, on my (working) 24.11 installation, I see something different:

          process_url() {
          	local file=$1
          	local url=$2
          	local filename=${url##*/}
          	local ext=${filename#*.}
          
          	/usr/bin/fetch -a -w 600 -T 30 -q -o $file "${url}"
          
          	if [ ! -f $file ]; then
          		echo "Could not download ${url}" | logger
          		proc_error="true"
          	fi
          
          	case "$ext" in
          		tar)
          			mv $file $file.tmp
          			/usr/bin/tar -xf $file.tmp -O > $file 2> /dev/null
          			;;
          		tar.gz)
          			mv $file $file.tmp
          			/usr/bin/tar -xzf $file.tmp -O > $file 2> /dev/null
          			;;
          		tgz)
          			mv $file $file.tmp
          			/usr/bin/tar -xzf $file.tmp -O > $file 2> /dev/null
          			;;
          		tar.bz2)
          			mv $file $file.tmp
          			/usr/bin/tar -xjf $file.tmp -O > $file 2> /dev/null
          			;;
          		*)
          			;;
          	esac
          
          	if [ -f $file.tmp ]; then
          		rm $file.tmp
          	fi
          
          	if [ ! -f $file ]; then
          		echo "Could not extract ${filename}" | logger
          		proc_error="true"
          	fi
          }
          

          It looks like the working code (in 24.11) tests for the file extension and untars if/as needed, but the code in github (and 2.8 beta) doesn't.

          GertjanG 1 Reply Last reply Reply Quote 0
          • GertjanG
            Gertjan @artenpie
            last edited by Gertjan

            That explains for 2.7.2 :

            @artenpie said in "Could not extract fullbogons-ipv6.txt":

            case "$ext" in

            When the file(s) was (were) downloaded, there is an test - the "if" statement :

            If it was an 'tar','tar.gz' or 'tar.bz2', then the file is "dezipped" with the correct "unzipper" command.

            I'm using 24.11 - actually, I was, right now I'm on 25.03B2, and the file to be downloaded is a .txt file.
            And it's still "dezipped" - and that fails for me.

            @artenpie said in "Could not extract fullbogons-ipv6.txt":

            local url="$2"

            What is the URL for these two file for 2.7.2 ?

            No "help me" PM's please. Use the forum, the community will thank you.
            Edit : and where are the logs ??

            1 Reply Last reply Reply Quote 0
            • A
              artenpie
              last edited by

              I don't have 2.7.2 running at the moment, but in 24.11, those two files are:

              v4url=${v4url:-"https://files.netgate.com/lists/fullbogons-ipv4.txt"}
              v6url=${v6url:-"https://files.netgate.com/lists/fullbogons-ipv6.txt"}
              
              GertjanG 1 Reply Last reply Reply Quote 0
              • GertjanG
                Gertjan @artenpie
                last edited by Gertjan

                @artenpie

                So you know now why it fails.
                And what to do about it ^^

                No "help me" PM's please. Use the forum, the community will thank you.
                Edit : and where are the logs ??

                1 Reply Last reply Reply Quote 0
                • M
                  marcosm Netgate
                  last edited by

                  Thanks for testing! For reference:
                  https://redmine.pfsense.org/issues/16129

                  A 2 Replies Last reply Reply Quote 0
                  • A
                    artenpie @marcosm
                    last edited by

                    @marcosm Thanks, Marcos.

                    1 Reply Last reply Reply Quote 0
                    • A
                      artenpie @marcosm
                      last edited by

                      @marcosm I manually edited rc.update_bogons.sh to apply the two changesets shown in the redmine. (I first manually applied the first changeset, then I manually applied the one with "additional improvements", which was clearly an edit on top of the first one.)

                      Both the bogonsv4 and bogonsv6 now update correctly. Here are my logs:

                      Apr 8 17:33:40 	root 	62500 	rc.update_bogons.sh is ending the update cycle.
                      Apr 8 17:33:40 	root 	61824 	Bogons V6 file downloaded: 150898 addresses added.
                      Apr 8 17:33:38 	root 	55652 	Bogons V4 file downloaded: 2807 addresses added.
                      Apr 8 17:33:27 	root 	42097 	rc.update_bogons.sh is beginning the update cycle.
                      Apr 8 17:33:27 	root 	40810 	rc.update_bogons.sh is starting up. 
                      

                      Thanks!

                      1 Reply Last reply Reply Quote 1
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.