Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [PFB_FILTER - 17] Failed or invalid Mime Type: [application/octet-stream|0]

    Scheduled Pinned Locked Moved pfBlockerNG
    16 Posts 3 Posters 577 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • tinfoilmattT Offline
      tinfoilmatt LAYER 8
      last edited by tinfoilmatt

      This has come up (at least on this forum) here, here, here, and here.

      As of today, I have a CE 2.8.1-RELEASE system with pfBlockerNG-devel 3.2.10 installed, and which during execution of...

      /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc

      ...the following error is printed to the pfblockerng.log file:

      [PFB_FILTER - 17] Failed or invalid Mime Type: [application/octet-stream|0]

      Upon additional inspection of extras.log, it appears that the error is being thrown specifcally during the attempted download of (configured-to-use-on-my-particular system) the Tranco TOP1M feed. Relevant snippet:

      Download Process Starting [ 11/20/25 16:14:45 ]
 /usr/local/share/GeoIP/GeoLite2-Country.tar.gz		200 OK
 /usr/local/share/GeoIP/GeoLite2-Country-CSV.zip		200 OK
 /var/db/pfblockerng/top-1m.csv.zip		200 OK

Failed to Download top-1m.csv
 /usr/local/share/GeoIP/asn.mmdb		200 OK
 /usr/local/share/GeoIP/asn.csv.gz		200 OK
ASN Lookup Table has been updated [ 11/20/25 16:14:53 ]
Download Process Ended [ 11/20/25 16:14:56 ]

Country code update Start
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 11/20/25 16:15:31 ]
 Creating pfBlockerNG Continent PHP files
 IPv4 Africa			 [ 11/20/25 16:15:56 ]
 IPv6 Africa			 [ 11/20/25 16:15:57 ]
 IPv4 Antarctica		
 IPv6 Antarctica		
 IPv4 Asia			
 IPv6 Asia			 [ 11/20/25 16:16:01 ]
 IPv4 Europe			 [ 11/20/25 16:16:04 ]
 IPv6 Europe			 [ 11/20/25 16:16:13 ]
 IPv4 North America		 [ 11/20/25 16:16:21 ]
 IPv6 North America		 [ 11/20/25 16:16:33 ]
 IPv4 Oceania			 [ 11/20/25 16:16:43 ]
 IPv6 Oceania			 [ 11/20/25 16:16:44 ]
 IPv4 South America		
 IPv6 South America		 [ 11/20/25 16:16:45 ]
 IPv4 Proxy and Satellite	 [ 11/20/25 16:16:46 ]
 IPv6 Proxy and Satellite	
 IPv4 Top Spammers		
 IPv6 Top Spammers		
 pfBlockerNG Reputation Tab
Country Code Update Ended

      Given the previous posts, and setting aside that this same issue has been observed with other feeds and at least one other unaccounted for MIME-type (i.e., application/SIMH-tape-data), I assume that if I add application/octet-stream to the list starting at L257 of /usr/local/pkg/pfblockerng/pfblocker.inc, then the download will complete and be parsed correctly. In fact, I had done that prior to the most recent package update and temporarily resolved the issue.

      @BBcan177, any insight as to why this 'fix', i.e., adding application/octet-stream to the list of acceptable MIME-type downloads, hasn't yet been officially made?

      This issue is particularly elusive because the error is printed rather nondescriptly to the pfblockerng.log, which is easily 'buried' within normally logged update operations—and because, so long as the TOP1M feed is successfully downloaded and parsed just one time, the package will continue to reuse its stale copy.

      1 Reply Last reply Reply Quote 0
      • tinfoilmattT Offline
        tinfoilmatt LAYER 8
        last edited by

        @BBcan177 Any ideas why "application/octet-stream" would need to be manually added to pfblocker.inc to permit the Tranco TOP1M feed?

        BBcan177B 1 Reply Last reply Reply Quote 0
        • BBcan177B Offline
          BBcan177 Moderator @tinfoilmatt
          last edited by

          @tinfoilmatt I think those are download errors or rate limiting. The file that is downloaded during these times are not the files requested but an error page. The File mime type of that error page reporting it as an octet stream. I plan on improving that reporting a bit more.

          Try to curl that feed to pfSense /tmp folder and run a "file --mime-type /tmp/filename"

          "Experience is something you don't get until just after you need it."

          Website: http://pfBlockerNG.com
          Twitter: @BBcan177  #pfBlockerNG
          Reddit: https://www.reddit.com/r/pfBlockerNG/new/

          tinfoilmattT 1 Reply Last reply Reply Quote 1
          • tinfoilmattT Offline
            tinfoilmatt LAYER 8 @BBcan177
            last edited by

            @BBcan177 Returns 307: Temporary Redirect. Makes sense then that the redirect apparently isn't being followed?

            I'm able to download the CSV from a web browser on the same LAN. Confirmed the redirect is also present requesting that way.

            1 Reply Last reply Reply Quote 0
            • tinfoilmattT Offline
              tinfoilmatt LAYER 8
              last edited by tinfoilmatt

              I also understand now, which I didn't until a few moments ago, that application/octet-stream is something like a generic, default-like MIME-type.

              So I would need to re-confirm then that adding it to the list of acceptable MIME-types @ L257 of pfblockerng.inc does, in fact, 'permit' the updater process to follow the redirect.

              BBcan177B 1 Reply Last reply Reply Quote 0
              • tinfoilmattT Offline
                tinfoilmatt LAYER 8
                last edited by tinfoilmatt

                Oooh, I think I understand what's happening now. The download link http://tranco-list.eu/top-1m.csv.zip is apparently redirecting to http://tranco-list.eu/download/daily/top-1m.csv.zip.

                1 Reply Last reply Reply Quote 0
                • BBcan177B Offline
                  BBcan177 Moderator @tinfoilmatt
                  last edited by

                  @tinfoilmatt

                  Need to add "-L" to allow the redirect. pfB already does that:

                  curl -L -o /tmp/top-1m.csv.zip https://tranco-list.eu/top-1m.csv.zip

                  file --mime-type /tmp/top-1m.csv.zip
                  /tmp/top-1m.csv.zip: application/octet-stream

                  So it seems that its reporting the zip as an "octet-stream". There have been several strange reporting's with mime-types. I guess its probably best to add that to the allow list of mime-types. ZIPs have also been an issue with StopForumSpam which I added a workaround for those URLs only, and not a blanket allow.

                  More info:

                  Many ZIP files — especially those created by certain tools (7-Zip, some Java libraries, Go's archive/zip, some web servers, etc.) — do not start with the standard ZIP magic signature PK\003\004 at the very beginning of the file. Instead, they might:

  Be self-extractable archives (start with a stub executable)
  Have extra data prepended (e.g., some update tools or download managers)
  Use a non-standard ZIP variant or extension (ZIP64, encrypted with headers, etc.)
  Be streamed or created in a way that shifts the local header

In all these cases, libmagic cannot reliably recognize it as ZIP just by looking at the beginning, so it falls back to the safe generic type: application/octet-stream.

                  So short term we can add octet-stream, but I need to add a workaround for these ZIP files that check the validity before extracting these files with something like "unzip -t" to confirm that it is an expected ZIP file and not mailicious

                  "Experience is something you don't get until just after you need it."

                  Website: http://pfBlockerNG.com
                  Twitter: @BBcan177  #pfBlockerNG
                  Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                  tinfoilmattT 1 Reply Last reply Reply Quote 1
                  • tinfoilmattT Offline
                    tinfoilmatt LAYER 8 @BBcan177
                    last edited by

                    @BBcan177 Ah, my bad on the curl syntax/lacking knowledge. But everything else well understood.

                    Both the guidance and acknowledgment of the sensitivity here (i.e., future checksum verification implementation) are especially appreciated. I feel much more comfortable adding back application/octet-stream for the time being.

                    Thank you, sir!

                    C 1 Reply Last reply Reply Quote 1
                    • C Offline
                      ChandlerL @tinfoilmatt
                      last edited by ChandlerL

                      @tinfoilmatt I'm still unclear on this as adding octet-stream in to the application type list in pfblocker.inc does get rid of the mime error but I still get a not found error on top-1m.csv. I can download the Tranco list fine when doing it from a browser: link text

                      ===[  DNSBL Process  ]================================================

 Loading DNSBL Statistics... completed
 Loading DNSBL SafeSearch... disabled
 Loading DNSBL Whitelist... completed
 Loading TOP1M Whitelist...
 TOP1M Database downloading ( approx 21MB ) ... Please wait ...
 Building TOP1M Whitelist [
TOP1M conversion Failed. File: top-1m.csv, not found...
 DNSBL - TOP1M changes found - Rebuilding!

                      How are you resolving that portion?

                      Thanks!

                      -Chan

                      tinfoilmattT 1 Reply Last reply Reply Quote 0
                      • tinfoilmattT Offline
                        tinfoilmatt LAYER 8 @ChandlerL
                        last edited by tinfoilmatt

                        @ChandlerL You could try rm /var/db/pfblockerng/top-1m.* (removes all TOP1M downloads)—and then /usr/local/bin/php -f /usr/local/www/pfblockerng/pfblockerng.php dcc ('manual' "Extras" update).

                        C 2 Replies Last reply Reply Quote 0
                        • C Offline
                          ChandlerL @tinfoilmatt
                          last edited by

                          @tinfoilmatt Will give that a shot. Thank you, Tin

                          1 Reply Last reply Reply Quote 0
                          • C Offline
                            ChandlerL @tinfoilmatt
                            last edited by ChandlerL

                            @tinfoilmatt I think what's happening here is that the pfblockerng.php is not unzipping the top1m list. It's downloading it as "top-1m.csv.zip.orig" but there's no top-1m.csv in the directory. For kicks, I tried unzipping the .orig file to top-1m.csv manually then running a DNSBL Force Reload and it worked because the top-1m.csv file is there.

                            total 9687
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 ET
drwxr-xr-x  2 root wheel       9 Dec 24 11:19 deny
drwxr-xr-x  2 root wheel      12 Dec 24 12:17 dnsbl
drwxr-xr-x  2 root wheel       5 Nov 20 02:13 dnsblalias
drwxr-xr-x  2 root wheel      13 Dec 24 00:45 dnsblorig
-rw-r--r--  1 root wheel   10093 Dec 24 11:59 geoip.txt
-rw-r--r--  1 root wheel  206207 Dec 24 12:15 mastercat
-rw-------  1 root wheel  386512 Dec 24 12:15 masterfile
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 match
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 native
drwxr-xr-x  2 root wheel       9 Dec 24 00:46 original
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 permit
-rw-r--r--  1 root wheel    4277 Dec 24 12:16 pfbdnsblsuppression.txt
-rw-r--r--  1 root wheel 9599178 Dec 23 17:16 top-1m.csv.zip.orig
[25.11-RELEASE][admin@pfSense.britannia.lan]/var/db/pfblockerng:
[25.11-RELEASE][admin@pfSense.britannia.lan]/var/db/pfblockerng: unzip top-1m.csv.zip.orig top-1m.csv
Archive:  top-1m.csv.zip.orig
 extracting: top-1m.csv
[25.11-RELEASE][admin@pfSense.britannia.lan]/var/db/pfblockerng: ls -l
total 24488
drwxr-xr-x  2 root wheel        2 Nov 20 02:11 ET
drwxr-xr-x  2 root wheel        9 Dec 24 11:19 deny
drwxr-xr-x  2 root wheel        3 Dec 24 12:22 dnsbl
drwxr-xr-x  2 root wheel        5 Nov 20 02:13 dnsblalias
drwxr-xr-x  2 root wheel       13 Dec 24 00:45 dnsblorig
-rw-r--r--  1 root wheel    10093 Dec 24 11:59 geoip.txt
-rw-r--r--  1 root wheel   206207 Dec 24 12:18 mastercat
-rw-------  1 root wheel   386512 Dec 24 12:18 masterfile
drwxr-xr-x  2 root wheel        2 Nov 20 02:11 match
drwxr-xr-x  2 root wheel        2 Nov 20 02:11 native
drwxr-xr-x  2 root wheel        9 Dec 24 00:46 original
drwxr-xr-x  2 root wheel        2 Nov 20 02:11 permit
-rw-r--r--  1 root wheel   103199 Dec 24 12:22 pfbalexawhitelist.txt
-rw-r--r--  1 root wheel     4277 Dec 24 12:22 pfbdnsblsuppression.txt
-rw-------  1 root wheel 22324047 Dec 23 22:16 top-1m.csv
-rw-r--r--  1 root wheel  9599178 Dec 23 17:16 top-1m.csv.zip.orig

                            Now, I can run the Force DNSBL Reload and...

                             UPDATE PROCESS START [ v3.2.13_4 ] [ 12/24/25 12:22:04 ]

===[  DNSBL Process  ]================================================

 Loading DNSBL Statistics... completed
 Loading DNSBL SafeSearch... disabled
 Loading DNSBL Whitelist... completed
 Loading TOP1M Whitelist... Building TOP1M Whitelist [.] [ Parsed 2711 lines | Found 2000 of 2000 ]...
 DNSBL - TOP1M changes found - Rebuilding!
 completed

                            Not sure why the built-in pfblocker cron job is not automatically unzipping the downloaded top1m file.

                            tinfoilmattT 1 Reply Last reply Reply Quote 0
                            • tinfoilmattT Offline
                              tinfoilmatt LAYER 8 @ChandlerL
                              last edited by

                              @ChandlerL Now you're calling my attention to some things here. I'm now noticing that my top-1m.csv.zip.orig file is new (timestamped as of the last cron update run), but the CSV itself is timestamped from months before I even created this thread. So I'm not actually sure what's happening either, whether the archive is actually being unzipped or not.

                              There's another argument for pfblockerng.php—"al"—that's commented as "Update TOP1M database only."

                              Could you run the same commands in-sequence (i.e., rm, then php), but using the al switch instead of dcc?

                              C 1 Reply Last reply Reply Quote 0
                              • C Offline
                                ChandlerL @tinfoilmatt
                                last edited by

                                @tinfoilmatt Looks like we get two files. The large .orig (zip) file and now we see a new file ending in .update but it's 0 bytes

                                [25.11-RELEASE][admin@pfSense.britannia.lan]/var/db/pfblockerng: ls -l
total 344
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 ET
drwxr-xr-x  2 root wheel       9 Dec 24 11:19 deny
drwxr-xr-x  2 root wheel      11 Dec 24 12:23 dnsbl
drwxr-xr-x  2 root wheel       5 Nov 20 02:13 dnsblalias
drwxr-xr-x  2 root wheel      13 Dec 24 00:45 dnsblorig
-rw-r--r--  1 root wheel   10093 Dec 24 11:59 geoip.txt
-rw-r--r--  1 root wheel  206207 Dec 24 12:23 mastercat
-rw-------  1 root wheel  386512 Dec 24 12:23 masterfile
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 match
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 native
drwxr-xr-x  2 root wheel       9 Dec 24 00:46 original
drwxr-xr-x  2 root wheel       2 Nov 20 02:11 permit
-rw-r--r--  1 root wheel  103199 Dec 24 12:22 pfbalexawhitelist.txt
-rw-r--r--  1 root wheel    4277 Dec 24 12:22 pfbdnsblsuppression.txt
-rw-r--r--  1 root wheel 9599178 Dec 23 17:16 top-1m.csv.zip.orig
-rw-r--r--  1 root wheel       0 Dec 24 12:59 top-1m.update
[25.11-RELEASE][admin@pfSense.britannia.lan]/var/db/pfblockerng:
                                tinfoilmattT 2 Replies Last reply Reply Quote 0
                                • tinfoilmattT Offline
                                  tinfoilmatt LAYER 8 @ChandlerL
                                  last edited by

                                  @ChandlerL So presumably now it takes either a cron or an update argument to do anything with them.

                                  1 Reply Last reply Reply Quote 0
                                  • tinfoilmattT Offline
                                    tinfoilmatt LAYER 8 @ChandlerL
                                    last edited by

                                    @ChandlerL Yeah, here's the function: pfblockerng_top1m—which from what I've perused only gets called during cron or otherwise 'Force' updates.

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.