pfBlockerNG-devel v3.1.0_9 / v3.1.0_15

alexander.deca

We have upgraded to this release and we have suddenly dropped rules as IPv4's have been deleted in the GeoIPv4 lists -> after the cron job runs to update the GeoIPv4 every 2 hours, when manually doing an update force reload it works again as it adds many address again to the different lists (see below) and it breaks again when the cron job runs again. So temporarily disabled the cron job and it is stable.

(re-installed the package but this did not help) Any advice or what we can do to troubleshoot?

Updating: pfB_PRI1_v4
1941 addresses added.9 addresses deleted.
Updating: pfB_PRI2_v4
no changes.
Updating: pfB_PRI3_v4
2647 addresses added.11 addresses deleted.
Updating: pfB_PRI4_v4
1811 addresses added.64 addresses deleted.
Updating: pfB_PRI5_v4
no changes.
Updating: pfB_GeoIP_Unifi_v4
2058 addresses added.
Updating: pfB_GeoIP_Belgium_v4
898 addresses added.
Updating: pfB_GeoIP_IPN_clients_v4
5617 addresses added.39 addresses deleted.
Updating: pfB_GeoIP_3CX_clients_v4
9496 addresses added.48 addresses deleted.
Updating: pfB_GeoIP_SBC_dyn_clients_v4
15411 addresses added.293 addresses deleted.
Updating: pfB_IPN_Client_3CX_pub_EDL_v4
no changes.
Updating: pfB_GeoIP_EU_v4
9 addresses added.88 addresses deleted.

best regards

crsesilva

Hello,

Is anyone else having this problem?

After upgrading to pfBlockerNG-devel v3.1.0_9, using UT1 blacklist.

The error only with this category the others are Ok.

Thanks.

Gertjan

@crsesilva

The issue : the "header" is "ut1_audio-video", this is checked, like our own feeds.
One of the criteria for the 'headers' is is : letters and underscores only. ( see preg_match("/\W/", $input) on line 447, pfblockerng.inc )
The thing is : "ut1_audio-video" contains a dash : '-' => the \W check fails.

Take a look at all the files here : /var/db/pfblockerng/ut1/
All the file names are correct, only '/var/db/pfblockerng/ut1/ut1_audio-video' has a dash, which should be a underscore, I guess.

So, rename the file, change the - for _, the file name becomes : ut1_audio_video
Edit /usr/local/pkg/pfblockerng/ut1_global_usage and locate (line 112 ?) "NAME: audio-video"
Change it for "NAME: audio_video".
Save.

Back in pfBlockerng :

Disable :

and a full reload.

Enable again.
What I saw : our "Audio video" is now unchecked, it was checked before -> normal, we changed the internal name.
Check it.
( I polluted the config now ?)
and a full reload.

Champagne !

( this is as it is : it worked for me [ because I found the issue during debugging, and I think I found it] - my solution is only a work around )

crsesilva

@gertjan

Thanks for responding so quickly.

I applied the mentioned changes and it worked great.

I sincerely appreciate all your efforts.

Thanks so much for your time and support.

juliokele

@bbcan177 the response is always empty:

I've figured it out, v3.1.0_9 don't work with my HA/HAProxy setup.
More precisely with Host Overrides for HAProxy on LAN-Carp-VIP Address (192.168.1.254).
v3.1.0_7 working fine.

Draco

@bbcan177 Just upgraded to 3.1.0_9 (from _7) on 22.05-RELEASE (amd64) after disabling pfBlocker before install (enable after, and ran Update). Everything looks great so far!

planedrop

@thexman I'll give this a shot and see how it goes, thank you!

JeGr

@bbcan177 Hi,

after upgrading to _9 installations that utilized the IOC lists from ThreatFox / abuse.ch
(https://threatfox.abuse.ch/export/)
won't download the list anymore. Be it the JSON file from the last 48 hours or the full data dump (zipped with "zip" not gzip), it always ends in a MIME Type Error:

[ Abuse_ThreatFox_v4 ]		 Downloading update .. 200 OK.
 PFB_FILTER - 18 | pfb_download Failed or invalid Mime Type Compressed: [application/x-decompression-error-gzip-Unknown-compression-format|0]

or

[ Abuse_ThreatFox_48h_v4 ]	 Downloading update .. 200 OK
[PFB_FILTER - 17] Failed or invalid Mime Type: [application/json|0]

 [ pfB_PRI1_v4 - Abuse_ThreatFox_48h_v4 ] Download FAIL
  DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.

That only happened recently after upgrading, before it was running fine with _6 or _7 I believe the systems were on. The old list from before the update was/is still being used so it worked before.

List URLs are working via shell/curl or in browser so no problem on that front. It's only when trying to download it with pfB that those Mime Type errors pop up.

Edit: Edit: Curl in shell sees normal content types: content-type: application/json or content-type: application/zip so no clue where that failed or invalid types come from.

Cheers
\jens

fireodo

@jegr

Hi and a good New Year.

Read this: Mime Types

I guess it adress your problem.

Cheers,
fireodo

BBcan177

@jegr
The latest code now validates the contents of all Compressed files before extraction to ensure that the file-mime type is allowed.

There is a currently an incompatibility with ZIP files and the 'file' mime-type magic database that validates the Compressed file type contents as "application/x-decompression-error-gzip-Unknown-compression-format".

I tried to see if the file maintainer could add functionality to fix this, but unfortunately I had no luck with that. So for now, I have no way to validate the ZIP file contents before extraction, so in the next version it will first Extract ZIP compressed files, and then perform the file-mime type validation on the extracted file. I will continue to see if I can find a way to validate before extraction.

The second part is that the next version will add "application/json" as a valid mime-type.

Thanks for the report!

Draco

@bbcan177 said in pfBlockerNG-devel v3.1.0_9 / v3.1.0_15:

The second part is that the next version will add "application/json" as a valid mime-type.

Does that mean a URL like MSFT Azure IP Blocks will be a valid download source? At present, I have to download this on my PC then upload it to /var/db/pfblockerng/deny on my pfSense box.

Note the MSFT Azure IP Blocks link sends you to a page where the file download starts; that page contains a link to the most current file (at present, https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20230102.json).

[edit: added more info on links]

sensei-two

Hi
I was reading about the DoH/DoT/DoQ Blocking feature in DNSBL SafeSearch of my
pfBlockerNG-devel v3.1.0_9.
As far as I have understood it, it blocks clients on my LAN to use DoH/DoT, so
I was wondering if this feature can also affect DoT queries from Unbound itself since I enabled DoT in its setting.

Thanks

xpxp2002

This post is deleted!

xpxp2002

@sensei-two I believe it does. I use unbound to funnel all public DNS queries to Cloudflare over DoT. But I took a different approach to blocking all other DoT/DoH.

I have pfB-NG creating a deny alias using these blocklists:
https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall_ipv4
https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall_ipv6

Then I created rules on the LAN-side interfaces that destination blocks the aliases that these blocklists create.

Unbound should be using your WAN interface to reach your DoT provider and it won't have an ingress interface (i.e. there's no LAN-to-WAN or WAN-to-LAN flow). The only way I think you could control the firewall's own egress-to-WAN would be using a floating rule.

sensei-two

@xpxp2002

I also use Unbound to funnel DNS queries to Cloudflare over DoT,
so, in your opinion, if I now enable the DoH/DoT/DoQ Blocking feature in pfng something might be wrong with my "legitimate" DNS queries over DoT, right?

I am interested in your approach to block DoT from LAN's clients, but I new to pfblockerNG, and I didn't use pfSense for a long time either, so bear with me, please.

Could you tell me how to set the deny alias using the blocklists above, please?
Should I delete my floating rules if I create the new rules on the LAN-side?
My floating rules:

Thanks

xpxp2002

@sensei-two It's been a while since I looked at the DoT/DoH blocking in pfB-NG, but I thought it creates floating rules, which would impact the unbound service's egress out of the WAN interface (as well as any other interface) unless you configure the floating rule to exempt the WAN address. The problem I see with that is that you can't generally edit auto-generated rules because they are overwritten after an update cron job.

Maybe that has since been fixed in pfB-NG? I don't know for sure. I can only speak to my solution, which I have tested and know works in my environment for the DoT and DoH providers in those blocklists.

I have pfBlocker-NG creating these aliases. These can probably be Alias Native instead of Alias Deny. My understanding of de-dup and reputation should be irrelevant in my config. IPv4 details shown here. I have an identical IPv6 one, as well.

Then assign these aliases to block rules near the top of your LAN side, before anything that might allow them.

sensei-two

@xpxp2002

Ok, I think I have understood how to set them, and
I'm going to give them a go.
This way you can do without the DoH/DoT/DoQ Blocking feature in DNSBL SafeSearch, can't you?
Out of curiosity. Did you also try the DoH/DoT/DoQ Blocking feature? If so, did you stumble upon some issues because of it?
Thanks

xpxp2002

@sensei-two I honestly don't remember. I probably did. I have a couple subnets where I do allow less restricted outbound access, so it's possible I did this just to have more control.

Looking at the feeds, I also threw these into the custom IP lists (IPv4 and IPv6, respectively) at the bottom since they weren't covered by the feeds themselves.
1.0.0.2/31
1.1.1.2/31
2606:4700:4700::1002
2606:4700:4700::1003
2606:4700:4700::1112
2606:4700:4700::1113

I noticed you mentioned DoQ, as well. For completeness, here's how I'm blocking everything DNS-related on my guest subnet except for my unbound resolver.

sensei-two

@xpxp2002

Very helpful. Thank you very much indeed

sensei-two

This post is deleted!