pfBlocker suddenly blocks all DNS lookups

Gblenn

Since a few months back I have been having what has seemed like reoccurring issues with loss of internet. I have now concluded that it's a once per month event, and always on the same weekday and at the time of the nightly scheduled pfBlocker update (Fridays at 00.15 exactly). Looking back at my server monitoring it seems it is actually the second Friday of the month... (at least it has been for past three months). The only other scheduled update I believe I have is for Suricata which is also nightly but a few hours later.

I have been using fpBlockerNG-devel since a long time and recently updated to v3.1.0_7 (the previous months was on at least one version older than 7)

The Firewall log gets completely filled with instances like this:
Dec 9 00:15:52 WAN pfB_Top_v4 auto rule (1770004179) MYIP:31453 9.9.9.9:53 UDP

Or like this if I turn of Forwarding Mode in Resolver:
Dec 10 10:35:52 WAN pfB_Top_v4 auto rule (1770004179) MYIP:15375 199.7.83.42:53 UDP

Pulling the log file 'ipblock.log' from pfBlocker shows over 630000 rows in only 35 minutes... During that time no internet, or rather no DNS, and CPU usage goes up drastically. My VPN's stay up and running and pinging external locations work just fine however.

Rebooting does not resolve it.
Restarting unbound does not fix it.
Reloading filters doesn't help.
Reinstalling pfBlocker (keeping settings) does not fix it (have not tried without 'Keep Settings' active).
Turning off Resolver 'Forwarding Mode' doesn't fix it, as example above

Disabling pfBlocker fixes it.
Disabling pfB_Top_v4 in firewall rules fixes it.
Recovering a backup VM (running Proxmox) from the day before also works. And it keeps working even after forcing an update in pfBlocker.

Interestingly I have another pfsense running on a different site which has never had this problem. It's the same version (2.6.0) and the configuration is as identical as I have been able to make it. That one is running on a PC engines APU2.

Not sure if this could have anything to do with it but for whatever reason my ISP shows up as NNN.NNN.0.0/16 in the pfB_TOP_v4.txt file (which means I'm on the list).
I'm thinking it shouldn't matter unless I was trying to "hit myself", right?? And things do work fine between the outages, with that list still in place...
The other site does use a different ISP however, and is not in the list.

Any ideas, thoughts??

SteveITS

@gblenn One idea is to set the Top list as Alias Native. Then create your own rules:
Allow to 9.9.9.9
Block top

?

Gblenn

@steveits Yes I suppose that would treat the symptom at least, and if I don't want forwarding mode I would put root server IP's in that list.

But it still doesn't fix the root cause, whatever that may be...
And I'm thinking that DNS servers used by unbound shouldn't be blocked anyway, right??!
Also, the setting I have in pfBlocker for this list is 'Deny Inbound', so it shouldn't even be involved with an outbound activity like a DNS lookup??

Another solution would of course be to disable the pfB_Top_v4 rule completely, and never use it...

And there is actually something strange going on with the list used for the Top_v4 rule...

As mentioned, I found my own IP on that list (as part of nnn.nnn.0.0/16). BUT, if I now make any changes to Top Spammers, like first selecting only one country (Save/Update) and then select all countries again, that IP range no longer shows up!! And it shouldn't have, since Sweden is not even on the list of Top Spammer countries you can select from...

Still, I think there is something else going on, as it actually works even when that IP range is in the list. It's only the one update per month that screws things up... I'm reading posts related to some unbound issue, and start thinking this may be related but can't really make the connection... and this happened also before the latest update to v3.1.0_7

Gblenn

It gets weirder... I decided to do some testing on my other site and found, after some trial and error, that for some reason my IPS is showing up in the United Kingdom GB_rep list.

Now, as mentioned in the previous post, this does NOT happen at home right now ("problem site")...

Also, the GeoIP Top Spammer lists are slightly different on the two sites, in that they seem to differ in size.
Home site: GB_rep (15932) my ISP not included
Site 2: GB_rep (16007) - my ISP included

Cool_Corona

@gblenn Same source for the lists??

Gblenn

@cool_corona I have no idea, but I don't see that you can even change that?! It's the Maxmind database that is used when "building" the lists?

I'm talking about pfBlockerNG/IP/GeoIP where you find Top Spammers as the first list you can edit. The rest are regions - Africa down to Proxy and Satellite...
You simply select countries but where the actual IP-lists come from, I have no idea...
In terms of sources you have feeds but they only relate to e.g. PRI1, 2 and TOR which are the one's I use.

Hmmm, just noticed that at the bottom of the page (pfBlockerNG/IP/GeoIP) it reads:
"(To avoid any MaxMind update delays, update is now scheduled for the first Thursday of each month.)
I wonder... Thursday early in the month, BUT Dec 8 was the second Thursday, not the first... so perhaps nothing related.

Now I stumbled on an old post talking about the exact same problem... Even the UK list is mentioned...
pfblockerng-devel-geoip-problems
Seems this has been around for a long time with no resolution.
Last post mentions not using floating rules, but no further comments or explanations. What is the problem with using floating rules?
I guess I can change...
And there is one thing with Floating rules when you look into them. They all have a list item saying 'Direction any' for all the pfB-rules in the floating list. In reality though, the direction is based on 'Source' / 'Destination' as far as I can tell.

SteveITS

@gblenn said in pfBlocker suddenly blocks all DNS lookups:

the setting I have in pfBlocker for this list is 'Deny Inbound', so it shouldn't even be involved with an outbound activity like a DNS lookup

Hm, yeah that would seem odd then.

However you do mention floating rules later. Are you using those? They can be a bit confusing as they have a direction and match differently (last match if not "quick").
https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html#precautions-caveats

Gblenn

@steveits Yes, I have had all pfBlocker rules as 'floating rules' since I first started using pfB a long way back. But the rules are automatically generated and I have never touched them (all have Quick enabled by default btw).
The do all have the 'Direction' set to 'any', which I find a bit curious, but it normally shouldn't matter as all those set as inbound in pfBlocker obviously have the Top_v4 list as 'Source', not destination.

If I create a floating rule with e.g. 8.8.4.4 as 'source' and set Direction to 'out', nothing happens, I can still ping it. Which makes sense since the source of the ping isn't google.

If I change 8.8.4.4 over to destination, and have Direction 'any' (or 'out), my ping is blocked...
But, if I now set direction to 'in', my ping is no longer being blocked... which also kind of makes sense as 8.8.4.4 may be the destination but it's not on the in side...

I think there may be something here though. As the documentation states : Another way to use floating rules is to control traffic leaving from the firewall itself. Floating rules can prevent the firewall from reaching specific IP addresses, ports, and so on.

Somehow something gets screwd up in a way that this particular rule blocks all outgoing traffic to DNS... possibly this only happens if UK is selected. And it only happens for me at a scheduled update on a Friday (CET), if an only if it's the second Friday day in the month.... ?!?!?

Gblenn

It's now been more than a month and this issue seem to be resolved. The only significant change was to stop using floating rules for pfBlocker.