pfBlockerng Stopped Packet Forwarding
-
For some reason, starting sometime Saturday morning (6/7) in the early hours all of my firewalls stopped passing traffic. After several hours of troubleshooting, what it looked like is that all packet forwarding to the WAN interface ceased whenever pfBlockerng was enabled. As soon as I disable pfBlockerng, all packet forwarding lights up and things run as normal.
The real fun part is that when pfBlockerng is enabled, the WAN stops being able to ping it's own gateway, and this is across several sights with different ISP's and IPs, so not sold that a single ISP is being blocked.
The only thing I can think of is that there was a bad update to a list that is either blocking 0.0.0.0/0 outbound or somehow has all my ISP's IPs in a blocklist. Can anyone think of anything else that could cause pfBlocker to stop all packet forwarding to the WAN?Not really sure, but after several reboots and enable/disable cycles on pfBlockerng, I no longer have the issue that packet forwarding no longer works when pfBlockerng is enabled, not sure why that's the case but it's what it is I guess. But now I have this floating rule that pfBlocker is creating, called pfB_DNSBLIP_v4, that is blocking all my outbound DNS from the firewall, which means I have no DNS resolution. I have searched now for 3 days trying to figure out where this rule get's set so I can make sure my public DNS services are not blocked on the firewall.
Any thoughts?
-
@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
Any thoughts?
Sure.
pfBlockerng, when initially installed, does ..... nothing.The the 'admin' starts adding IP lists and DNSBL lists, and suddenly you can visit some site anymore, or entire networks aren't accessible anymore.
This should be exactly what you want.But, example, what happens when you've installed a list with IP networks, and that list suddenly contains also "8.8.8.8", and you are not using the resolver in the default Resolve mode, but you've decided to forward your DNS requests to "8.8.8.8" ? (this really happened ones ^^)
I guess you get it by now : you've just lost DNS on pfSense.How to solve the issue : easy :
Disable all IP feeds. (and gid rid of all GEO lists, and disable all reputation settings)
Disable all DNSBL feeds.
and do force -> reload.
Done. No more issues.As said, pfBlockerng by itself does nothing.
It does download and create DNSBL lists to be used by unbound, so unbound stops resolving (or forwarding) host names that are listed.
It creates lists that are included into a ... one or more firewall rules. And these rules can 'stop' traffic. That is what pfBlockerng should do in the first place.
And that's what you see happening right now.
You pick lists and feeds.
pfBlockerng uses them.
If the lists or feeds contain "BS", then yeah, "BS" will happen.So : again : disable the IP feeds -> Force reload -> firewall rules are gone -> problem solved.
From here : activate your IP feeds one by one. Force reload -> test for a (long) while. Activate next IP feed etc. As soon as things block, you know what feeds to inspect. What IP / network to whitelist.
For later : check - yes, read with your eyes, the IP feeds (and DNSBL) that pfBlockerng loads / updates - as per your instructions. If it contains a false positive, you know now what can happen.
After all, these lists are maintained by people like you and me, and a problem might arrive.
pfBlockerng is like a kid. You can leave it alone for a while .... but never to long, you have to check and correct it ones in a while. Not doing so will ... well, you know what happens. -
@Gertjan said in pfBlockerng Stopped Packet Forwarding:
How to solve the issue : easy :
Disable all IP feeds. (and gid rid of all GEO lists, and disable all reputation settings)
Disable all DNSBL feeds.
and do force -> reload.
Done. No more issues.I really wish it was that easy, as I already went that route, and guess what, still being blocked. So not sure where else this would be getting the IPs from if not in my feeds, but there's definately something going awry here. And it's not just 8.8.8.8, it's Cloud9, Cloudflare, Google, and OpenDNS to name a few, so seems something somewhere got really hosed to start blocking every public DNS server I use.
There has to be some "baseline" feed somewhere or some built-in blocking list that I'm not seeing that is causing this to occur.
FWIW, I am writing a script that will pull each of the lists and check for public DNS IPs in there, so maybe I can find something there, but again, if the list is disabled, then why would it matter if it contained the IPs or not?
@Gertjan said in pfBlockerng Stopped Packet Forwarding:
and you are not using the resolver in the default Resolve mode, but you've decided to forward your DNS requests to "8.8.8.8"
Not sure what you mean here to be honest. I have the firewall set to use public DNS and not the ISP crap DNS, and I told unbound to use system DNS as it's forwarder, is this what you mean by not using the "default resolver"? If so, what is the "default resolver"?
-
@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
There has to be some "baseline" feed somewhere or some built-in blocking list that I'm not seeing that is causing this to occur.
pfSense doesn't come with pfBlocker installed, and doesn't have any 'list'.
Remove pfBlocker entirely, and the issue is gone ?@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
And it's not just 8.8.8.8, it's Cloud9, Cloudflare, Google, and OpenDNS
I can activate such a behavior with one click in pfBlockerng :
and if your LAN device don't use pfSEnse as a DNS, but use some DNS grabber like DoT or DoH then would look like "nothing would work anymore".
@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
FWIW, I am writing a script that will pull each of the lists and check for public DNS IPs in there, so maybe I can find something there, but again, if the list is disabled, then why would it matter if it contained the IPs or not?
Exact : no.
Although I was just giving an example.
IP and DNSBL list can get poluted by IP or host names that don't belong in them.
It can happen, it has happened in the past. But it would be a rare situation.@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
Not sure what you mean here to be honest.
I mean : if the reolsver is in forwarding mode, and you forward to, for example 8.8.8.8 (and 8.8.8.8 was blocked by an pfBlocker IP list) then you would have DNS problems.
@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
public DNS and not the ISP crap DN
pubic DNS ? You mean you are resolving then ? That is : Root servers => TLD servers => domain name server, which is the resolver's default behavior (the resolver wants to resolve ^^)
"default resolver" : Unbound is a resolver.
It doesn't use some DNS server avaible out there. So no "ISP DNS", no "8.8.8.8" ,no "1.1.1.1", none of it.
Unbound, by default, and that's the way Netgate has set it up for you, uses the Internet's one and only standard / official / the one and only DNS system.But back to the
pfBlockerng Stopped Packet Forwarding
You understand by now that pfBlockerng doesn't do anything.
It can place firewall rules for you - on selected interfaces, or as a Floating rule :Here is an example : what I use :
My example works the other way around :
This rule accepts (passs) all OpenVPN UDP traffic from any IP in Europe.
( and the rest of the planet gets blocked )
This rule permits me to contact my pfSense OpenVPN server, as long as I stay in Europe.Normally, you could make the rules to 'log' so you can see what LAN device was hitting what IP on which list ... And if you know the name of the list you're close to a solution.
-
@Gertjan Public DNS, i.e. Google, Quad9, etc as such:
I suppose I could use root DNS servers, that's often much slower on the uptake, but not impossible.
So in the context of disabling lists and such, here's the DoH/DoT/DoQ status. Disabled or not it's still blocking Quad9, which I can't correlate that to the settings.
I fully understand pfBlocker is about firewall rules for IP blocking and unbound hooks for DNS blocking, never disputed that.
So with everything turned down, this is what I still see:
As you can see from the logs, 9.9.9.9 is still being blocked even though it's not selected on the DoH/DoT/DoQ list, which is disabled anyway, and all my lists being disabled. So something has changed in the basic configuration of pfBlocker, or there's some other setting I'm missing.
I understand your stance that by default pfBlocker does nothing, but that's not the behaviour I'm witnessing after toggling all the list controls off, so there's still something missing. Short of nuking pfBlocker and considering persona non-grata, not sure what else I can do.
BTW, 30 years in IT both infrastructure and networking tells me there's no magic coincidence or just "oh well, something happened", everything has a reason behind it, I'm just not seeing that reason right now.
-
DNSBL_IP is a firewall aliastable that is created from a DNSBL tab option that collects any IPs found in any of the enabled DNSBL feeds and adds those to this aliastable. You could disable that option, or remove the feed that contains that IP, or from the Alerts/Reports tab, suppress or whitelist that IP.
-
There is an alias, used by a firewall rule, on the firewall.
It's called pfB_DNSBLIP_v4 and contains all DNS IPs.
Btw : the interface is WAN ?@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
I understand your stance that by default pfBlocker does nothing, but that's not the behavior I'm witnessing
Go figure that @BBcan177 just proved me wrong.
He mentioned about this :
Knowing that DNSBL list can include IP addresses (and not only host names - this is understandable for DNS servers) and if that option is activated, every IP found will be add to a 'global' pf alias table called "pfB_DNSBL_IP".
This means that you had a "DNSBL IPs" activated (maybe just ones) and it started to include all found IPs, that included all DNS servers, as you would have had also used (ones ?) a list with all these DNS IP addresses ... the same ones you also use to forward to.
This table by itself does nothing (me presuming again) but if it is used in a firewall rule, even when pfBlockerng is de-activated or even removed completely, this rule still functions (?)
This rule is probably placed on the floating rules tab.
Disable this option, remove, if needed, this firewall rule, and your issue is gone.
And I learned something
@jlw52761 said in pfBlockerng Stopped Packet Forwarding:
I suppose I could use root DNS servers, that's often much slower on the uptake, but not impossible.
Forwarding to close by DNS server can be a bit faster.
Using root servers (and then tld servers, and then the domain server of a domain) can have an advantage : DNSSEC can be used as more and more domains support it. DNSSEC protects you against DNS spoofing. Go see a good movie about DNS spoofing and you'll forward never again ^^ -
@Gertjan I specifically chose my list of public DNS servers becasue they do support DNSSEC, I've seen what DNS poisining can do. I don't need to see a movie I've lived it. Did I mention damned near 30 years in the business? BTW, doing just a root hint forward doesn't do DNSSEC as root hint servers are not DNSSEC complaint yet that.
BTW, in the beginning of the post packet forwarding was being stopped, not just DNS being blocked when pfBlocker was enabled. Through a lot of reboots I was able to get the packet forwarding going again with pfBlocker going, but then found the DNS block.
I did use ICMP from the firewall itself to validate the lack of packet forwarding. I wish the logs would indicate which rule has the "offending" match that caused the block, but it sounds like the process roles all the lists up into a single firewall rule.
I really don't want to tear down all of pfBlocker and start over, but it sounds like I will have to do that. Need to see if I can pull all the data out of a backup so that I have all my lists then can just recreate them as needed.
I'm going to look more into what @BBcan177 mentioned, although I think I am already there since I have disabled all my lists, just not 100% sure.
