Update to 3.2.0_7 breaks DNSBL
-
No other changes made. Updated to 3.2.0_7 from 3.2.0_6 and DNSBL stopped blocking. DNSBL blocks stopped appearing in the Alerts tab from the time pfblocker was updated. I confirm it's not blocking by grabbing a domain from one of the block lists and pinging it from a client machine.
Running pfsense 23.09 with System_Patches 2.2.8 applied.
I've tried the following, and checked for blocking after each step:
- Reinstalled pfblockerNG-devel and ran Reload update;
- Rebooted the firewall;
- In DNS Resolver I disabled Prefetch Support and Serve Expired and restarted unbound;
- In pfblockerNG I disabled Resolver cache and ran Reload update;
- Switched to python mode, ran Reload update and rebooted;
- Flushed the DNS cache in the client machine.
It still isn't logging any blocking and pinging a listed domain goes through. The IP part seems to be working perfectly. All services show as running and nothing unusual in any of the logs I've looked at, with the following exception: the DNS resolver log is showing a lot of THROWAWAY responses from Netgate domains whenever I go to the System Packages tabs and it's now taking much longer for those to resolve if at all.
With the understanding that the problem is likely between the keyboard and chair, this has me stumped and any help is appreciated.
-
3.2.0_7 according to what was changed in the package. _7 changes one line of code.
There have been no other code changes checked in since _6That one line has no impact directly on anything you are mentioning there. Unless you installed/reinstalled with the keep settings turned off I believe.
I've had no issues with the _7 update. Still blocking as expected.
just had to force reload the DNSBLI'm on 23.09 Netgate device in production,
and 2.7.1 on a test machine,
applicable patch applied in both cases.both work.
do you have any "not normal" or error messages in pfblockerng.log, error.log or dnsbl_parsed_error.log
-
@jrey Thanks for the reply and the insight. I spent some time on this and I seem to have it isolated.
This firewall is an "always on" OpenVPN client to an OpenVPN service. I recently enabled the new DCO option in OpenVPN to test. With no other changes, enabling DCO breaks DNSBL, and disabling DCO allows DNSBL to function normally. This seems to be reliably reproducible in my setup.
If this is still the case in a day or 3, I'll post my observation in the appropriate forum section.
Thanks Again,
-
@py said in Update to 3.2.0_7 breaks DNSBL:
the DNS resolver log is showing a lot of THROWAWAY responses from Netgate domains whenever I go to the System Packages tabs and it's now taking much longer for those to resolve if at all.
What log file are you looking at ?
I never saw anything logged on this page :
@py said in Update to 3.2.0_7 breaks DNSBL:
I recently enabled the new DCO option in OpenVPN to test
Just did the same thing.
Never used DCO before. I activated it.
I did not regenerate a new VPN export file.
Tested with my phone the VPN server access : it worked just fine.pfBlockerng keeps on humming on.
I forced reloaded it. All is still well. -
@Gertjan said in Update to 3.2.0_7 breaks DNSBL:
What log file are you looking at ?
That should say "DNS Resolver" and not "System Packages." Not sure how I managed that.
I tried DCO again this morning and it did stop DNSBL from blocking on my setup. I thought it may have something to do with the VPN routing so I tried changing some of those settings but nothing changed regarding DNSBL. DNSBL started blocking again only after I disabled DCO.
After DNSBL was working again, I did note that DNSBL blocks were logged in the "dnsbl.log" file, but did not show in the GUI "Reports" -> "Alerts" for some time, maybe about 20-30 minutes. The blocked domains during this time that didn't show up in the GUI remain missing from the GUI.
I don't see anything obviously wrong or "not normal" or error messages in pfblockerng.log, error.log or dnsbl_parsed_error.log. I'll post them if you think it may help.
-
@py said in Update to 3.2.0_7 breaks DNSBL:
I did note that DNSBL blocks were logged in the "dnsbl.log" file, but did not show in the GUI "Reports" -> "Alerts" for some time, maybe about 20-30 minutes.
Did you restart anything else in the 20-30 minute window?
this is unrelated to _7 specifically. Tracking down another issue, that people are associated to _7 because that is the last package update that has been applied, but i looks like it has more to do with something related to the system.
can you execute the following two commands and post the results please
ls -l /usr/bin/tai*and
ps alx | grep tail -
@jrey said in Update to 3.2.0_7 breaks DNSBL:
Did you restart anything else in the 20-30 minute window?
No. However, i did just recently try DCO again with the same results. This time I did restart DHCP, DNSBL & unbound in that order with the same results. I waited for each to restart fully before restarting the next one. After disabling DCO it still took about 20 minutes for DNSBL logging to start showing anything, even though pinging listed domains did result in them being directed to 10.10.10.1. It also takes a few minutes for DNS to start resolving.
@jrey said in Update to 3.2.0_7 breaks DNSBL:
this is unrelated to _7 specifically. Tracking down another issue, that people are associated to _7 because that is the last package update that has been applied, but i looks like it has more to do with something related to the system.
You know more bout this than I do I'm sure, but I tend to agree.
ls -l /usr/bin/tai*
Results in:
-r-xr-xr-x 2 root wheel 22256 Nov 1 16:57 /usr/bin/tail
-r-xr-xr-x 2 root wheel 22256 Nov 1 16:57 /usr/bin/tail_pfbps alx | grep tail
Results in:
0 13357 24980 18 31 0 13400 3000 wait S - 0:00.00 sh -c ps alx | grep tail 2>&1
0 13761 13357 26 29 0 12832 2468 piperd S - 0:00.00 grep tail
0 59459 1 18 20 0 12916 2492 kqread SC - 0:00.23 /usr/bin/tail_pfb -n0 -F /var/log/filter.log
0 59972 1 11 20 0 12916 2428 select S - 0:00.19 tail_pfb: system.fileargs (tail_pfb) -
@py
When you use your OpenVPN connection, it's the OpenVPN server running on pfSense (DCO activated) and some remote OpenVPN client on a phone, right ?Is your resolver (unbound) set up like this :
?
"All" isn't laybe the best setting, but it will listen on my VPNS interface (my OpenVPN server interface).What is the (IP) DNS used by the OpenVPN client ? It should be the IP of the OpenVPN interface.
When you do a dig or nslookup on the OpenVPN client, did you saw the reply in the dns_reply.log ? -
@Gertjan This firewall (pfsense) is an "always on" OpenVPN client to an OpenVPN service. There is no VPN server and no remote VPN client on this firewall.
Unbound is setup like this:
OpenVPN is setup like this:
I'm not seeing any entries in the dns_reply.log since Dec 1 when I updated pfBlockerNG to the _07. In fact, the only entries in that log file are for about an hour on Dec 1.
-
@py said in Update to 3.2.0_7 breaks DNSBL:
dns_reply.log
on the top list, the 10.10.10.1 is not even a DNS server -- that is the DNSBL web page. Don't select that in your list. (I've actually created a patch on my system to just remove that from the list for testing, so in cases where "ALL" is selected it can't possibly be used. )
is the DNSBL service running? have seen a couple of other cases where the DNSBL isn't starting properly because the lighttpd (web service) is encountering an error that trips things up. Sometimes it says it running and it is not, other times is just says it is not running.
Rather than re-typing, here are a couple of threads that have some steps you could look at.
https://forum.netgate.com/topic/184306/pfb_dnsbl-wont-start-in-clean-installation/3?_=1701790524595
https://forum.netgate.com/topic/184032/pfb_dnsnl-pfblockerng-dnsbl-service-won-t-start/15?_=1700748361493
the part you would be most interested in this this
from a shell (ssh in) change to this directorycd /usr/local/etc/rc.dthen run this
./pfb_dnsbl.sh restartwhat do you get? you might see entries that are not normally logged. Have seen various errors in here from lighttpd that are fatal, that don't get trapped properly and the service is then hit or miss to being operational.
don't worry about message that contain "ssl.cipher-list is deprecated" they are not fatal. -
I have not had a chance to review the links and will do so later today or this evening.
./pfb_dnsbl.sh restart
Resulted in "ssl.cipher-list is deprecated. Please prefer lighttpd secure TLS defaults, etc..." with no errors or strange messages.
I then enabled DCO and confirmed DNSBL was not blocking.
I then ran:
./pfb_dnsbl.sh restart
again and got the same result.
I then disabled DCO and DNSBL still would not block. I noticed that I didn't have DNS resolution and checked the DNS Resolver log which was continuously priming many of the same domains for over 15 minutes . I restarted unbound and both DNS and DNSBL began working within a minute or two, however blocked domains are not always logged in DNSBL.This makes me suspect it may be more of a problem with unbound, or perhaps with how the VPN is handling DNS, than with DNSBL.
-
@jrey I reviewed the linked posts and they clarified some things, thank you.
After reviewing the DCO limitations I'm giving up on that until it's ready for prime time. At least one of the limitations makes it completely unsuitable for my environment and I should have been more careful before trying it out.
Right now DNSBL is working, but it won't log many of the blocked domains. It doesn't seem to log any blocked domains from one machine in particular. It does seem to be logging all IPv4 blocks and Whitelisted domains. So far I've tried the following:
- Restarting pfb_dnsbl.sh as previously suggested;
- Restartng dnsbl from the Services page;
- Restarting unbound from the Services page;
- Disabling DNSBL and pfBlockerNG and reinstalling;
- Clearing all the log files and restarting DNSBL;
- Restarting DHCP.
No change.
At this point I'm considering disabling "Keep Settings", uninstalling pfBlockerNG, rebooting pfSense and reinstalling pfBlockerNG. But reconfiguring is at least 2 hours of work.
Any suggestions welcome. You all have been a big help already and I've learned a few things.
-
@py said in Update to 3.2.0_7 breaks DNSBL:
At this point I'm considering disabling "Keep Settings", uninstalling pfBlockerNG, rebooting pfSense and reinstalling pfBlockerNG. But reconfiguring is at least 2 hours of work
Don't.
If there is an issue, its not the code or the scripts.
Your "pfSense", "unbound" and "pfBlockerng" is the same as mine. The same like 'every bit ....' where a bit is a bit in a byte.
The only thing that is different between your pfSense and mine is : the settings ..... (that you want to keep).
So, win 2 hours direct. Re installing is only a solution if you suspect that files are damaged, something that nearly never happens. If it happens, its often the disk dying on you.I'm using right now :
I have pfBlockerng with some small DNSBL, and I can see that this
tail -f /var/unbound/var/log/pfblockerng/dns_reply.log
keeps on scrolling - it never stops.
When I open a browser on my phone - with the OpenVPN client activated and logged into pfSense, I can see that it also resolves DNS requests from my phone .... the dns_reply.log log scrolls even faster.Keep in mind : pfblockerng can only do its job for DNS requests it sees. That is, DNS records that are handled by unbound, the resolver. DNS traffic not handled by the resolver will not be handled by pfblockerng neither.
I'm using the pfBlockerng 'pythion' mode (of course).
My pfSense is a Intel x86 23.09 running on a '4100'.
@py said in Update to 3.2.0_7 breaks DNSBL:
Resulted in "ssl.cipher-list is deprecated. Please prefer lighttpd secure TLS defaults, etc..." with no errors or strange message
and @jrey :
I saw the lighttpd message.
Good news : Google saw the message. Here is the info needed to blast the warning./usr/local/pkg/pfblockerng/pfblockerng.inc
The way pfblockerng is creating the lighttpd config file is a bit outdated.
Look ,for the 4 (four !) occurrences of "cipher-list".
Replace the lines with the more modern, according the redmine sited above.
Btw : now you're edting the source ..... if things go bad : reinstall pfBlockerng now makes sense ^^
Btw : the message from lighttpd while (re) starting is not a problem. It's just a warning. -
@Gertjan said in Update to 3.2.0_7 breaks DNSBL:
the message from lighttpd while (re) starting is not a problem. It's just a warning.
the cipher message in the file, is exactly what I said, not fatal don't worry about it. That wasn't the message causing other folks to have issues. The "other" messages if they are there "are not logged" so you can view them in a log file, and that's a problem. They do cause problems. By checking, the OP, has simply ruled those potential issues out.
I agree, with the "Don't." - in regards to uninstalling/reinstalling.
I disagree somewhat with this: "If there is an issue, its not the code or the scripts."
There are literally 100s of issues, and unless there is an investigation, nothing will ever change.For example, the one line of code, that changed in pfBlockerNG _7 is being perceived by several as the cause of their "new" problem. Somewhat true, but not because of that line of code, as the scope of when it is executed is very limited.
There is an issue, because of something else that gets exposed when the package is updated. (but the pfb code has changed). Specifically on a system, where the underlying system versions have changed. I can recreate this "issue" all day long (in a Break/fix test cycle). Does it have anything to do specifically with the change in pfBlockerNG _7, yes, but also no.
Yes, but only because the update was applied to an underlying system that has changed. No because the code in pfBlockerNG that is causing these issues, hasn't changed, but is being impacted by what has changed under the hood.Does it impact logging? Very possible, however there is not enough data to make that determination conclusively yet. When "broken", logging stops /delays as described. But there are also "other" clues when it is "broken", the OP in this case doesn't seem to have any of those "other" clues. The lack of other clues, of course may or may not be a result of settings.
-
@jrey @Gertjan Okay, so I'm not reinstalling pfBlockerNG.
Thanks for telling me about the dns_reply.log because mine is empty and stays that way.
Hardware is an Intel Xeon E5-4667v4 with 18 cores on an ASRock X99 Taichi, and the NICs are Intel PRO/1000 on a PCI card.
error.log logged 4 of these since I cleared the log yesterday:
PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 12/6/23 03:16:41 ] Failed validation [ - ]dnsbl.log and unified.log frequently log a block that is not reflected in the GUI "Alerts" tab, and they never log a block from one of my workstations, even though the ping itself is blocked.
Updates are set at 04:15, so the entries from the dnsbl_parsed_error.log must be happening during the update:
I don't see any errors in the VPN log and it seems normal to me.
The DNS Resolver log frequently shows THROWAWAY responses which I don't recall seeing before this incident:
I tried python mode for DNSBL about a year ago but ran into trouble with that and my research showed those problems were inherent in the python implementation so I stopped using it since. I've long since forgotten the specifics.
-
@py said in Update to 3.2.0_7 breaks DNSBL:
PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 12/6/23 03:16:41 ] Failed validation [ - ]
^^ in the error.log and dnsbl_parsed_error (screen image) time stamps don't match.
in the pfblockerng.log find this section in the latest processing
===[ DNSBL Domain/IP Counts ]what is the number for total ?
and what do the numbers under this section look like
pfSense Table Statsand then at very end what is the timestamp here ?
UPDATE PROCESS ENDED [ 12/6/23 10:30:53 ]Can you share your complete DNS resolver settings, General and Advance.
also the from the DNSBL config page. (just the sections DNSBL, DNSBL Webserver Config, and DNSBL Configuration)Thanks
-
@jrey Those timestamps were not intended to match.
===[ DNSBL Domain/IP Counts ] ===================================
3632259 total
pfSense Table Stats
table-entries hard limit 400000
Table Usage Count 36389UPDATE PROCESS ENDED [ 12/6/23 04:34:48 ]
DNS Resolver settings:
DNSBL Settings:
Thanks!
-
@jrey said in Update to 3.2.0_7 breaks DNSBL:
on the top list, the 10.10.10.1 is not even a DNS server -- that is the DNSBL web page. Don't select that in your list. (I've actually created a patch on my system to just remove that from the list for testing, so in cases where "ALL" is selected it can't possibly be used. )
I don't suppose you'd care to share that patch? I have Unbound set to ALL so it's grabbing the 10.10.10.1 entry as well.
- draco
-
Of course I could.
but a counter question to this statement
I have Unbound set to ALL
might be:
Why? -
if you want to before starting, you can clear pfblockerng.log and error.log on the
firewall > pfBlockerNG > Logs (select the log to display and there is a a trash can on the right) will just make it cleaning to see new entriesFirewall > pfBlockerNG > DNSBL
Change DNSBL Mode to "Unbound Python Mode"
for Options that appear after doing thatSelect (Check Enable) DNS Reply Logging, DNSBL Blocking, HSTS Mode
Save DNSBL SettingsThen DNS Resolver config
Enable Python Mode (if not already check)
Module order is Pre Validator
Python Module Script is pfb_unbound
SAVE (button at bottom)
(then scroll back up top page) Apply ChangesThen Firewall > pfBlockerNG -> Update
Force -> Reload -> DNSBLProcess start time ?
Process End time?
new errors in either of the above mentioned log files ?test and report the changes you see.
