DNS Resolver doesn't work with my university domain.
-
I disabled DNSSEC and voilá!
[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: dig ufscar.br +trace ; <<>> DiG 9.16.23 <<>> ufscar.br +trace ;; global options: +cmd . 86400 IN NS e.root-servers.net. . 86400 IN NS f.root-servers.net. . 86400 IN NS d.root-servers.net. . 86400 IN NS c.root-servers.net. . 86400 IN NS a.root-servers.net. . 86400 IN NS k.root-servers.net. . 86400 IN NS l.root-servers.net. . 86400 IN NS j.root-servers.net. . 86400 IN NS m.root-servers.net. . 86400 IN NS h.root-servers.net. . 86400 IN NS i.root-servers.net. . 86400 IN NS g.root-servers.net. . 86400 IN NS b.root-servers.net. . 86400 IN RRSIG NS 8 0 518400 20220919050000 20220906040000 20826 . Hx5LBSTiDo3WZuqW/D0FLRM8ATgzAv+gwpQ353b0q566Vkeg367wNdWO oey2OeTl8yQGcKFR2Pr5JlgzKrUMqmvzavCQ/XTGJFVKhVJPlyf2aTYt lTHRYDIL6ZDlGWcunwM3D+q182+HqTSGRu0lY0LU38ObcTjTJut632Wo HJBpq4VhrNhoE08GGn9gOYniOvf6Sltd2aCmVuyvKZpLrcsenEp5vrCn KYAlFK4ZpIWfgliVcdJ9aGHftU6XxjUgim9znlKAfBWZZBpB0nBDE6Sn +m9B1MVB7bysMrDPne95Ysa0mEav7nj8TXx5DJDY9zY5roxySFCWEXR3 9Xf06w== ;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 695 ms br. 172800 IN NS a.dns.br. br. 172800 IN NS b.dns.br. br. 172800 IN NS c.dns.br. br. 172800 IN NS d.dns.br. br. 172800 IN NS e.dns.br. br. 172800 IN NS f.dns.br. br. 86400 IN DS 2471 13 2 5E4F35998B8F909557FA119C4CBFDCA2D660A26F069EF006B403758A 07D1A2E4 br. 86400 IN RRSIG DS 8 1 86400 20220919050000 20220906040000 20826 . LYCKl//tGrWDYY66d2U+I7pIcJdL/TPiR1TiK6EL3G4Uo4okZEx/J/Su iERyk/zuPzl6ybKW7RfqWRjk9KxBw68jEZY4n+FSBRgDSBno55VSeC5O kGhEI68BX2V6Xt6XYmz/MmbfG2aq4IJ4lDNgnbouE8p8J0l1pxVipcJ0 Sx5+efcTlWt5TXpUcledJKd4LrkIyA7pIj+Eu4JFdIeUromIvCI9UeZ3 4YpPu/z8idCydYVQXpkfJelYM3pzQi6ciAzkNLHcDt3Fw11EGq4WuEUC JzoN8pOWKTKt09O6s3pXuyxmsOsB9gHP3kmVfPFU7SOyNAMhbhzx6Wiq nPj4Tg== ;; Received 765 bytes from 199.9.14.201#53(b.root-servers.net) in 121 ms ufscar.br. 3600 IN NS ns-1946.awsdns-51.co.uk. ufscar.br. 3600 IN NS ns-467.awsdns-58.com. ufscar.br. 3600 IN NS ns-531.awsdns-02.net. ufscar.br. 3600 IN NS ns-1501.awsdns-59.org. ufscar.br. 3600 IN DS 26184 13 2 21CA2F8D54DECD6E41A6EC47957C10029AB6D6A670D31D9E3D0F2791 83D7AAAE ufscar.br. 3600 IN RRSIG DS 13 2 3600 20220919121008 20220905111008 65267 br. equEtuASx+Ulcjtez8+5tBJXIOmJ6FBD3KJuGJ2Y+0Z1n/ddMkW3pgsR sTvKz57Xyr+A/hnts5EPPgiU9MVKfQ== ;; Received 352 bytes from 200.229.248.10#53(e.dns.br) in 10 ms ufscar.br. 14400 IN A 200.136.207.122 ufscar.br. 14400 IN RRSIG A 13 2 14400 20220906140638 20220906080638 6215 ufscar.br. oFYAjIlBisxx/2ZeOicEdyazuDxqxNpNE8VYH4FklBYokaoZp4uZMXXA AMJDViJxz7u/eEZqUUN6+12gxn5OSw== ;; Received 159 bytes from 205.251.199.154#53(ns-1946.awsdns-51.co.uk) in 9 ms [2.6.0-RELEASE][admin@pfSense.home.arpa]/root: ping ufscar.br PING ufscar.br (200.136.207.122): 56 data bytes 64 bytes from 200.136.207.122: icmp_seq=0 ttl=55 time=17.377 ms 64 bytes from 200.136.207.122: icmp_seq=1 ttl=55 time=17.592 ms 64 bytes from 200.136.207.122: icmp_seq=2 ttl=55 time=17.514 ms ^C --- ufscar.br ping statistics --- 4 packets transmitted, 3 packets received, 25.0% packet loss round-trip min/avg/max/stddev = 17.377/17.494/17.592/0.089 mswhile now it works what do you mean by pointing pfsense to itself for DNS?
And one thing I'm noticing while browsing, when accessing a site for the first time it's really slow, images take a long time to load and the site fails the first time loading.
-
@fandangos need to figure out why dnssec is failing.. I show it all good via online testing, and I am not having any problems doing dnssec for it
https://dnssec-debugger.verisignlabs.com/ufscar.br
https://dnsviz.net/d/ufscar.br/dnssec/
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
I think I solved the DNS Resolver slowness by increasing Outgoing TCP Buffers and Incoming TCP Buffers to 50, plus Number of Queries per Thread to 2048 but what made it faster was enable Prefetch Support and Prefetch DNS Key Support.
But enabling DNSSEC kills the access to ufscar.br indeed.
Looking at reddit:
https://www.reddit.com/r/PFSENSE/comments/vuhcni/dnssec_broken_recently/"It's always been broken for me on CE 2.6. You probably just started noticing it with certain websites."
Could it be a Pfsense 2.6 issue?
-
@fandangos let me fire up my 2.6 vm and query that domain. But seems unlikely.. I am running 22.05 and there is a difference in the unbound between them..
BRB..
Ok I fired up my 2.6 VM of pfsense, not having any issues with that domain with dnssec enable and resolving.
To validate dnssec is being used, I queried for 2 known bad fail validation fqdn.. And both of them fail..
So clearly its not a specific hey that domain doesn't work in 2.6 or that dnssec doesn't work in 2.6..
Your issue "could be" something as nefarious as some sort of dns interception breaking dnssec.. I stress "could be"! But more likely than not it could just be an issue with geographic location and what NS your hitting and an issue with with dnssec.. But 2 different dnssec sites show nothing wrong with dnssec for that fqdn.
Have to dig a bit deeper to find out what is causing the problem..
edit2: here is a quick and simple test for interception.. Do a directed query to say 1.2.3.4 for something like www.google.com.. that should just time out, if you get a answer - then your isp is doing some very blatant dns interception
[22.05-RELEASE][admin@sg4860.local.lan]/root: dig @1.2.3.4 www.google.com ; <<>> DiG 9.16.26 <<>> @1.2.3.4 www.google.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached [22.05-RELEASE][admin@sg4860.local.lan]/root:edit3: can we test for dnssec working on other sites... Its possible maybe this is the only site you have run into that is actually dnssec..
So if you query
sigok.verteiltesysteme.netYou should get back an IP.. If you query
sigfail.verteiltesysteme.net
It should give back servfail if you have dnssec enabled.
You can also just go like here, do you get a thumbs up that your using dnssec when you have it enabled?
-
Ok
[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: dig @1.2.3.4 www.google.com ; <<>> DiG 9.16.23 <<>> @1.2.3.4 www.google.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached [2.6.0-RELEASE][admin@pfSense.home.arpa]/root: dig @1.2.3.4 www.google.com ; <<>> DiG 9.16.23 <<>> @1.2.3.4 www.google.com ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reachedwithout and with DNSSEC.
[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: dig sigok.verteiltesysteme.net ; <<>> DiG 9.16.23 <<>> sigok.verteiltesysteme.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40230 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;sigok.verteiltesysteme.net. IN A ;; ANSWER SECTION: sigok.verteiltesysteme.net. 17 IN A 134.91.78.139 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Tue Sep 06 07:24:06 GMT 2022 ;; MSG SIZE rcvd: 71 [2.6.0-RELEASE][admin@pfSense.home.arpa]/root: dig sigfail.verteiltesysteme.net ; <<>> DiG 9.16.23 <<>> sigfail.verteiltesysteme.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 51731 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;sigfail.verteiltesysteme.net. IN A ;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Tue Sep 06 07:24:21 GMT 2022 ;; MSG SIZE rcvd: 57but ufscar.br
[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: dig ufscar.br ; <<>> DiG 9.16.23 <<>> ufscar.br ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 6332 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;ufscar.br. IN A ;; Query time: 1159 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Tue Sep 06 07:25:11 GMT 2022 ;; MSG SIZE rcvd: 38Servfail.
The thing is, this is mine and my wife's university website, it's not something I can just stop using it so the solution would be just disabling DNSSEC for now?It looks like some problem beyond my setup from all the tests we did so far.
-
@fandangos well the simple solution sure would just be turn off dnssec, but other option for specific domains that you need to get to, but for reason can not determine yet dnssec is failing.
A way to allow access to that domain, and still use dnssec for other domains would be setting
domain-insecure:
In your advanced options box..
so for example you see how sigfail.verteiltesysteme.net fails.. if I set that fqdn as domain-insecure it doesn't do dnssec on it and now get back an IP
This should allow you to get to your website, but still be doing dnssec for other sites - and work around until you can figure out what is causing the problem.
btw does for example www.ufscar.br work?
;; QUESTION SECTION: ;www.ufscar.br. IN A ;; ANSWER SECTION: www.ufscar.br. 14400 IN CNAME web-07-nova.ufscar.br. web-07-nova.ufscar.br. 14400 IN A 200.136.207.122 -
@fandangos said in DNS Resolver doesn't work with my university domain.:
removed pfBlockerNG-devel, disabled python.
Went into System / General Setup, removed all DNS servers and set use local DNS only, ignore remote.
Disabled DNS Query Forwarding in DNS Resolver.
Because that's the way I want it to work.Rebooted my PC with Pfsense.
I'm running a small BGA Intel board with 2.6.0-RELEASE.Went into DNS lookup and search for ufscar.br and :
Host "ufscar.br" could not be resolved.If I look for google.com it will give a result in 60ms
Result Record type
142.250.219.46 A
2800:3f0:4001:822::200e AAAAlike so.
Not sure what else to try.
@johnpoz is the resident DNS expert here, so you are in good hands troubleshooting with his assistance.
Reading the threads above I see that the problem has at least been isolated down to DNSSEC. In the beginning there were three possibilities: (1) DNSSEC; (2) pfBlockerNG-devel; and (3) Python Mode with
unbound. By eliminating options and testing with a vanilla setup you eventually found DNSSEC as the apparent problem. Now you can concentrate efforts to figuring out why that option is failing. -
@johnpoz said in DNS Resolver doesn't work with my university domain.:
@fandangos need to figure out why dnssec is failing.. I show it all good via online testing, and I am not having any problems doing dnssec for it
https://dnssec-debugger.verisignlabs.com/ufscar.br
https://dnsviz.net/d/ufscar.br/dnssec/
I saw the same thing : these two looks good.
I refreshed the latest scan on dnssec-debugger.verisignlabs.com - still all ok.But : I did a zonemaster.net scan :
https://www.zonemaster.net/result/e74f48762d19101d
and that looks messy ....I count about twenty "DNSKEY tags", so the zone size must be ... huge.
Something is definitely not ok.
The domain name admin is playing with zone ZSK resigning ? ( and oh boy, he fails ).If "ufscar.br" is DNSSEC singed, and it is, and DNSSEC verification fails, then all is well : "no answer" is the good answer.
-
@bmeeks yeah could take a bit of digging to figure out why dnssec is failing, if its working for other sites, it doesn't make a lot of sense
But maybe its not really working all that well on all other sites??
@Fandangos maybe you should check that the time and timezone are correct or at least close to being correct..
The RRSIG has a 2 times there, you have to be within that time frame etc.. Or dnssec would fail one is the Expiration the other is Inception of that signature.
It could get really tricky because different NS could use say a wider range of when the signature is good, and if your time is not off too much site A works, but if maybe site B has smaller window of the time of the signature.. So it fails..
edit: Looking at the report @Gertjan linked too - this warning jumps out at me
"RRSIG with keytag 26184 and covering type(s) DNSKEY has a remaining validity of 17732 seconds, which is too short. "
So if they are using really short validity ranges and time is off on pfsense that could explain the problem..
-
This time situation is always problematic. I have a unraid NAS running here that suffers the same problem.
I live near São Paulo (the city) within São Paulo the state.
If I select America/Sao Paulo using 2.pfsense.pool.ntp.org the time is wrong by 3 hours (minus 3 hours).
Right now is 12:00 and it shows 9:00 AM.
I have the same behavior on my NAS using Unraid that uses another ntp pool.So to get the right time I have to use ETC/GMT+0 which, looking at the clock on my phone that register with the cell towers and services it is 4 minutes behind.
So that might be the problem. There's no correct time zone to be selected and the one that is closer is 4 minutes behind.
But.. 17732 is 289 minutes, so it's not it but it would be good to actually fix the ntp situation, I've never found a solution for this.
EDIT:
Even using 0.br.pool.ntp.org server in Brazil follows the same problem: 3 hours behind with America/Sao_Paulo.EDIT 2:
Looking online the correct time zone for São Paulo is Etc/GMT+3, and this results in the same problem using America/Sao_Paulo, 3 hours behind. It's like if GMT is wrong.Does NTP`pool time uses anything within the BIOS?
ETC/UTC timezone also gives the 4 minutes behind time zone.
-
EDIT wasn't working.
EDIT 3:
https://forum.netgate.com/topic/158490/pfsense-php-time-is-incorrect-in-america-sao_paulo-time-zone/
It seems somebody else saw this and it should have been fixed in 2.5 but it seems it was not. -
@fandangos said in DNS Resolver doesn't work with my university domain.:
There's no correct time zone to be selected and the one that is closer is 4 minutes behind.
Huh that makes zero sense to be honest.. There are no timezones on the planet that I am aware of that are 4 minutes different.. That would be insane..
I know of some that are 30 minutes, and some that are 45 minutes, and I believe there is one that has 15 minute difference other than just the 0 hour differences..
Clearly you have something wrong with time.. But 4 minutes? That seems insane..
So looking up the timezone for São Paulo, and my time zone CDT, and looking on my phone, everything is pretty much dead nuts on for correct time.
edit: doesn't really matter what zone name you use be it +1, -3 etc.. Worse case is switch from Standard to Daylight savings could be off if zone your actually in is different than the zone you choose. But I am lost to a 4 minute difference - that should never be the case no matter what zone your in, other than just the clock being off and not in sync with a ntp server.
-
@johnpoz
I know it doesn't make sense.My guess is this: the bios time is wrong by 4 minutes and the NTP code only fixes the date and hours, leaving minutes and seconds up to the BIOS clock.
Tomorrow I'll take this PC down and check the BIOS. -
@fandangos no not how it works, ntp would take it to the ms to be honest.. You got something weird that is for sure..
But it can take a bit to adjust the time using ntp..
-
ok check this out, I believe my ISP blocks port 123 or it's block NTP access:
So, my BIOS is 4 minutes behind and since Pfsense can't access the NTP servers it just changes the time zone adding or subtracting a few hours.
Now I used to have this problem with a FPGA system that runs linux and I used to solve this by this nice startup script in bash that sadly will not work on pfsense:
RET=-1 while [ ${RET} -ne 0 ] ; do ping -c1 google.com RET=$? sleep 1 done date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"so I've found a small utility on android called Time Server that uses GPS and is able to run a server but on port 1234.
I wasn't able to add the port to the system like 192.168.0.20:1234 but I was able to set 192.168.0.20.
The error is that the : makes it a invalid ipv4 or ipv6 address.I don't see on the system logs anything about a successful NTP sync with the app but now the time is correct.
If I look at/var/etc/ntpd.confIt's like this:
# Upstream Servers server 192.168.0.20 iburst maxpoll 9And now look what works:
[2.6.0-RELEASE][admin@pfSense.home.arpa]/var/log: ping ufscar.br PING ufscar.br (200.136.207.122): 56 data bytes 64 bytes from 200.136.207.122: icmp_seq=0 ttl=55 time=18.258 ms 64 bytes from 200.136.207.122: icmp_seq=1 ttl=55 time=18.267 ms 64 bytes from 200.136.207.122: icmp_seq=2 ttl=55 time=18.256 ms 64 bytes from 200.136.207.122: icmp_seq=3 ttl=55 time=18.279 ms ^C --- ufscar.br ping statistics ---So it's indeed a time sync problem that prevents DNSSEC to work.
Also, changing to America/Sao_Paulo actually set the clock right this time, so, it was fixed in pfsense 2.5.
Now only if there's a way to bypass the port 123 block from my ISP so I could normally use NTP servers because honestly I don't know what exactly fixed the problem.
I also selected all the interfaces under NTP settings but that was it, the android server and selecting all interfaces something fixed it for now. -
I'm not sure what fixed it, really here I believe (not sure) is the moment that the clock was fixed:
maybe it has nothing to do with the android time server, I don't know.
The log doesn't give much info. -
@fandangos said in DNS Resolver doesn't work with my university domain.:
I don't know.
The log doesn't give much info.I know. It looks like that.
But reading and understanding logs needs a learning phase ;)Lines like this :
means that syncing is progress
Hardware that uses an RTC that is some seconds off every day isn't a big deal, but if the hardware RTC can't be set using ntpd, and it start to lag behind a lot, then you have a problem.
Your hardware is nice for a game device, not a router/firewall.If ntp can't set the hardware RTC clock, then on every boot your system clock is lagging behind even more.
ntpd needs time to sync back to the real time.
During this sync time, everything that uses 'time' can be off.
Like DNSSEC. And more.
So, consider :
Not rebooting pfSense ;)
If you reboot, set the time manually (as ntpd can brute force the time, it will go slowly).
And when your done with it : change hardware, with a good RTC.
Do check the battery, if there is one.The exact time is an important resource these days.
This is strange :
as the ntpd should stay in memory 'all the time'.
-
The thing is, it never synced before (I have this pfsense setup for months now) and now it's not syncing again.
If I look at Status / NTP
It's not synchining and I've added several pools to try.
Here's the log for the NTP service trying several servers without any success:
https://pastebin.com/Hqna2gvQ [too big to post here]The lan ip is the android server that is off for now.
I have no idea how it managed to sync that only time but as you can see something (proably my ISP) is blocking it because I see this same problem with windows server, with unraid.. and so on.
-
@fandangos have you contacted your isp about this? Blocking ntp is very detrimental to proper option many things these days. You need actuate time for many things to work correctly.
If they block external ntp - do you they run their own ntp you can sync with, atleast that would be something. How many isp block outbound smtp, but they provide you with a smtp server you can use, etc.
edit: as a work around you could run your own ntp like you were doing off your phone. I run one via a raspberry pi and a gps hat. Total project should cost under $100.
So out of curiosity what is the isp device in front of pfsense, is pfsense IP a rfc1918 address or are you getting a public IP on pfsense wan. It could be possible that maybe the isp device is doing something with ntp, and having issue natting it for clients downstream - like pfsense.
I would suggest you contact your isp about not being able to talk to any ntp servers, see what they say. Its possible you will have to escalate past level 1 support to get to someone that understands what your talking about.
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.7.2, 24.11 -
@johnpoz
Yeah, I manage quiet well to even not think : "we've seen it all".
But now I say it : "@Fandangos, your ISP is definitely not expensive enough" ;)Not relaying ntp ..... what's next ? Blocking https and pop, why not imap ? No ? Block SSH as it is scary.
Or has 'ntp' been thrown on some s@nction list ? ( You see what I mean ?)
