Insanely weird issue with DNS resolution to www.cdc.gov

Gertjan

@gregarios said in Insanely weird issue with DNS resolution to www.cdc.gov:

@gertjan My openDNS settings are set correctly since directly using them from my computer works fine. Only when the router uses forwarding does it not work.

The 'DNS' settings might be ok.
But OpenDNS - chekc your OpenDNS account - can use 'filters' like : nothing, no adds - no spam - no sex - no bitcoin - no whatever.

Again, check the (non filtered) OpenDNS resolver manually : https://cachecheck.opendns.com/
It resolves.

Depending on what you use as for a filtering
If your not uses any filtering on the OpenDNS side, then why use OpenDNS ? Use the real, build in stuff, : the official 13 root servers == use the resolver as your Resolver. Easy to set up, easy to maintain. Plain works.

Btw : Finally, the admins of cdc.org finally ditched DNSSEC support.
https://dnsviz.net/d/www.cdc.org/dnssec/

gregarios

@gertjan As I said... my setting on the router and my computer are correct. I have tested it by connecting successfully to the CDC using OpenDNS numbers on my computer. If I use the router's DNS that forwards to the same OpenDNS IP addresses, it does not work. If my filters were too strict it would not work when I connected to it from my computer.

I'll have to retest this though since now CDC has dropped their DNSSEC. Yesterday it was active and in error mode.

Gertjan

@gregarios said in Insanely weird issue with DNS resolution to www.cdc.gov:

I'll have to retest this though since now CDC has dropped their DNSSEC. Yesterday it was active and in error mode.

I saw the same thing.
First time it look clean like that : just a non DNSSEC domain.

No DNSSEC errors.

OpenDNS is a Resolver, you have to check with their policies, but I can image that if a domain announces "I have DNSSEC" that it should be implemented correctly. If not, well, it should fail.

If you were using the Resolver as a resolver, you could use an option in the custom box that excludes cdc.gov from any DNSSEC issues.
Maybe OpenDNS offers also such a possibility.

edit : Oh sh*t. OpenDNS has become less 'open' : it's bought by Cisco .... well. What to say : great ! (?)

johnpoz

I have not seen that - they still have a shit ton of errors.. If you do not know how to do dnssec - then you shouldn't even attempt to do it.. Which is what I suggest they should do.. Since clearly they have no idea how to do it correctly.

Its still a horrible mess as of test just did
2021-05-17 09:56:53 UTC

Gertjan

@johnpoz :

Strange, asked a new "Analysis" this morning, and it looked some what ok.

Re tested again just now, and it's pure BS again !

Now, I'm just a European guy, and I know one has to go to "cdc" when things go bad ( learned this from the Walking Dead ? ).
It looks like an important governmental agency to me.

edit : oh, wait : www.irs.gov isn't any better. And no one is complaining .....

gregarios

Just more proof the currnt U.S. government doesn't react in the slightest to any "warnings" — only flat-out catastrophic failure.

Gertjan

@gregarios
It's just an admin ** messing up.
I don't think using or adding DNSSEC is a governmental decision.
And give them some time, this issue started more then a year ago, not everybody (admins) have been replaced yet ;)

** On both sides : pfSense, using default DNS settings, works just fine.

DrPhil

@gertjan

To answer your question

If your not uses any filtering on the OpenDNS side, then why use OpenDNS ?

I am using their free family filter DNS. 208.67.222.123.
(I am a home user and need the filtering).

Use the real, build in stuff, : the official 13 root servers == use the resolver as your Resolver. Easy to set up, easy to maintain. Plain works.

I would love to. Is there an easy way to setup a filter that I don't have to maintain on an ongoing basis? I am willing to put in the time / effort upfront. I tried pfBlockerNG, but didn't like the experience and gave up.

Btw : Finally, the admins of cdc.org finally ditched DNSSEC support.
https://dnsviz.net/d/www.cdc.org/dnssec/

Whatever they did, isn't working for me yet. And I have a feeling it's not OpenDNS filtering that's causing the issue but an incorrect setup on CDC's side. (could it be that you're looking at cdc.org instead of cdc.gov?)

oh, wait : www.irs.gov isn't any better. And no one is complaining

irs.gov is working fine for me (through my openDNS setup). Whatever is making cdc.gov fail, it does not share with irs.gov

johnpoz

@drphil said in Insanely weird issue with DNS resolution to www.cdc.gov:

irs.gov

I don't show any issues with irs.gov

Now I have not personally seen any issues with accessing cdc.gov - not something I go to every hour or even every day, or week even, etc. But their dnssec is not optimal that is for sure - could it cause issues, possible for sure depending on the resolver, and what specific NS they end up talking to, etc.

It is disappointing to see such a mess that is for sure.. But I am not doing anything specific to not do dnssec for that domain, nor anything that would forward requests for that domain to some other resolver.

Issues in accessing it from where your at on the globe could have nothing to do with their mess that is their dnssec deployment.. But there was mention of cloudflare trying to mitigate their problems on cloudflares end, etc.

If for whatever reason your having issues resolving - its easy enough to set unbound to not do dnssec for that domain, or another option is a domain override for the domain to something like cloudflare or googledns, etc.

Its quite possible that their have been issues with ddos attacks or other congestion problems accessing their site.. I am quite sure since covid, there are enough crazies in the world that would like to attack resources in any manner possible, etc.

As with all problems when trying to access it - you need to troubleshoot it with the specifics. Disable all of dnssec is not as solution - could it be a troubleshooting aid, ok.. I would look more into the specifics if I thought dnssec was a problem vs just system wide disable.. But for some with less understanding, that is a valid option. I personally would not suggest they leave it like that for the sake of 1 site.. And just set that domain to not do dnssec, until such time that have fully corrected what is clearly an not optimal configuration.. From what I see they are signing stuff with alg 7 which the spec clearly says not to do. And they are also doing sha-1 also no longer valid. And they clearly seem to be missing some NS entries that are delegation but not in the authority.. I would say their whole setup is just borked ;)

Gertjan

@johnpoz said in Insanely weird issue with DNS resolution to www.cdc.gov:

@drphil said in Insanely weird issue with DNS resolution to www.cdc.gov:

irs.gov

I don't show any issues with irs.gov

I knew it.
The "last update" is stored on a 'per user' history list.

I saw your last 'udate scan' , and I checked again without updating :
This is what I saw :

So, my last update is not your last update : they are stored with our IP's ( ?) so we all have our own individual history of events.

These are actually really usefull, and can't get disrupted by other who scan the same domain :

Updating it shows the same thing for me : warnings and insecure :

Btw : dnsviz.net must have some huge database behind the screens. Sometimes they do announce that they have database == user history, issues.

.....is just borked ;)

Same thing for me.
If I need to access these sites, and DNSSEC stops me from doing so, and I know that these sites really can't be forged **, I just put them on the "DNSSEC exclusion list", and, because they are listed now, check ones in a while if I can remove them from my list.

To @all ; when I say "list" I mean : I've read the manual](https://www.nlnetlabs.nl/documentation/unbound/howto-turnoff-dnssec/).
So, example ::

Sites like https://dnsviz.net should be part of the box with basic tools that any (DNS) (pfSense) admin uses.

** well, just hope ...

johnpoz

@gertjan Not sure about all that.. I just looked and I am seeing warnings with www.irs.gov, but not with just irs.gov

stompro

Hello, we just ran into this today.

Unbound as a forwarder using OpenDNS (Cisco Umbrella DNS Essentials subscription).

I don't see any red errors on the https://dnsviz.net/d/www.cdc.gov/dnssec/ site.

Thanks for all the resources here for figuring out how to bypass the errors. The 'domain-insecure: "cdc.gov"' bypass seems to work for us also.

I'll send a note to the CDC IT contact that popped up in this thread.

Josh

johnpoz

@stompro said in Insanely weird issue with DNS resolution to www.cdc.gov:

www.cdc.gov

I still show that domain a mess from dnssec point of view..

Unbound as a forwarder using OpenDNS

If your forwarding with unbound, you should not have dnssec checked. Where you forward to does dnssec or it doesn't.. That checkmark telling unbound to do dnssec isn't going to do anything other than problems...

stompro

I wasn't sure if warnings were a problem or not. Some of the other screenshots were filled with red errors... which is what I though the real problem was.

@johnpoz said in Insanely weird issue with DNS resolution to www.cdc.gov:

If your forwarding with unbound, you should not have dnssec checked. Where you forward to does dnssec or it doesn't.. That checkmark telling unbound to do dnssec isn't going to do anything other than problems...

Thank you for saying this again so it would get through to me.

So if the servers that I'm using to forward my request to do not support dnssec... then unbound doesn't do any extra checks on the returned information if dnssec is checked? There is no possible benefit there?

And if the upstream dns server does do dnssec, then it has already performed the extra checks, so the unbound setting is redundant?

Am I understanding that correctly?
Josh

johnpoz

@stompro asking who your forwarding for dnssec info gets you nothing.. They could send you whatever they have cached, etc.

For dnssec to be valid you need to directly talk to the authoritative name servers and validate the info, etc.

Its just going to be extra queries and bandwidth for no actual real benefit.. And could cause problems..

There is zero reason to have your dns software ask for dnssec info if its not actually resolving..

stompro

@johnpoz said in Insanely weird issue with DNS resolution to www.cdc.gov:

For dnssec to be valid you need to directly talk to the authoritative name servers and validate the info, etc.
Its just going to be extra queries and bandwidth for no actual real benefit.. And could cause problems..
There is zero reason to have your dns software ask for dnssec info if its not actually resolving..

Thank you for the extra explanation. I'll fix all my installs to uncheck the "DNSSEC" unbound config option.

Gertjan

@stompro

Keep in mind that you are forwarding to ... a resolver.
That resolver probably does DNSSEC checks, and the cdc announced it supports DNSSEC.
If there was more then a warning, this an error, and the errors concerns a DNS record you were askinf for, protection kicks in : no answer or NXDOMAIN.
You think : my DNS is bad.
The reality is : the zone cdc is bad.
You have just one choice : use a resolver without DNSSEC. So now cdc records as any other DNS can be spoofed etc. I hope this "cdc" site isn't important for you.

can you imagine what happens when facebook start to implement DNSSEC, and they f*ck up on their side ?

johnpoz

@gertjan said in Insanely weird issue with DNS resolution to www.cdc.gov:

Keep in mind that you are forwarding to ... a resolver.

Might not actually be true - could be another forwarder ;) But yeah at some point in the chain there has to be a resolver... So your point is valid.

You know for sure its not actually say 8.8.8.8 doing the actual resolving.. This is why for example when you setup 8.8.8.8 as your dns, and you do one of those dns leak tests, it shows other IPs as your dns that are not 8.8.8.8, same when you point to 1.1.1.1

I would be pretty sure in saying that the IP you point to when using any of these services, isn't actually doing the "resolving" itself. But just forwarding to some other box in their network. DNS wouldn't work at all if at some point a resolver doesn't get asked.

stompro

@gertjan Thank you for the extra info.

I don't quite understand how the unbound DNSSEC option and the work around of setting 'server:domain-insecure: "cdc.gov"' relate to each other, if at all. But I haven't looked up the domain-insecure option yet to see what it actually does. You don't need to google it for me. :-)

I'll try and do more testing when I have a chance.

Cisco Umbrella DNS Servers
- With DNSSEC Enabled - cdc.gov doesn't work for clients.
- With DNSSEC enabled in unbound and 'server:domain-insecure: "cdc.gov"' set, cdc.gov resolves fine for clients.

I can test with cloudflare and google dns, with dnssec on and off.

The pfsense dns lookup gui seems to always work... but I don't know if that is DNSSEC aware.

Maybe the end clients are doing some sort of dnssec validation... but I think @johnpoz explained that that isn't how it works. The resolvers do the validation.

I kind of hate not knowing why something isn't working... but who knows if this whole DNSSEC thing is really going to stick around,,, seems like a fad, like fidget spinners. :-)

johnpoz

@stompro said in Insanely weird issue with DNS resolution to www.cdc.gov:

seems like a fad, like fidget spinners

Not sure if I would say that - but the overall adoption is disappointing to be sure..

Here is the thing that site is all kinds of messed up when it comes to dnssec... I don't have any problem resolving it, using dnssec - but with some of the errors I see, it could for sure be hit or miss.

If your forwarding, and also have dnssec enabled that can cause issues. So are you saying when you uncheck dnssec in unbound, and forward to cisco it fails? Is that something you have to enable do disable in your subscription.. Cisco Umbrella is a subscription service is in not?

 $ dig @192.168.9.253 www.cdc.gov

; <<>> DiG 9.16.27 <<>> @192.168.9.253 www.cdc.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15485
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.cdc.gov.                   IN      A

;; ANSWER SECTION:
www.cdc.gov.            3600    IN      CNAME   www.akam.cdc.gov.
www.akam.cdc.gov.       3600    IN      A       104.98.82.250

;; Query time: 185 msec
;; SERVER: 192.168.9.253#53(192.168.9.253)
;; WHEN: Fri Apr 22 09:45:15 Central Daylight Time 2022
;; MSG SIZE  rcvd: 79