Insanely weird issue with DNS resolution to www.cdc.gov

mboylan

@bmeeks What doesn't make a lot of sense in my case though is that my clients are using the pfSense box as their DNS server. pfSense is forwarding off the query to CloudFlare, getting a response, and then somehow that response is not making it back to the clients. This seems different from your case where once you told your windows servers to forward to unbound, it started working. I'm already doing that, and I get SERVFAIL. I'm happy to escalate to CloudFlare, but seeing as I can query the host from the pfSense box itself, as well as directly against CloudFlare using dig from my clients (but NOT when forwarding through unbound), I'm hard pressed to believe it's a CloudFlare issue. :-/

Edit: I can post the packet captures later today.

bmeeks

@mboylan said in Insanely weird issue with DNS resolution to www.cdc.gov:

@bmeeks What doesn't make a lot of sense in my case though is that my clients are using the pfSense box as their DNS server. pfSense is forwarding off the query to CloudFlare, getting a response, and then somehow that response is not making it back to the clients. This seems different from your case where once you told your windows servers to forward to unbound, it started working. I'm already doing that, and I get SERVFAIL. I'm happy to escalate to CloudFlare, but seeing as I can query the host from the pfSense box itself, as well as directly against CloudFlare using dig from my clients (but NOT when forwarding through unbound), I'm hard pressed to believe it's a CloudFlare issue. :-/

Edit: I can post the packet captures later today.

I agree your issue does not make sense. Are you 100% positive those clients are actually using unbound on pfSense? As I posted, in my case letting the AD DNS server forward to unbound on pfSense solved the issue. And I have unbound on pfSense resolving, not forwarding. I think in your case you have it forwarding to Cloudfare if I recall correctly. But then you said on pfSense itself unbound can resolve "www.cdc.gov". I assume that is with the Cloudfare forwarding in place ??

tman222

Saw this thread last night and for kicks tried to go www.cdc.gov - page would not load. Tried again this morning with a dig www.cdc.gov and came back with SERVFAIL. This is using a Pi-hole / Unbound setup (i.e. clients talk to Pi-hole and Pi-hole forwards the DNS query to pfSense/Unbound if not cached, and Unbound then resolves if not already cached). Tried again this afternoon (a few hours ago) and now all is working fine (i.e. DNS resolves properly and page loads fine). I made no changes on my end in the meantime.

I think @johnpoz might be on to something - perhaps the related name servers aren't or weren't properly configured and that causes issues. I do have DNSSEC enabled as well on Unbound - could that have been what was failing?

johnpoz

Just look at
https://dnsviz.net/d/www.cdc.gov/dnssec/

They have quite a few problems going on.. Its not cloudflare's job to fix it.. Its the domain owners job to make sure their dns works correctly and is valid.

I would contact the cdc webmaster and show him that above dnsviz link.. Tell him to fix his shit..

All kinds of stuff wrong..

net to edgekey.net: The following NS name(s) were found in the authoritative NS RRset, but not in the delegation NS RRset (i.e., in the net zone):
a11-65.akam.net,
ns1-2.akam.net,
a9-65.akam.net,
a3-65.akam.net
net to edgekey.net: The following NS name(s) were found in the delegation NS RRset (i.e., in the net zone), but not in the authoritative NS RRset: 
ns1-66.akam.net, 
ns4-66.akam.net, 
ns5-66.akam.net, 
ns7-65.akam.net
www.akam.cdc.gov/CNAME: The server returned CNAME for www.akam.cdc.gov, but records of other types exist at that name.

That it resolves sometimes at all is just luck to be honest ;)

They have issues way up the chain..

    gov to cdc.gov: The following NS name(s) were found in the authoritative NS RRset, but not in the delegation NS RRset (i.e., in the gov zone): 
icdc-us-ns1.cdc.gov, 
icdc-us-ns3.cdc.gov, 
icdc-us-ns2.cdc.gov
    gov to cdc.gov: The following NS name(s) were found in the delegation NS RRset (i.e., in the gov zone), but not in the authoritative NS RRset: 
auth00.ns.uu.net, 
auth100.ns.uu.net

So again its all going to depend on which NSs your talking too, and what info they have or don't have

Sometimes it will work, sometimes it won't.. the cdc.gov is who should get this fixed..

If a domain has issues with their dnssec - and you forward to somewhere that does dnssec like cloudflare. Your setting of dnssec isn't on or off isn't going to do anything. It should be OFF if you forward.. Where you forward either does dnssec or it doesn't.. There is no point for asking for dnssec when you forward. If you want dnssec when you forward, then pick a place to forward to that does dnssec. I have been over this countless times ;)

edit: Even asking clouldflare you get different responses.. Depending I assume which NS you hit of theirs via anycast..

;www.cdc.gov.                   IN      A

;; ANSWER SECTION:
www.cdc.gov.            78      IN      CNAME   www.akam.cdc.gov.
www.akam.cdc.gov.       3378    IN      CNAME   www.cdc.gov.edgekey.net.
www.cdc.gov.edgekey.net. 20544  IN      CNAME   e9313.dscb.akamaiedge.net.
e9313.dscb.akamaiedge.net. 20   IN      A       23.222.138.25

;; Query time: 15 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Dec 29 06:17:04 Central Standard Time 2020
;; MSG SIZE  rcvd: 152


sec later

;www.cdc.gov.                   IN      A

;; ANSWER SECTION:
www.cdc.gov.            76      IN      CNAME   www.akam.cdc.gov.
www.akam.cdc.gov.       19      IN      A       23.222.138.25

;; Query time: 132 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Dec 29 06:17:05 Central Standard Time 2020
;; MSG SIZE  rcvd: 79

The cdc really should fix up their shit ;)

timtrace

@johnpoz said in Insanely weird issue with DNS resolution to www.cdc.gov:

The cdc really should fix up their shit ;)

I’m experiencing this problem, also. When I disable DNSSEC the problem goes away and CDC.GOV loads.

Can anything else be done as a workaround, which wouldn’t have as broad an scope as toggling DNSSEC?

Thank you —

johnpoz

@timtrace said in Insanely weird issue with DNS resolution to www.cdc.gov:

Can anything else be done as a workaround

One way would be to do a domain override to say 9.9.9.10, which is quad9 that doesn't do dnssec.. So that shouldn't fail.. You do a domain override for cdc.gov to any NS that doesn't do dnssec..

Another option should be to set unbound not to do dnssec for that domain.. In the options box

server:
domain-insecure: "cdc.gov"

You would think they would have fixed their shit by now to be honest.. You might actually have to do it for domains the cnames point to if you don't do the domain override forwarding to a non dnssec ns..

But looks like they just have the 1 cname currently www.akam.cdc.gov, so cdc.gov as the unsecure domain should work.

Worse case is you add the other domains as unsecure as well

www.akam.cdc.gov.       3378    IN      CNAME   www.cdc.gov.edgekey.net.
www.cdc.gov.edgekey.net. 20544  IN      CNAME   e9313.dscb.akamaiedge.net.

Who ever is in charge of their dns should really be fired..

timtrace

@johnpoz said in Insanely weird issue with DNS resolution to www.cdc.gov:

server:
domain-insecure: "cdc.gov"

Thanks, man! That worked perfectly.

mboylan

@johnpoz Thanks! This option fixed the issue immediately.

gsmithe

@johnpoz said in Insanely weird issue with DNS resolution to www.cdc.gov:

Another option should be to set unbound not to do dnssec for that domain.. In the options box
server:
domain-insecure: "cdc.gov"

Thank you! Worked for me, too.

johnpoz

I have been checking on this now and then... They are still freaking hosed.. They clearly made some sort of changes - but its still a freaking mess..

If you don't understand how to dnssec - just don't do it.. A borked dnssec is worse than not having it that is for damn sure..

edit:
Just some of the errors still being seen

RRSIG cdc.gov/A alg 7, id 42473: DNSSEC specification recommends not signing with DNSSEC algorithm 7 (RSASHA1NSEC3SHA1).
RRSIG cdc.gov/DNSKEY alg 7, id 42473: DNSSEC specification recommends not signing with DNSSEC algorithm 7 (RSASHA1NSEC3SHA1).
RRSIG cdc.gov/DNSKEY alg 7, id 65139: DNSSEC specification recommends not signing with DNSSEC algorithm 7 (RSASHA1NSEC3SHA1).

chewie198

I had been experiencing this issue myself for the last several months and it wasn't until my wife complained about it last week that I decided to investigate and came across this thread. I actually went ahead and submitted a message via their contact form and linked to this exact forum thread describing the issue. It's hard to say whether that was the impetus for their finally updating their DNS records, but it does at least seem to be working correctly now without using the 'domain-insecure: "cdc.gov"' workaround. Big thanks to everybody who actually took the time to discover the underlying issue. Now hopefully they don't break it again.

Imtech

This DNS issue for www.cdc.gov site should have been resolved. If anyone is still seeing similar issues, please let CDC know at imtech@cdc.gov. Thank you!

espalmer

This was driving me nuts also. Needed to access the CDC for Covid info and check in after vaccination. Yes "lucky" enough to get one, unless you count having a compromised immune system as not lucky :)

At first I blamed my pfsense and Ngblocker setup as I make ALOT of mistakes as I am just learning. But I found NO errors....I then found this topic and adding the server: domain-insecure: "cdc.gov" solved the issue.

Thanks I was getting ready to pull my DNS setup apart to figure out where I screwed up..........

johnpoz

Not sure I would call it "fixed" while its better - they still have stuff wrong if you ask me.. They are using algos they shouldn't be.

If your not going to follow best practices for dnssec - why even try and use it to be honest..

From RFC 8624

   +--------+--------------------+-----------------+-------------------+
   | Number | Mnemonics          | DNSSEC Signing  | DNSSEC Validation |
   +--------+--------------------+-----------------+-------------------+
   | 1      | RSAMD5             | MUST NOT        | MUST NOT          |
   | 3      | DSA                | MUST NOT        | MUST NOT          |
   | 5      | RSASHA1            | NOT RECOMMENDED | MUST              |
   | 6      | DSA-NSEC3-SHA1     | MUST NOT        | MUST NOT          |
   | 7      | RSASHA1-NSEC3-SHA1 | NOT RECOMMENDED | MUST              |
   | 8      | RSASHA256          | MUST            | MUST              |
   | 10     | RSASHA512          | NOT RECOMMENDED | MUST              |
   | 12     | ECC-GOST           | MUST NOT        | MAY               |
   | 13     | ECDSAP256SHA256    | MUST            | MUST              |
   | 14     | ECDSAP384SHA384    | MAY             | RECOMMENDED       |
   | 15     | ED25519            | RECOMMENDED     | RECOMMENDED       |
   | 16     | ED448              | MAY             | RECOMMENDED       |
   +--------+--------------------+-----------------+-------------------+

They are using 7.. plus their NS listed have issues.

gov to cdc.gov: The following NS name(s) were found in the authoritative NS RRset, but not in the delegation NS RRset (i.e., in the gov zone):
icdc-us-ns1.cdc.gov,
icdc-us-ns3.cdc.gov,
icdc-us-ns2.cdc.gov

gov to cdc.gov: The following NS name(s) were found in the delegation NS RRset (i.e., in the gov zone), but not in the authoritative NS RRset: 
auth00.ns.uu.net, 
auth100.ns.uu.net

But hey if its actually working, better than it was ;)

Gertjan

@johnpoz said in Insanely weird issue with DNS resolution to www.cdc.gov:

I would contact the cdc webmaster and show him that above dnsviz link.. Tell him to fix his shit..

Or : he was fired after all, and the new one wiped everything.

https://dnsviz.net/d/www.cdc.org/dnssec/ is clean for me. No more issues.

That is : www.cdc.org becames like the other 85 % (?) of the net : no DNSSEC.
( so, all is well, www.cdc.org is DNS spoofable again ).

https://dnsviz.net/d/www.cdc.org/dnssec/

@espalmer said in Insanely weird issue with DNS resolution to www.cdc.gov:

domain-insecure: "cdc.gov" solved the issue.

shouldn't be needed any more as they cleaned up (very recently ?).

Btw : Compare with this site : https://dnsviz.net/d/forum.netgate.com/dnssec/
No DNSSEC neither, and no need to inform unbound about it.

Maybe Netgate should activate DNSSEC, just to show that DNSSECworks without any settings on the 'viewers' side (neither our pfSense). After all, its about 'network' security and so.

msf2000

@johnpoz
I know this is very late, but CDC still hasn't fixed everything.

But thank you for the server directive tip. That resolved the issue for me.

server:domain-insecure: "cdc.gov"

johnpoz

Yeah I still show it as a big mess..

But at least they are only warnings, and not full on failures, etc.

gregarios

I'm also having this exact issue now on a Turris Omnia 2020 router with DNSSEC enabled or disabled and forwarding to OpenDNS turned on. If I bypass the router and use direct OpenDNS numbers on my computer I can get to cdc.gov but when I use the router's DNS which is supposed to send my requests up stream to OpenDNS it won't let it through. I don't even get the Block page from OpenDNS it just acts like it won't resolve in the router at all. Other sites seem to work just fine.

I've manually added cdc.gov numbers to the router's /etc/hosts file and it works for now, but this is too hacky and not sustainable, really. Also would like to know how many other sites on the internet aren't working unbeknownst to me.

DrPhil

I stumbled on this thread because I was having the same issue right now. I use openDNS on DNS resolver (unbound) on Pfsense (Netgate device).

I used JohnPoz domain override suggestion as a band aid, and it works. For other non-sophisticated home users like myself, here's an example of how to do it on the WebGUI.

Services / DNS Resolver / General Settings

"8.8.8.8" is Google's DNS and for some reason tolerates the incorrect DNS setup at cdc.gov (as described in this thread by people who understand such stuff).

Gertjan

@drphil :

https://cachecheck.opendns.com/

cdc.gov
and
www.cdc.gov

did resolve for me.

Maybe your own 'opendns' settings are to strict ?