Single website won't resolve for clients - resolves fine for pfSense itself

lparker

@johnpoz You're right, I just saw the UDP setup failure with those IPv6 addresses but re-enabling it didn't change anything. Any other thoughts on what I could try?

johnpoz

@lparker see my edit, can you directly query one of their NS and get an answer?

Their other ns is 97.74.104.45

lparker

@johnpoz I get the following when running dig queries from Ubuntu WSL virtual client behind my pfSense router:

XXXXX@XXXXX-DESKTOP:~$ dig @173.201.72.45 advantechwifi.com

; <<>> DiG 9.16.1-Ubuntu <<>> @173.201.72.45 advantechwifi.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31211
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;advantechwifi.com.             IN      A

;; ANSWER SECTION:
advantechwifi.com.      0       IN      A       199.38.182.75
advantechwifi.com.      0       IN      A       199.38.182.52

;; Query time: 40 msec
;; SERVER: 173.201.72.45#53(173.201.72.45)
;; WHEN: Mon Jun 12 10:08:18 EDT 2023
;; MSG SIZE  rcvd: 78

johnpoz

@lparker said in Single website won't resolve for clients - resolves fine for pfSense itself:

dig @173.201.72.45 advantechwifi.com

that doesn't seem right.. Why are you getting back a 0 ttl?

When you directly query an auth ns for a domain, you should always get the full ttl they have set on the record.

your not doing any redirection of dns are you?

edit: you should be seeing a aa flag in your response as well - since your directly talking to auth ns, that would seem like your dns is being intercepted.

see mine, just did another one

ser@i9-win:~$ dig @173.201.72.45 advantechwifi.com

; <<>> DiG 9.18.15-1+ubuntu22.04.1+isc+1-Ubuntu <<>> @173.201.72.45 advantechwifi.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15175
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;advantechwifi.com.             IN      A

;; ANSWER SECTION:
advantechwifi.com.      10800   IN      A       192.124.249.7

;; AUTHORITY SECTION:
advantechwifi.com.      3600    IN      NS      ns70.domaincontrol.com.
advantechwifi.com.      3600    IN      NS      ns69.domaincontrol.com.

;; Query time: 20 msec
;; SERVER: 173.201.72.45#53(173.201.72.45) (UDP)
;; WHEN: Mon Jun 12 09:26:41 CDT 2023
;; MSG SIZE  rcvd: 114

user@i9-win:~$

Notice the aa flag.. See when ask say google for that, you don't see the aa flag

user@i9-win:~$ dig @8.8.8.8 advantechwifi.com

; <<>> DiG 9.18.15-1+ubuntu22.04.1+isc+1-Ubuntu <<>> @8.8.8.8 advantechwifi.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12805
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;advantechwifi.com.             IN      A

;; ANSWER SECTION:
advantechwifi.com.      10800   IN      A       192.124.249.7

;; Query time: 79 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Mon Jun 12 09:30:05 CDT 2023
;; MSG SIZE  rcvd: 62

user@i9-win:~$

And if you continue to check it - you notice that the ttl starts to drop depending on which google server answered you - you never know which one will answer you since anycast, etc. That mine shows the full ttl 10800, would tell me that it had to resolve it vs cache - but no aa in the flags, so tells me that came from a non authoritative ns, ie I asked google..

;; ANSWER SECTION:
advantechwifi.com.      10518   IN      A       192.124.249.7

no aa flag, and that 0 ttl screams dns redirection to me.

lparker

@johnpoz Originally had DNS redirection of anything using UDP/TCP 53 back to 127.0.0.1 but that's been removed. DNS forwarding is disabled, only have on DNS Resolver. DHCP isn't declaring any special DNS servers and I can see on my clients that DHCP is handing out the default LAN address of the pfsense as the DNS server for clients.

In System > General > DNS Resolution Behavior - that is set to Use local DNS, fall back to remote DNS Servers (Default). I'm pretty new with pfSense (as you probably guessed) but it sounds like the pfSense can resort to using my WAN declared DNS servers if 127.0.0.1 fails for itself but when a client requests from the pfSense to lookup this domain, it's not requesting from these public DNS servers. I'm using 1.1.1.1 and 8.8.4.4

Edit: When I query other domains, I do receive TTL values

johnpoz

@lparker yeah if you don't have forwarders setup in unbound, then no clients can not use those listed in pfsense dns settings, only pfsense itself could do that..

Clearly you are not actually talking to that NS.. you should see aa in the flags if you were, and you should see the full ttl on every query 10800 is what they have set.

I would say you have serve zero set in unbound, but that would only come into play if you were redirecting dns to unbound since in your dig your calling out to directly talk to that ns IP..

You sure you have redirection turned off in pfsense? So what should happen even if redirected, you wouldn't see the aa but say your first query could be 0 ttl, but if you queried again that ttl should go up to the full when unbound actually resolved it. and then start counting down.. That is staying 0 tells me unbound isn't actually able to talk to that auth ns for some reason.

If you do that query on pfsense what happens - that shouldn't be redirected..

[23.05-RELEASE][admin@sg4860.local.lan]/root: dig @173.201.72.45 advantechwifi.com

; <<>> DiG 9.18.13 <<>> @173.201.72.45 advantechwifi.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31830
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;advantechwifi.com.             IN      A

;; ANSWER SECTION:
advantechwifi.com.      10800   IN      A       192.124.249.7

;; AUTHORITY SECTION:
advantechwifi.com.      3600    IN      NS      ns70.domaincontrol.com.
advantechwifi.com.      3600    IN      NS      ns69.domaincontrol.com.

;; Query time: 10 msec
;; SERVER: 173.201.72.45#53(173.201.72.45) (UDP)
;; WHEN: Mon Jun 12 09:46:16 CDT 2023
;; MSG SIZE  rcvd: 114

[23.05-RELEASE][admin@sg4860.local.lan]/root:

Your serve zero response is also different IP than I am seeing talking to the auth ns - so that could explain why the site isn't actually working, because your IP getting back are not valid any more.

lparker

@johnpoz Here's from the pfSense below. I would say maybe Comcast is interfering, but I still have my Wi-Fi running over a different firewall using a public IP in the same /29 and clients behind that can resolve it just fine.

[23.05-RELEASE][admin@pfSense.ad.techzenit.com]/root: dig @173.201.72.45 advantechwifi.com

; <<>> DiG 9.18.13 <<>> @173.201.72.45 advantechwifi.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42458
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;advantechwifi.com.             IN      A

;; ANSWER SECTION:
advantechwifi.com.      0       IN      A       199.38.182.52
advantechwifi.com.      0       IN      A       199.38.182.75

;; Query time: 35 msec
;; SERVER: 173.201.72.45#53(173.201.72.45) (UDP)
;; WHEN: Mon Jun 12 10:49:13 EDT 2023
;; MSG SIZE  rcvd: 78

johnpoz

@lparker that points to redirection being done upstream of pfsense.. What if you query say a different authoritative ns? Say the one for netgate.com

;; ADDITIONAL SECTION:
ns1.netgate.com.        1241    IN      A       208.123.73.80
ns2.netgate.com.        1241    IN      A       208.123.73.90
ns3.netgate.com.        1241    IN      A       34.197.184.5

23.05-RELEASE][admin@sg4860.local.lan]/root: dig @208.123.73.80 netgate.com

; <<>> DiG 9.18.13 <<>> @208.123.73.80 netgate.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19153
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 90de2fc3f45ad07a01000000648731f2c23be620901f7f07 (good)
;; QUESTION SECTION:
;netgate.com.                   IN      A

;; ANSWER SECTION:
netgate.com.            60      IN      A       199.60.103.104
netgate.com.            60      IN      A       199.60.103.4

;; Query time: 33 msec
;; SERVER: 208.123.73.80#53(208.123.73.80) (UDP)
;; WHEN: Mon Jun 12 09:55:46 CDT 2023
;; MSG SIZE  rcvd: 100

[23.05-RELEASE][admin@sg4860.local.lan]/root:

Notice the aa flag, and the ttl - 60 is insanely low if you ask me, but that is what they have set..

edit: another test you can do about redirection.. if you query an IP that you know is not serving dns - you know for a fact your being redirect ;)

[23.05-RELEASE][admin@sg4860.local.lan]/root: dig @1.2.3.4 www.google.com
;; communications error to 1.2.3.4#53: timed out

But if I check it from box behind pfsense that I am doing dns redirection on ;)

$ dig @1.2.3.4 www.google.com

; <<>> DiG 9.16.41 <<>> @1.2.3.4 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37495
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         2616    IN      A       142.250.190.132

;; Query time: 0 msec
;; SERVER: 1.2.3.4#53(1.2.3.4)
;; WHEN: Mon Jun 12 10:00:51 Central Daylight Time 2023
;; MSG SIZE  rcvd: 59

if I remove the redirection, then my client times out as it should

$ dig @1.2.3.4 www.google.com                                  
                                                               
; <<>> DiG 9.16.41 <<>> @1.2.3.4 www.google.com                
; (1 server found)                                             
;; global options: +cmd                                        
;; connection timed out; no servers could be reached

1.2.3.4 isn't doing dns - so clearly yeah I shouldn't get an answer if I ask it directly ;) That is a smoking gun that your dns is being messed with..

lparker

@johnpoz I've actually just started getting more widespread DNS issues so for now I've moved clients to using public DNS and I'll probably do a config restore later on and check from there but it will be too disruptive for me to work on it now. Thanks for all your help!

johnpoz

@lparker on pfsense do that query check for me, do a query to 1.2.3.4 - do you get a response? If so then your dns is being messed with upstream that is for damn sure!

But a query to an auth NS not seeing aa in flags also screams dns being messed with.

lparker

@johnpoz Hah, yeah, I'm still getting responses from 1.2.3.4 even. Is there a way to flush unbound cache or what settings can I look for where this redirection would be enabled? The only rule I previously had was to redirect traffic targeting port 53 to use 127.0.0.1 via NAT rule but this has been removed for awhile now. DNS Resolver service restart didn't make a difference and neither did rebooting the entire firewall.

johnpoz

@lparker said in Single website won't resolve for clients - resolves fine for pfSense itself:

getting responses from 1.2.3.4 even

That is not an unound cache thing.. Your doing a directed query.. Your dns is being intercept at some point between where your sending traffic and were your trying to go..

I don't see how you could be doing a redirect on pfsense for that? Do you have some outbound rule in floating to do redirection?? That would be the only thing that you could of setup that would possible to do that, if that was the case then yeah dns is just going to utterly fail.. I don't even see how your directed queries to something like 8.8.8.8 from a client could work if you were doing such a thing.

Would seem to me your isp, or do you have some other device like another router in front of pfsense is redirecting your dns.. Creating a redirection of dns on your lan wouldn't have anything to do with pfsense doing a directed query.. The only way it would even be possible to redirect pfsense doing dns would be on some outbound rule in floating.

your not routing traffic through a vpn are you?

lparker

@johnpoz Yeah its extremely weird to say the least. Clients are working fine when using alternate DNS servers, but it seems all DNS requests sent to resolver are failing. I'm going to try doing a restore after hours today and see where that gets me.

lparker

Just as a further note - if I disable Resolver and instead turn on DNS Forwarder - clients can now use send DNS requests to the pfSense and it forwards it out and clients get the response back as expected - even when reaching out to the previously mentioned advantechwifi.com

johnpoz

@lparker tell you right now this is NOT a pfsense issue...

How would redirect its own traffic to itself and resolve something when you asked 1.2.3.4 for something?

You have restarted unbound I am sure right? how would you be getting back that zero ttl? since once you restart unbound there is no way for that to be in the cache, etc.

lparker

@johnpoz Well since clients worked with statically set public DNS servers and DNS Forwarding works in place of DNS Resolver, are you still convinced its a DNS redirect somewhere? I almost think maybe unbound got corrupted? It was already reloaded, yes.

johnpoz

@lparker yeah - they are not redirecting 8.8.8.8 maybe... But how would pfsense redirect traffic to itself, especially when you stated you don't have any redirection setup.. And if it was - why would it not redirect 8.8.8.8?

You can't do an outbound nat on your wan to yourself..

Sorry but asking 1.2.3.4 for dns and getting an answer is clearly redirection of dns.. period.. Here is an idea - why don't you stop unbound completely.. Now do your query to 1.2.3.4 as your test.

Look at your +trace you did from before

advantechwifi.com.      0       IN      A       199.38.182.75
advantechwifi.com.      0       IN      A       199.38.182.52
;; Received 78 bytes from 199.9.14.201#53(b.root-servers.net) in 36 ms

There is no possible way the root server answered with that.. The root servers don't have such info.. So your saying pfsense somehow redirect outbound traffic to itself, and answered with that info? After a restart of unbound how would it have that info in its cache?

Do you even have serve zero setup - that is not a default setting..

To prove it to yourself - turn off the unbound service.. Make sure pfsense isn't listening on 53 with netstat or something, now do your directec query tests.. query 1.2.3. 4 etc..