Strange issue - not sure how to fix

pfguy2018

@johnpoz said in Strange issue - not sure how to fix:

dig h.root-servers.net

I just rebooted, so I am able to access feedly.com again. But who knows for how long?

This is the output:
; <<>> DiG 9.12.2-P1 <<>> h.root-servers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8969
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;h.root-servers.net. IN A

;; ANSWER SECTION:
h.root-servers.net. 86400 IN A 198.97.190.53

;; Query time: 3199 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Feb 22 17:57:27 EST 2020
;; MSG SIZE rcvd: 63

; <<>> DiG 9.12.2-P1 <<>> @198.97.190.53 com ns
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49732
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;com. IN NS

;; AUTHORITY SECTION:
com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.

;; ADDITIONAL SECTION:
a.gtld-servers.net. 172800 IN A 192.5.6.30
b.gtld-servers.net. 172800 IN A 192.33.14.30
c.gtld-servers.net. 172800 IN A 192.26.92.30
d.gtld-servers.net. 172800 IN A 192.31.80.30
e.gtld-servers.net. 172800 IN A 192.12.94.30
f.gtld-servers.net. 172800 IN A 192.35.51.30
g.gtld-servers.net. 172800 IN A 192.42.93.30
h.gtld-servers.net. 172800 IN A 192.54.112.30
i.gtld-servers.net. 172800 IN A 192.43.172.30
j.gtld-servers.net. 172800 IN A 192.48.79.30
k.gtld-servers.net. 172800 IN A 192.52.178.30
l.gtld-servers.net. 172800 IN A 192.41.162.30
m.gtld-servers.net. 172800 IN A 192.55.83.30
a.gtld-servers.net. 172800 IN AAAA 2001:503:a83e::2:30
b.gtld-servers.net. 172800 IN AAAA 2001:503:231d::2:30
c.gtld-servers.net. 172800 IN AAAA 2001:503:83eb::30
d.gtld-servers.net. 172800 IN AAAA 2001:500:856e::30
e.gtld-servers.net. 172800 IN AAAA 2001:502:1ca1::30
f.gtld-servers.net. 172800 IN AAAA 2001:503:d414::30
g.gtld-servers.net. 172800 IN AAAA 2001:503:eea3::30
h.gtld-servers.net. 172800 IN AAAA 2001:502:8cc::30
i.gtld-servers.net. 172800 IN AAAA 2001:503:39c1::30
j.gtld-servers.net. 172800 IN AAAA 2001:502:7094::30
k.gtld-servers.net. 172800 IN AAAA 2001:503:d2d::30
l.gtld-servers.net. 172800 IN AAAA 2001:500:d937::30
m.gtld-servers.net. 172800 IN AAAA 2001:501:b1f9::30

;; Query time: 27 msec
;; SERVER: 198.97.190.53#53(198.97.190.53)
;; WHEN: Sat Feb 22 17:57:53 EST 2020
;; MSG SIZE rcvd: 828

Does it matter that my DNS is set to 127.0.0.1?

johnpoz

That was after you rebooted.. Or was that before you rebooted... You shouldn't have to reboot - just restart unbound.

johnpoz

if you are resolving - then yes your dns should be set to loopback! That is unbound directly..

pfguy2018

This was after I rebooted. Prior to that, I did in fact restart unbound manually, but it did not fix the problem. Rebooting always solves this issue, for some length of time, but then it always recurs

It occurs to me that the reason I can access sites in the browser that I have never accessed before even when I am experiencing this issue is that I am using Firefox, which internally sets its own DNS to cloudfare DNS. So it is probably bypassing the pfSense DNS and allowing me to access these sites. So it was masking the issue. Strangely though, it was still not allowing me to access feedly.com. So something is still being blocked somewhere at some point.

pfguy2018

Other than rebooting, is there anything else I can try to get Unbound connected to the root servers when this occurs again?

johnpoz

@pfguy2018 said in Strange issue - not sure how to fix:

which internally sets its own DNS to cloudfare DNS

Mine doesn't - I have that shit turned off!! F that going to hand everything to cloudflare..

I would validate the query is actually going out, or not... Do a sniff on your wan - are you seeing dns going out.. Up the logging of unbound to see if you can see errors, etc.

If your saying restarting unbound doesn't fix it but reboot, then points to something with the connection... Sniff to see if your dns goes out and your just not getting an answer.

pfguy2018

I don't see any errors in the unbound log at level 2 or 3 logging. I am not sure how to sniff the WAN traffic - can you point me to some instructions for how to do that?

johnpoz

Go to diag menu, packet capture - pick your wan interface.

pfguy2018

Doing that, how do I check if the root servers are being queried?

johnpoz

Well query something ;) While your sniffing..

pfguy2018

OK, now I just have to wait for the DNS resolution to start failing again before I sniff the traffic. Will report back later.

johnpoz

You sniff now right to get a feel for what sort of traffic you see, etc. When its working...

You might want to grab say wireshark for your pc, so you can load the capture into that - download on the diag page.. This will give you way more insight and details of what is going on.. But really your just wanting to see that hey the query went out.. Did you or did you not get an answer..

What you were showing you were having timeouts..

pfguy2018

I am trying to post the traffic output, but it is being flagged as spam by the software running the forum, and I can't post it.

pfguy2018

08:57:57.060415 ARP, Request who-has 99.228.243.128 tell 99.228.242.1, length 46
08:57:57.071807 IP xxx.xxx.xxx.xxx > 99.228.144.1: ICMP echo request, id 42349, seq 6292, length 8
08:57:57.082227 IP 99.228.144.1 > xxx.xxx.xxx.xxx: ICMP echo reply, id 42349, seq 6292, length 8
08:57:57.084566 IP xxx.xxx.xxx.xxx.23495 > 24.156.130.43.80: tcp 442
08:57:57.092988 IP 24.156.130.43.80 > xxx.xxx.xxx.xxx.23495: tcp 0
08:57:57.097414 ARP, Request who-has 206.188.75.254 tell 206.188.75.225, length 46
08:57:57.100039 IP 24.156.130.43.80 > xxx.xxx.xxx.xxx.23495: tcp 1448
08:57:57.100058 IP 24.156.130.43.80 > xxx.xxx.xxx.xxx.23495: tcp 572
08:57:57.101500 IP xxx.xxx.xxx.xxx.23495 > 24.156.130.43.80: tcp 0
08:57:57.115341 ARP, Request who-has 99.228.195.183 tell 99.228.194.1, length 46
08:57:57.132347 ARP, Request who-has 209.142.43.80 tell 209.142.43.65, length 46
08:57:57.136346 ARP, Request who-has 104.158.199.48 tell 104.158.199.33, length 46
08:57:57.153353 ARP, Request who-has 72.53.212.137 tell 72.53.212.129, length 46
08:57:57.155347 ARP, Request who-has 99.228.150.0 tell 99.228.150.1, length 46
08:57:57.157348 ARP, Request who-has 135.23.223.90 tell 135.23.223.65, length 46
08:57:57.176453 ARP, Request who-has 108.168.98.253 tell 108.168.98.225, length 46
08:57:57.183379 ARP, Request who-has 99.228.197.247 tell 99.228.196.1, length 46
08:57:57.189359 ARP, Request who-has 99.228.237.142 tell 99.228.236.1, length 46
08:57:57.190311 IP xxx.xxx.xxx.xxx.24725 > 151.101.126.99.443: tcp 111
08:57:57.192357 ARP, Request who-has 99.228.237.217 tell 99.228.236.1, length 46
08:57:57.202537 IP 151.101.126.99.443 > xxx.xxx.xxx.xxx.24725: tcp 1448
08:57:57.202631 IP 151.101.126.99.443 > xxx.xxx.xxx.xxx.24725: tcp 1340
08:57:57.202682 IP 151.101.126.99.443 > xxx.xxx.xxx.xxx.24725: tcp 1448
08:57:57.202876 IP 151.101.126.99.443 > xxx.xxx.xxx.xxx.24725: tcp 1340

pfguy2018

I was able to post part of it above. Not sure if it contains any useful info.

johnpoz

None of that is dns... Port 53 is dns..

This is dns traffic

08:51:11.915816 IP 64.53.xxx.xxx.1499 > 204.13.251.3.53: UDP, length 47
08:51:11.916510 IP 64.53.xxx.xxx.24691 > 208.78.71.3.53: UDP, length 47
08:51:11.918974 IP 64.53.xxx.xxx.63171 > 208.80.125.4.53: UDP, length 46
08:51:11.919863 IP 64.53.xxx.xxx.29586 > 208.80.126.4.53: UDP, length 46
08:51:11.943785 IP 208.80.126.4.53 > 64.53.xxx.xxx.29586: UDP, length 92
08:51:11.944094 IP 64.53.xxx.xxx.26752 > 192.229.254.5.53: UDP, length 47
08:51:11.945081 IP 208.80.125.4.53 > 64.53.xxx.xxx.63171: UDP, length 95
08:51:11.945275 IP 64.53.xxx.xxx.34945 > 72.21.80.6.53: UDP, length 47
08:51:11.945456 IP 208.78.71.3.53 > 64.53.xxx.xxx.24691: UDP, length 116
08:51:11.945733 IP 64.53.xxx.xxx.29783 > 208.80.125.4.53: UDP, length 46
08:51:11.959085 IP 204.13.251.3.53 > 64.53.xxx.xxx.1499: UDP, length 116
08:51:11.959591 IP 64.53.xxx.xxx.2756 > 208.80.125.4.53: UDP, length 46
08:51:11.963659 IP 192.229.254.5.53 > 64.53.xxx.xxx.26752: UDP, length 63
08:51:11.963870 IP 64.53.xxx.xxx.11652 > 192.229.254.6.53: UDP, length 47
08:51:11.969938 IP 208.80.125.4.53 > 64.53.xxx.xxx.29783: UDP, length 95
08:51:11.970255 IP 64.53.xxx.xxx.42952 > 192.229.254.6.53: UDP, length 47
08:51:11.971483 IP 72.21.80.6.53 > 64.53.xxx.xxx.34945: UDP, length 63
08:51:11.971721 IP 64.53.xxx.xxx.23430 > 72.21.80.6.53: UDP, length 47
08:51:11.976236 IP 208.80.125.4.53 > 64.53.xxx.xxx.2756: UDP, length 95
08:51:11.976409 IP 64.53.xxx.xxx.57024 > 72.21.80.5.53: UDP, length 47
08:51:11.977384 IP 192.229.254.6.53 > 64.53.xxx.xxx.11652: UDP, length 63

And if open in wireshark I can see info..

pfguy2018

Here is the port 53 UDP traffic when running the dig feedly.com +trace command

09:51:27.591255 IP 72.246.46.129.53 > xxx.xxx.xxx.xxx.22371: UDP, length 63
09:51:27.592186 IP xxx.xxx.xxx.xxx.13656 > 23.61.199.64.53: UDP, length 44
09:51:27.596218 IP 192.5.5.241.53 > xxx.xxx.xxx.xxx.17958: UDP, length 786
09:51:27.596325 IP 199.19.56.1.53 > xxx.xxx.xxx.xxx.51001: UDP, length 1021
09:51:27.596694 IP xxx.xxx.xxx.xxx.28502 > 43.230.48.1.53: UDP, length 34
09:51:27.597226 IP xxx.xxx.xxx.xxx.54324 > 2.16.130.131.53: UDP, length 47
09:51:27.598180 IP 84.53.139.66.53 > xxx.xxx.xxx.xxx.17594: UDP, length 113
09:51:27.598532 IP xxx.xxx.xxx.xxx.50451 > 23.211.132.65.53: UDP, length 54
09:51:27.602183 IP 23.211.133.131.53 > xxx.xxx.xxx.xxx.61509: UDP, length 62
09:51:27.602257 IP 184.26.161.130.53 > xxx.xxx.xxx.xxx.19037: UDP, length 63
09:51:27.604184 IP 217.160.81.195.53 > xxx.xxx.xxx.xxx.27810: UDP, length 62
09:51:27.605758 IP 217.160.83.196.53 > xxx.xxx.xxx.xxx.36489: UDP, length 62
09:51:27.605868 IP 192.41.162.30.53 > xxx.xxx.xxx.xxx.37184: UDP, length 745
09:51:27.606422 IP xxx.xxx.xxx.xxx.28094 > 217.160.82.194.53: UDP, length 46
09:51:27.606709 IP xxx.xxx.xxx.xxx.42365 > 217.160.81.195.53: UDP, length 46
09:51:27.607408 IP 192.42.93.30.53 > xxx.xxx.xxx.xxx.64205: UDP, length 1374
09:51:27.608006 IP xxx.xxx.xxx.xxx.10823 > 199.7.91.13.53: UDP, length 47
09:51:27.608960 IP 96.7.49.129.53 > xxx.xxx.xxx.xxx.29227: UDP, length 109
09:51:27.609059 IP 193.108.88.128.53 > xxx.xxx.xxx.xxx.24107: UDP, length 116
09:51:27.609384 IP xxx.xxx.xxx.xxx.58014 > 2.22.230.130.53: UDP, length 65
09:51:27.610087 IP xxx.xxx.xxx.xxx.29106 > 193.108.88.128.53: UDP, length 58
09:51:27.610604 IP 199.249.112.1.53 > xxx.xxx.xxx.xxx.24105: UDP, length 1021
09:51:27.611073 IP xxx.xxx.xxx.xxx.59245 > 2.16.130.131.53: UDP, length 47
09:51:27.612161 IP 217.160.83.196.53 > xxx.xxx.xxx.xxx.4652: UDP, length 62
09:51:27.612257 IP 23.74.25.128.53 > xxx.xxx.xxx.xxx.35281: UDP, length 63
09:51:27.612334 IP 95.100.168.128.53 > xxx.xxx.xxx.xxx.35288: UDP, length 63
09:51:27.614307 IP 216.239.34.10.53 > xxx.xxx.xxx.xxx.56661: UDP, length 64
09:51:27.615835 IP 217.160.82.194.53 > xxx.xxx.xxx.xxx.30538: UDP, length 62
09:51:27.615934 IP 217.160.83.196.53 > xxx.xxx.xxx.xxx.29119: UDP, length 62
09:51:27.616010 IP 217.160.82.194.53 > xxx.xxx.xxx.xxx.26764: UDP, length 62
09:51:27.616110 IP 217.160.83.196.53 > xxx.xxx.xxx.xxx.35771: UDP, length 62
09:51:27.616185 IP 217.160.83.196.53 > xxx.xxx.xxx.xxx.32909: UDP, length 62
09:51:27.617363 IP 217.160.81.195.53 > xxx.xxx.xxx.xxx.21281: UDP, length 62
09:51:27.617460 IP 217.160.82.194.53 > xxx.xxx.xxx.xxx.13681: UDP, length 62
09:51:27.626193 IP 216.239.34.10.53 > xxx.xxx.xxx.xxx.48475: UDP, length 71
09:51:27.630169 IP 23.211.132.65.53 > xxx.xxx.xxx.xxx.50451: UDP, length 113
09:51:27.630273 IP 81.91.164.5.53 > xxx.xxx.xxx.xxx.47478: UDP, length 691
09:51:27.630882 IP xxx.xxx.xxx.xxx.29570 > 217.160.80.193.53: UDP, length 45
09:51:27.630885 IP xxx.xxx.xxx.xxx.42631 > 95.100.168.130.53: UDP, length 51
09:51:27.631227 IP xxx.xxx.xxx.xxx.24142 > 217.160.81.196.53: UDP, length 45
09:51:27.631510 IP xxx.xxx.xxx.xxx.48389 > 217.160.82.194.53: UDP, length 45
09:51:27.631761 IP xxx.xxx.xxx.xxx.35040 > 217.160.80.193.53: UDP, length 45
09:51:27.634252 IP 199.7.91.13.53 > xxx.xxx.xxx.xxx.10823: UDP, length 825
09:51:27.636334 IP xxx.xxx.xxx.xxx.10416 > 192.112.36.4.53: UDP, length 47
09:51:27.637532 IP 77.67.63.105.53 > xxx.xxx.xxx.xxx.45244: UDP, length 691
09:51:27.637936 IP xxx.xxx.xxx.xxx.56311 > 217.160.80.193.53: UDP, length 45
09:51:27.638203 IP xxx.xxx.xxx.xxx.49832 > 217.160.82.195.53: UDP, length 45
09:51:27.638558 IP xxx.xxx.xxx.xxx.45100 > 217.160.80.193.53: UDP, length 45
09:51:27.638841 IP xxx.xxx.xxx.xxx.40751 > 217.160.81.196.53: UDP, length 45
09:51:27.639090 IP xxx.xxx.xxx.xxx.15400 > 217.160.80.193.53: UDP, length 45
09:51:27.640279 IP 192.33.14.30.53 > xxx.xxx.xxx.xxx.54683: UDP, length 861
09:51:27.640641 IP xxx.xxx.xxx.xxx.42994 > 205.251.193.222.53: UDP, length 49
09:51:27.641838 IP 23.61.199.64.53 > xxx.xxx.xxx.xxx.13656: UDP, length 75
09:51:27.642357 IP xxx.xxx.xxx.xxx.50850 > 23.211.133.65.53: UDP, length 57
09:51:27.643439 IP 217.160.81.195.53 > xxx.xxx.xxx.xxx.42365: UDP, length 62
09:51:27.643538 IP 217.160.82.194.53 > xxx.xxx.xxx.xxx.28094: UDP, length 62
09:51:27.651518 IP 193.108.88.128.53 > xxx.xxx.xxx.xxx.6554: UDP, length 119
09:51:27.651878 IP xxx.xxx.xxx.xxx.18779 > 2.22.230.130.53: UDP, length 63
09:51:27.652855 IP 192.55.83.30.53 > xxx.xxx.xxx.xxx.30344: UDP, length 860
09:51:27.653222 IP xxx.xxx.xxx.xxx.60974 > 205.251.192.36.53: UDP, length 49
09:51:27.654541 IP 96.7.50.128.53 > xxx.xxx.xxx.xxx.20917: UDP, length 62
09:51:27.656047 IP 205.251.193.222.53 > xxx.xxx.xxx.xxx.42994: UDP, length 335
09:51:27.656517 IP xxx.xxx.xxx.xxx.59577 > 205.251.194.236.53: UDP, length 61
09:51:27.656820 IP xxx.xxx.xxx.xxx.29228 > 199.249.120.1.53: UDP, length 42
09:51:27.657617 IP 2.22.230.67.53 > xxx.xxx.xxx.xxx.64172: UDP, length 60
09:51:27.660843 IP 95.100.168.130.53 > xxx.xxx.xxx.xxx.42631: UDP, length 78
09:51:27.661332 IP xxx.xxx.xxx.xxx.51498 > 84.53.139.129.53: UDP, length 55
09:51:27.664356 IP 194.146.107.6.53 > xxx.xxx.xxx.xxx.63915: UDP, length 691
09:51:27.664745 IP xxx.xxx.xxx.xxx.7053 > 217.160.80.193.53: UDP, length 45
09:51:27.669345 IP 217.160.80.193.53 > xxx.xxx.xxx.xxx.29570: UDP, length 61
09:51:27.670896 IP 23.211.133.65.53 > xxx.xxx.xxx.xxx.50850: UDP, length 93
09:51:27.671397 IP xxx.xxx.xxx.xxx.48083 > 192.55.83.30.53: UDP, length 43
09:51:27.672499 IP 205.251.192.36.53 > xxx.xxx.xxx.xxx.60974: UDP, length 334
09:51:27.672595 IP 217.160.80.193.53 > xxx.xxx.xxx.xxx.35040: UDP, length 61
09:51:27.672694 IP 217.160.82.194.53 > xxx.xxx.xxx.xxx.48389: UDP, length 61
09:51:27.672771 IP 217.160.80.193.53 > xxx.xxx.xxx.xxx.56311: UDP, length 61
09:51:27.672870 IP 217.160.81.196.53 > xxx.xxx.xxx.xxx.24142: UDP, length 61
09:51:27.675649 IP 193.108.88.128.53 > xxx.xxx.xxx.xxx.29106: UDP, length 95
09:51:27.676171 IP xxx.xxx.xxx.xxx.63756 > 2.22.230.65.53: UDP, length 44
09:51:27.679194 IP 217.160.82.195.53 > xxx.xxx.xxx.xxx.49832: UDP, length 61
09:51:27.679295 IP 217.160.81.196.53 > xxx.xxx.xxx.xxx.40751: UDP, length 61
09:51:27.679370 IP 217.160.80.193.53 > xxx.xxx.xxx.xxx.45100: UDP, length 61
09:51:27.680771 IP 217.160.80.193.53 > xxx.xxx.xxx.xxx.15400: UDP, length 61
09:51:27.682372 IP 2.16.130.131.53 > xxx.xxx.xxx.xxx.8658: UDP, length 63
09:51:27.685197 IP 84.53.139.129.53 > xxx.xxx.xxx.xxx.51498: UDP, length 82
09:51:27.685719 IP xxx.xxx.xxx.xxx.51725 > 23.61.199.131.53: UDP, length 63
09:51:27.688401 IP 205.251.194.236.53 > xxx.xxx.xxx.xxx.59577: UDP, length 326
09:51:27.690010 IP 2.16.130.131.53 > xxx.xxx.xxx.xxx.33996: UDP, length 63
09:51:27.691586 IP 192.112.36.4.53 > xxx.xxx.xxx.xxx.10416: UDP, length 825
09:51:27.692067 IP xxx.xxx.xxx.xxx.23482 > 192.33.4.12.53: UDP, length 47
09:51:27.693312 IP 199.249.120.1.53 > xxx.xxx.xxx.xxx.29228: UDP, length 805
09:51:27.693687 IP xxx.xxx.xxx.xxx.32444 > 205.251.196.15.53: UDP, length 50
09:51:27.698206 IP 217.160.80.193.53 > xxx.xxx.xxx.xxx.7053: UDP, length 61
09:51:27.698273 IP 2.22.230.130.53 > xxx.xxx.xxx.xxx.58014: UDP, length 104
09:51:27.698984 IP xxx.xxx.xxx.xxx.60192 > 192.5.6.30.53: UDP, length 43
09:51:27.699784 IP 195.243.137.26.53 > xxx.xxx.xxx.xxx.24580: UDP, length 691
09:51:27.700132 IP xxx.xxx.xxx.xxx.9782 > 217.160.82.193.53: UDP, length 45
09:51:27.701374 IP 2.16.130.131.53 > xxx.xxx.xxx.xxx.30060: UDP, length 63
09:51:27.702983 IP 43.230.48.1.53 > xxx.xxx.xxx.xxx.28502: UDP, length 546
09:51:27.703322 IP xxx.xxx.xxx.xxx.33833 > 213.248.216.1.53: UDP, length 44

johnpoz

Open that up in wireshark and you can see way more info on what is being asked, what is being returned... See my edit above post.

But for example, you see that query to 192.33.4.12 that is one of the root servers.

;; QUESTION SECTION:
;12.4.33.192.in-addr.arpa.      IN      PTR

;; ANSWER SECTION:
12.4.33.192.in-addr.arpa. 10800 IN      PTR     c.root-servers.net.

pfguy2018

@johnpoz
So that would suggest that things are working right now? Which is good. I will repeat this traffic capture once things go south again.

I have installed Wireshark, but am not sure how to get my traffic into it from the text file I have. Any pointers there?

johnpoz

Not the text that is shown, the download button.