DNS Resolver returns SERVFAIL for a valid domain

jofrep · Mar 22, 2019, 5:51 PM

Hello,
I use the pfsense 2.4.4 and pfBlockerNG-devel 2.2.5_22
I have a problem with a particular domain that is not resolving properly when using the pfsense resolver:

The examples below are in the firewall but I have the same result when using a client.

When I try to resolve www.ara.cat in the firewall forcing to use the local resolver it can't resolve the name.

[2.4.4-RELEASE][admin@fw.xxxx.com]/var/log: nslookup www.ara.cat 127.0.0.1
Server: 127.0.0.1
Address: 127.0.0.1#53

** server can't find www.ara.cat: SERVFAIL

If I don't force the local resolver it fallsback to the second resolver 1.1.1.1 and it works:

[2.4.4-RELEASE][admin@fw.xxxx.com]/var/log: nslookup www.ara.cat
;; Got SERVFAIL reply from 127.0.0.1, trying next server
Server: 1.1.1.1
Address: 1.1.1.1#53

Non-authoritative answer:
www.ara.cat canonical name = www.ara.cat.cdn.bitban.net.
www.ara.cat.cdn.bitban.net canonical name = caching.ara.edge2befaster.com.
caching.ara.edge2befaster.com canonical name = cec01.euc.edgetcdn.com.
Name: cec01.euc.edgetcdn.com
Address: 51.255.81.138

on the other hand the if I use another subdomain forcing to use the local resolver it works fine

2.4.4-RELEASE][admin@fw.xxxx.com]/var/log: nslookup ara.cat 127.0.0.1
Server: 127.0.0.1
Address: 127.0.0.1#53

Non-authoritative answer:
Name: ara.cat
Address: 176.34.179.218

Why is this happening and why it can't resolve that name?
How can I fix this?

Thanks

johnpoz · Mar 25, 2019, 2:05 AM

@jofrep said in DNS Resolver returns SERVFAIL for a valid domain:

www.ara.cat

[2.4.4-RELEASE][admin@sg4860.local.lan]/root: dig www.ara.cat

; <<>> DiG 9.12.2-P1 <<>> www.ara.cat
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48284
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.ara.cat.                   IN      A

;; ANSWER SECTION:
www.ara.cat.            3528    IN      CNAME   www.ara.cat.cdn.bitban.net.
www.ara.cat.cdn.bitban.net. 3528 IN     CNAME   caching.ara.edge2befaster.com.
caching.ara.edge2befaster.com. 3528 IN  CNAME   cec01.usg.edgetcdn.com.
cec01.usg.edgetcdn.com. 3529    IN      A       185.59.223.130
cec01.usg.edgetcdn.com. 3529    IN      A       74.208.86.233
cec01.usg.edgetcdn.com. 3529    IN      A       185.152.67.178
cec01.usg.edgetcdn.com. 3529    IN      A       74.208.80.126

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Mar 24 20:54:21 CDT 2019
;; MSG SIZE  rcvd: 220

[2.4.4-RELEASE][admin@sg4860.local.lan]/root:

I would suggest you do a +trace with did to find out where your failing in the path.

When doing a +trace you will have to walk down through all the cnams... BTW - that is HORRIBLE setup... that many cnames is not good practice.

Any sort of basic analysis of that chain of nonsense finds errors.

www.ara.cat.cdn.bitban.net/CNAME: A query for www.ara.cat.cdn.bitban.net results in a NOERROR response,
while a query for its ancestor, cdn.bitban.net, returns a name error (NXDOMAIN), which indicates that subdomains of
 cdn.bitban.net, including www.ara.cat.cdn.bitban.net, don't exist.
(205.251.192.153,
205.251.195.183,
205.251.197.5,
205.251.198.35,
2600:9000:5300:9900::1,
2600:9000:5303:b700::1,
2600:9000:5305:500::1,
2600:9000:5306:2300::1,
UDP_-_EDNS0_4096_D_K)

Such setups with that many chains of cnames make for horrible resolving time.. An any point in the chain that fails the end record being looked up can fail.

jofrep · Mar 25, 2019, 7:32 PM

@johnpoz what I don't understand is that if I use an external DNS Server it is able to resolve but if I use the one in the firewall, no. I tested all the CNAME chain and I think the problem is in caching.ara.edge2befaster.com

With the local resolver it fails:

[2.4.4-RELEASE][admin@fw.xxxx.com]/root: nslookup caching.ara.edge2befaster.com 127.0.0.1
Server:		127.0.0.1
Address:	127.0.0.1#53

** server can't find caching.ara.edge2befaster.com: SERVFAIL

while if I use an external, it works:

[2.4.4-RELEASE][admin@fw.xxxxx.com]/root: nslookup caching.ara.edge2befaster.com 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
caching.ara.edge2befaster.com	canonical name = cec01.euc.edgetcdn.com.
Name:	cec01.euc.edgetcdn.com
Address: 51.255.81.138

on the other hand dig, if I use +trace, fails always, regadless of the DNS Server I use.

[2.4.4-RELEASE][admin@fw.xxxxx.com]/root: dig +trace caching.ara.edge2befaster.com

; <<>> DiG 9.12.2-P1 <<>> +trace caching.ara.edge2befaster.com
;; global options: +cmd
.			445467	IN	NS	h.root-servers.net.
.			445467	IN	NS	c.root-servers.net.
.			445467	IN	NS	a.root-servers.net.
.			445467	IN	NS	d.root-servers.net.
.			445467	IN	NS	b.root-servers.net.
.			445467	IN	NS	m.root-servers.net.
.			445467	IN	NS	g.root-servers.net.
.			445467	IN	NS	j.root-servers.net.
.			445467	IN	NS	l.root-servers.net.
.			445467	IN	NS	f.root-servers.net.
.			445467	IN	NS	i.root-servers.net.
.			445467	IN	NS	e.root-servers.net.
.			445467	IN	NS	k.root-servers.net.
.			445467	IN	RRSIG	NS 8 0 518400 20190406170000 20190324160000 16749 . rLXyu/LEACofze4mu6Vf4OTFpN27oua9xwvyU0qflNfN1J4EDyfqAI5O 8/+wUav7qT5qM6VwK57cr9ICgB4FFh5MAxC48bvmDmC+zpAbeecOmTst VZQw869wnlXgMQw5xvF+VF44/CuJus831qhFnDqseC8BaqdzhlHg4igw z5k12SufvvSR1+tkdedsN+gNpyY29/cBfnzLOjh47gYUmnLVZgXIJCFQ Gn16YOOJxm+4TYBb/HIv+GpOoTUvK2AFjuJq491ZQVypFwftXAwiN53S J0tW/M4aZlY9nEDugVFHQs1AvGU/3STjHTqbOuEYG0UUn94JCznd2H2e BMDfxw==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

com.			172800	IN	NS	h.gtld-servers.net.
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	m.gtld-servers.net.
com.			172800	IN	NS	e.gtld-servers.net.
com.			172800	IN	NS	l.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
com.			172800	IN	NS	j.gtld-servers.net.
com.			172800	IN	NS	f.gtld-servers.net.
com.			172800	IN	NS	k.gtld-servers.net.
com.			172800	IN	NS	i.gtld-servers.net.
com.			172800	IN	NS	c.gtld-servers.net.
com.			172800	IN	NS	g.gtld-servers.net.
com.			172800	IN	NS	d.gtld-servers.net.
com.			86400	IN	DS	30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.			86400	IN	RRSIG	DS 8 1 86400 20190407170000 20190325160000 16749 . Qy/AmWqETxJm2ll5xmbvAqgMFZbklRsHsiwqSAdE0pUkmrhEFQyrKAQZ ESUZ3JWTwtSsXy6/g8DJ7HV1yMg3BnacNK6LM0jb8OZ/GYqXhDs1Nn5U Zak3f3O/+WmpKfajqZi21xv2MYxxomxy2dYGyM7SoJcEOJF/I6/3fozE mKiB0UWTGx0w6mkW6xEG14Q5905rHc2jy8oDvMttd3HJFwdpJqRw5hXw ZAnMMspSIAiOxFbGx29NrkgHAu7athuKXHtUmFEXOfGYAhEPAYYL9qQG ejy7zYznpwNjrkLq0O8A4eN9sXAFSJlazGNBazXBMuq2g7+T1b3/Zgl9 stR/Bg==
;; Received 1189 bytes from 192.33.4.12#53(c.root-servers.net) in 16 ms



edge2befaster.com.	172800	IN	NS	ns1.servotic.net.
edge2befaster.com.	172800	IN	NS	ns2.servotic.net.
edge2befaster.com.	172800	IN	NS	ns3.servotic.net.
edge2befaster.com.	172800	IN	NS	ns4.servotic.net.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20190401044736 20190325033736 16883 com. BDlPwLf7uAMZUG7A6dWBmY4kWfxNQ2OaI6ZcbcPfcrqxexOPBnN+a2FO PS57EdevQeaIn0fuGbcEhjOwiKrmZKtBdhXEpLOOUqPKeAY86uN1zoL6 cNkx9NCFaECjv5pmyKKBlI5dGFQspo+qZ2n2JsG2ujDF69BGBLhXQn0q gQg=
5HETG8IO18HIO1MIR701P3P4Q0JF7ES5.com. 86400 IN NSEC3 1 1 0 - 5HEUP9CBJ16U15L8TB1HI62O6RMMSAVK NS DS RRSIG
5HETG8IO18HIO1MIR701P3P4Q0JF7ES5.com. 86400 IN RRSIG NSEC3 8 2 86400 20190331041613 20190324030613 16883 com. C+/Hs33q24x3XqJAlWmIJV9wmuAlDA5vSJUbL9ZOtwJm1buERSs+WJON ZlXw7Tg4j0ivWGWTEuCynbwV4tRe4YAnxbc7g8fuXA1bvGoeKCA8NZxp RCY2xNaLXOEoBhLIdAe2RZlMONJtImJtm/48Ra+UG7InSvsZQJJ5NqRM 8eI=
;; Received 627 bytes from 192.55.83.30#53(m.gtld-servers.net) in 31 ms

but works as nslookup when not using +trace. It is able to resolve it when using an external DNS Server:

[2.4.4-RELEASE][admin@fw.xxxx.com]/root: dig  caching.ara.edge2befaster.com

; <<>> DiG 9.12.2-P1 <<>> caching.ara.edge2befaster.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47126
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;caching.ara.edge2befaster.com.	IN	A

;; ANSWER SECTION:
caching.ara.edge2befaster.com. 63 IN	CNAME	cec01.euc.edgetcdn.com.
cec01.euc.edgetcdn.com.	63	IN	A	51.255.81.138

;; Query time: 7 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Mon Mar 25 20:28:14 CET 2019
;; MSG SIZE  rcvd: 107

but fails with the local one:

[2.4.4-RELEASE][admin@fw.xxxx.com]/root: dig caching.ara.edge2befaster.com @127.0.0.1

; <<>> DiG 9.12.2-P1 <<>> caching.ara.edge2befaster.com @127.0.0.1
;; global options: +cmd
;; connection timed out; no servers could be reached

what is diferent in the local resolver?
I see in the trace that DNSSEC is used.
Could that be the reason?

johnpoz · Mar 25, 2019, 7:33 PM

Because your resolver actually wanting to talk too the NS in question... When you ask 1.1.1.1 or whatever you have no idea from where its resolving,.. And he for sure has a different network path than your connection.

jofrep · Mar 25, 2019, 8:03 PM

Is there a mitigaton I could use?
I could hardcode the final IP to the name but this is risky.
Can I indicate that for a given name it should use an external resolver?

johnpoz · Mar 25, 2019, 8:17 PM

You need to figure out what is wrong with your path to that NS.. Contact your isp... What does a traceroute look like to it?

As for a mitigation of the problem, simple is to just create a domain override for that domain to ask say googledns, or something.. Or your ISP dns, does that resolve it?

Lets be clear if you can not resolve
caching.ara.edge2befaster.com

Is because you can not talk to the NS for that domain, to find out it just points to a freaking another cname... Have you contacted the idiots running this domain dns?

Most of these NS are amazon NS... They charge you per query... Do they do this sort of nonsense on purpose to force like a 1000 freaking queries that should take one... And then have ttls on this shit of 5 minutes..

jofrep · Mar 25, 2019, 8:24 PM

No, but I should!

Thanks, my problem is certainly accesing the NS ns1.servotic.net (or any other of the NS) from that location. If I ssh to another server in a different network I can query it without problems.

Thanks for the help.

johnpoz · Mar 25, 2019, 8:31 PM

ns1.servotic.net 82.223.244.165
ns2.servotic.net 195.190.78.1
ns3.servotic.net 195.21.32.43
ns4.servotic.net 192.83.254.181

You can not reach any of those? Where does your trace die, later on or right away just into your isp network?

jofrep · Mar 25, 2019, 8:38 PM

@johnpoz later. I can trace the exit and it dies at hop 10 to 15. If I query the server directly for the name I get no response, but if I do it from another network it works.

johnpoz · Mar 26, 2019, 7:34 AM

for all 4 of those IPs?

jofrep · Mar 26, 2019, 7:34 AM

@johnpoz Yes :-S

The same result for the 4 IPs:

The DNS request are ignored.
Ping works
Traceroute dies what looks like close to the destination

see and example below

 7  145.254.2.179 (145.254.2.179)  9.291 ms  9.670 ms
    145.254.2.195 (145.254.2.195)  9.125 ms
 8  decix.bb-a.fra3.fra.de.oneandone.net (80.81.192.123)  10.427 ms  10.224 ms  9.675 ms
 9  ae-10-0.bb-a.bap.rhr.de.oneandone.net (212.227.120.147)  15.933 ms  17.048 ms  15.870 ms
10  ae-1-0.bb-a.bv.crb.fr.oneandone.net (212.227.120.41)  23.707 ms  23.173 ms  23.215 ms
11  irb-66.bb-a.mad2.mad.es.oneandone.net (212.227.120.17)  40.407 ms  42.199 ms  40.061 ms
12  lpwro09.xe-1-0-0.arsys.es (82.223.200.41)  43.407 ms  43.624 ms  43.807 ms
13  * * *
14  82.223.244.165 (82.223.244.165)  44.600 ms  44.419 ms  44.244 ms
15  * * *
16  * * *
17  * * *

johnpoz · Mar 26, 2019, 9:18 AM

But you can ping them? Well if you can ping them, its not a connectivity thing.. But they don't answer your query???

jofrep · Mar 26, 2019, 10:05 AM

@johnpoz yes, nothing. Like if they dropped the request.

johnpoz · Mar 26, 2019, 10:25 AM

Odd... both udp and tcp queries? Even for their own SOA?

Or just that one record?

; <<>> DiG 9.12.3-P1 <<>> @ns2.servotic.net servotic.net SOA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21175
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 5
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;servotic.net.                  IN      SOA

;; ANSWER SECTION:
servotic.net.           300     IN      SOA     ns1.servotic.net. hostmaster.servotic.com. 2018111730 10800 900 604800 86400

;; AUTHORITY SECTION:
servotic.net.           300     IN      NS      ns2.servotic.net.
servotic.net.           300     IN      NS      ns4.servotic.net.
servotic.net.           300     IN      NS      ns3.servotic.net.
servotic.net.           300     IN      NS      ns1.servotic.net.

;; ADDITIONAL SECTION:
ns1.servotic.net.       300     IN      A       82.223.244.165
ns2.servotic.net.       300     IN      A       195.190.78.1
ns3.servotic.net.       300     IN      A       195.21.32.43
ns4.servotic.net.       300     IN      A       192.83.254.181

;; Query time: 29 msec
;; SERVER: 195.190.78.1#53(195.190.78.1)
;; WHEN: Tue Mar 26 05:22:53 Central Daylight Time 2019
;; MSG SIZE  rcvd: 236

jofrep · Mar 26, 2019, 10:54 AM

@johnpoz UDP are ignored, TCP looks like reset:

[2.4.4-RELEASE][admin@fw.xxxx.com]/root: dig @82.223.244.165 +tcp caching.ara.edge2befaster.com
;; communications error to 82.223.244.165#53: connection reset

and the dump:

[2.4.4-RELEASE][admin@fw.xxxx.com]/root: tcpdump -n port 53 -i igb0 -Xs0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes


11:46:47.177332 IP 192.168.0.139.27293 > 82.223.244.165.53: Flags [S], seq 1933958727, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 265315576 ecr 0], length 0
	0x0000:  4500 003c 0000 4000 4006 3204 c0a8 008b  E..<..@.@.2.....
	0x0010:  52df f4a5 6a9d 0035 7345 de47 0000 0000  R...j..5sE.G....
	0x0020:  a002 fecc 08e7 0000 0204 05b4 0103 0307  ................
	0x0030:  0402 080a 0fd0 64f8 0000 0000            ......d.....
11:46:47.221421 IP 82.223.244.165.53 > 192.168.0.139.27293: Flags [S.], seq 2916622363, ack 1933958728, win 14480, options [mss 1452,sackOK,TS val 3687093890 ecr 265315576,nop,wscale 6], length 0
	0x0000:  4500 003c 0000 4000 3506 3d04 52df f4a5  E..<..@.5.=.R...
	0x0010:  c0a8 008b 0035 6a9d add8 201b 7345 de48  .....5j.....sE.H
	0x0020:  a012 3890 954c 0000 0204 05ac 0402 080a  ..8..L..........
	0x0030:  dbc4 9682 0fd0 64f8 0103 0306            ......d.....
11:46:47.221573 IP 192.168.0.139.27293 > 82.223.244.165.53: Flags [.], ack 1, win 510, options [nop,nop,TS val 265315620 ecr 3687093890], length 0
	0x0000:  4500 0034 0000 4000 4006 320c c0a8 008b  E..4..@.@.2.....
	0x0010:  52df f4a5 6a9d 0035 7345 de48 add8 201c  R...j..5sE.H....
	0x0020:  8010 01fe 08df 0000 0101 080a 0fd0 6524  ..............e$
	0x0030:  dbc4 9682                                ....
11:46:47.221918 IP 192.168.0.139.27293 > 82.223.244.165.53: Flags [P.], seq 1:73, ack 1, win 510, options [nop,nop,TS val 265315621 ecr 3687093890], length 725072+ [1au] A? caching.ara.edge2befaster.com. (70)
	0x0000:  4500 007c 0000 4000 4006 31c4 c0a8 008b  E..|..@.@.1.....
	0x0010:  52df f4a5 6a9d 0035 7345 de48 add8 201c  R...j..5sE.H....
	0x0020:  8018 01fe 0927 0000 0101 080a 0fd0 6525  .....'........e%
	0x0030:  dbc4 9682 0046 13d0 0120 0001 0000 0000  .....F..........
	0x0040:  0001 0763 6163 6869 6e67 0361 7261 0d65  ...caching.ara.e
	0x0050:  6467 6532 6265 6661 7374 6572 0363 6f6d  dge2befaster.com
	0x0060:  0000 0100 0100 0029 1000 0000 0000 000c  .......)........
	0x0070:  000a 0008 9236 b8b5 1cf1 5c23            .....6....\#
11:46:47.266090 IP 82.223.244.165.53 > 192.168.0.139.27293: Flags [.], ack 73, win 227, options [nop,nop,TS val 3687093901 ecr 265315621], length 0
	0x0000:  4500 0034 b691 4000 3506 867a 52df f4a5  E..4..@.5..zR...
	0x0010:  c0a8 008b 0035 6a9d add8 201c 7345 de90  .....5j.....sE..
	0x0020:  8010 00e3 fb3c 0000 0101 080a dbc4 968d  .....<..........
	0x0030:  0fd0 6525                                ..e%
11:46:47.266333 IP 82.223.244.165.53 > 192.168.0.139.27293: Flags [R.], seq 1, ack 73, win 227, options [nop,nop,TS val 3687093901 ecr 265315621], length 0
	0x0000:  4500 0034 b692 4000 3506 8679 52df f4a5  E..4..@.5..yR...
	0x0010:  c0a8 008b 0035 6a9d add8 201c 7345 de90  .....5j.....sE..
	0x0020:  8014 00e3 fb38 0000 0101 080a dbc4 968d  .....8..........
	0x0030:  0fd0 6525                                ..e%
^C
6 packets captured

same result regardless of the record

johnpoz · Mar 26, 2019, 12:16 PM

Hmmmm, since firewalls don't normally send RST, nor would they normally answer a s with sa if they are blocking I have to assume your moving up the stack and something decided not to answer your query, and closed the session with R..

Possible ACL on their NS(s)...