[solved] DNS root request - rootserver doesn't response via udp

[andreas_at_work]

Hey Community,

due to some requirements we would like to implement a separate unbound (with enabled full logging features) on a machine behind the pfsense. Unbound was not able to connect to the root servers via udp. After a few tests we identified the general problem:

pfsense (somehow) don't get any response for dns root request, doesn't matter if they come from a client or directly from the pfsense:

drill @199.7.91.13 .

tcpdump of this request:

[2.2.5-RELEASE][admin@internet-fw.localdomain]/tmp: grep '199.7' log.udp 
12:24:02.628288 IP 87.142.207.148.57947 > 199.7.91.13.53: 45119+ A? . (17)
12:24:07.709898 IP 87.142.207.148.32130 > 199.7.91.13.53: 45119+ A? . (17)
12:24:12.712889 IP 87.142.207.148.11598 > 199.7.91.13.53: 45119+ A? . (17)

Analysis we have done so far:

1. The same request via tcp does work:

drill -t @199.7.91.13 .

tcpdump of this request:

[2.2.5-RELEASE][admin@internet-fw.localdomain]/tmp: grep '199.7.91.13' log.tcp
12:23:07.336688 IP 87.142.207.148.54043 > 199.7.91.13.53: Flags [s], seq 2131652584, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 15240744 ecr 0], length 0
12:23:07.358575 IP 199.7.91.13.53 > 87.142.207.148.54043: Flags [S.], seq 1548891628, ack 2131652585, win 14480, options [mss 1452,sackOK,TS val 2534769972 ecr 15240744,nop,wscale 7], length 0
12:23:07.358623 IP 87.142.207.148.54043 > 199.7.91.13.53: Flags [.], ack 1, win 517, options [nop,nop,TS val 15240766 ecr 2534769972], length 0
12:23:07.358675 IP 87.142.207.148.54043 > 199.7.91.13.53: Flags [P.], seq 1:20, ack 1, win 517, options [nop,nop,TS val 15240766 ecr 2534769972], length 1927848+ A? . (17)
12:23:07.379791 IP 199.7.91.13.53 > 87.142.207.148.54043: Flags [.], ack 20, win 114, options [nop,nop,TS val 2534769993 ecr 15240766], length 0
12:23:07.379948 IP 199.7.91.13.53 > 87.142.207.148.54043: Flags [P.], seq 1:95, ack 20, win 114, options [nop,nop,TS val 2534769993 ecr 15240766], length 9427848*- 0/1/0 (92)
12:23:07.379978 IP 87.142.207.148.54043 > 199.7.91.13.53: Flags [.], ack 95, win 516, options [nop,nop,TS val 15240787 ecr 2534769993], length 0
12:23:07.380043 IP 87.142.207.148.54043 > 199.7.91.13.53: Flags [F.], seq 20, ack 95, win 517, options [nop,nop,TS val 15240787 ecr 2534769993], length 0
12:23:07.401486 IP 199.7.91.13.53 > 87.142.207.148.54043: Flags [F.], seq 95, ack 21, win 114, options [nop,nop,TS val 2534770014 ecr 15240787], length 0
12:23:07.401526 IP 87.142.207.148.54043 > 199.7.91.13.53: Flags [.], ack 96, win 517, options [nop,nop,TS val 15240809 ecr 2534770014], length 0

2\. Request for ".de" does also work:
[code]drill @199.7.91.13 de.[/code]
tcpdump:
[code][2.2.5-RELEASE][admin@internet-fw.localdomain]/tmp: grep '199.7' log.de.udp
12:25:45.102826 IP 87.142.207.148.60452 > 199.7.91.13.53: 35136+ A? de. (20)
12:25:45.125961 IP 199.7.91.13.53 > 87.142.207.148.60452: 35136- 0/6/10 (334)
[/code]

Does anyone have an idea why udp does not work?

Setup:
Hardware:
SYS-5018A-TN7B, http://www.supermicro.com/products/system/1U/5018/SYS-5018A-TN7B.cfm
16 Gig of Ram
80 Gig Intel SSD 
Lan bypass deactivated
PPPOE connection (50/10) via vdsl modem

Software:
2.2.5-RELEASE (amd64)
Mbuf set to 1000000

Thank you
Andreas[/s]

[johnpoz]

Are you saying that pfsense itself can not query via udp?  Or a client behind pfsense can not?  Either way it points to udp being blocked somewhere between where your doing the query and where sending the query.

Does pfsense have public IP on its wan, or is it behind a NAT?

"PPPOE connection (50/10) via vdsl modem"  So pfsense gets public or rfc1918 address?

I can tell you for sure that roots clearly answer udp queries ;)

[andreas_at_work]

Hey johnpoz,

it does not matter if its a client or the pfsense it self. This explicit (but valid) request seems to be the problem. I have tried from pfsense via drill or from a client (ubuntu) via dig.
In both scenarios it looks like the request is leaving the wan interface, but there is no answer.

ubuntu client:

dig @199.7.91.13 .

does not work.

dig @199.7.91.13 . +tcp

or

dig @199.7.91.13 de.

does work.

UDP is not blocked because

drill @199.7.91.13 de.

does work like expected from pfsense and client.

Pfsense does get a public ipv4 ip via PPPOE. In my first post it was '87.142.207.148'.

All i can imagine is, that the "network stack" or the network driver or my ISP discard the packet somehow. I cannot capture packets 'behind' the pppoe interface. Maybe i should set up a dns somewhere in the internet and perform the request towards this server.

greetings
Andreas

[johnpoz]

Can you query say 8.8.8.8 ??  This is a public dns server..

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> @8.8.8.8 .
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57426
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;.                              IN      A

;; AUTHORITY SECTION:
.                      318    IN      SOA    a.root-servers.net. nstld.verisign-grs.com. 2015112700 1800 900 604800 86400

;; Query time: 39 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Nov 27 09:09:37 CST 2015
;; MSG SIZE  rcvd: 103

[andreas_at_work]

nope.

Same problem. TCP (dig @8.8.8.8 . +tcp) does work, UDP (dig @8.8.8.8 .) does not.

[johnpoz]

So udp is blocked upstream from were your doing query from either pfsense nar router in front of pfsense or isp

[andreas_at_work]

UDP is not blocked, feel free to scroll up and take a look at my examples, i have posted two of them. Only the UDP request for "." does not work.

[Derelict]

It really does look like something upstream is jacking with your traffic. You see the packet going out and nothing comes back.

[andreas_at_work]

Yep. That seems to be the case.

I will do two steps now:

1. setup a DNS in the Internet and perform this request towards this server. Then i will be able to see, if or how the request really do leave my "internet access".
2. Post this case in the German subfolder because of the "non-starndard" configuration. I am using an IAD-Router as a pppoe modem. By using it in the "bridged mode" it do not forward the vlan tags. Maybe this might be somehow a problem.

Thanks anyway :)

[johnpoz]

"1. setup a DNS in the Internet and perform this request towards this server."

What do you think the query to 8.8.8.8 was??  There are plenty of public dns you could send your request too also, no need to setup anything..

Does your isp expect you to use their dns?  Can you do anything UDP outbound?  NTP for example?  Query a public ntp server.. or just the ntp pool..

[andreas_at_work]

Dear johnpoz,

at the moment i am not able to check the logfiles of the google dns. This might change in the future, but is not possible yet… ;)

My idea is: As soon as i setup my own dns i will be able to see if this damn request reached the server or got caught somewhere between the pfSense and my own dns.

In this case i don't want to use the dns of my isp. The use case is to monitor malicious hostnames (means hostnames which where used for "bad stuff") and detect modifications. Based on this modifications it is sometimes possible to identify some hints about future attacks or attack vectors.

Outbound UDP is working without any problem. I will try to create very small ntp and snmp packets tomorrow.

Thanks
Andreas

[andreas_at_work]

Problem solved. Since i have replaced the vdsl modem and configured pfSense to use the vlans on wan interface the problem did not appear anymore.