unbound not resolving - dig on ssh session works

scope · Apr 12, 2019, 7:56 AM

Good evening!

I'm currently in the process of evaluating pfsense as a new platform for my home network.
First things first: Thanks for this awesome project!

I have a VDSL2 line, a separate DSL-modem and currently use a zotac zbox ci (dual gigabyte nics) for my setup.
So I'm using PPPoE, which works fine.... but...

It appears that unbound is not resolving external domains. I digged through the forum for a while now, but I can't find a thread that seems to be similar to my case.

Here are some details about my setup:

I currently run 2.4.4-RELEASE-p2 (amd64).
LAN interface on 10.0.0.2/24
WAN interface on PPPoE for IPv4 with credentials for my provider and none for IPv6
No DNS servers configured in System -> General Setup
DNS Resolver active, interfaces set to All / All, Static DHCP checked and some host overrides
WAN connects without problems and WAN_PPPOE is the only (and default) gateway

It appears to me, that this is a fairly basic setup. Yet, something must be wrong (or I've been too stupid and misconfigured something...).

If I manually set DNS to e.g. 1.1.1.1 for my Windows box, everything works as excpected, so my internet connection seems to be ok (latency and bandwidth is ok, too).

If I try to use 10.0.0.2 as a resolver, I just get a timeout.

The resolver log just show something like:

Apr 12 00:18:04	unbound	25146:0	notice: init module 0: validator
Apr 12 00:18:04	unbound	25146:0	notice: init module 1: iterator
Apr 12 00:18:04	unbound	25146:0	info: start of service (unbound 1.8.1).

So I tried to dig into this (literally).

What I've tried so far:

Unbound is running and answers queries that correspond to the configured host overrides:

[2.4.4-RELEASE][admin@pfsense.home]/root: dig zbox.home

; <<>> DiG 9.12.2-P1 <<>> zbox.home
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57640
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;zbox.home.           IN      A

;; ANSWER SECTION:
zbox.home.    3600    IN      A       10.0.0.8

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Apr 12 00:20:00 CEST 2019
;; MSG SIZE  rcvd: 64

I then dumped the query list:

[2.4.4-RELEASE][admin@pfsense.home]/: unbound-control -c /var/unbound/unbound.conf dump_requestlist
thread #0
#   type cl name    seconds    module status
  0   NS IN . - iterator wait for 202.12.27.33
  1    A IN 0.pfsense.pool.ntp.org. 349.483801 iterator wants NS IN .
  2    A IN 0.pfsense.pool.ntp.org.home. 324.312648 iterator wants NS IN .
  3    A IN pkg.pfsense.org. 324.193176 iterator wants NS IN .
  4    A IN pkg.pfsense.org.home. 294.051367 iterator wants NS IN .
  5    A IN stun.gigaset.net. 254.992681 iterator wants NS IN .
  6    A IN dynreg.gigaset.net. 276.110724 iterator wants NS IN .
  7 AAAA IN 0.pfsense.pool.ntp.org. 339.347040 iterator wants NS IN .
  8 AAAA IN 0.pfsense.pool.ntp.org.home. 309.203612 iterator wants NS IN .
  9 AAAA IN pkg.pfsense.org. 309.141421 iterator wants NS IN .
 10 AAAA IN pkg.pfsense.org.home. 349.483801 iterator wants NS IN .
 11  SRV IN _https._tcp.pkg.pfsense.org. 339.351057 iterator wants NS IN .

So it's collecting queries, but these just seem to be stuck and waiting forever.

If I manually try to query I get:

[2.4.4-RELEASE][admin@pfsense.home]/root: dig 0.pfsense.pool.ntp.org.

; <<>> DiG 9.12.2-P1 <<>> 0.pfsense.pool.ntp.org.
;; global options: +cmd
;; connection timed out; no servers could be reached

Talking to a root server works, so these seem not to be blocked by my ISP:

[2.4.4-RELEASE][admin@pfsense.home]/root: dig @202.12.27.33 +dnssec 0.pfsense.pool.ntp.org.

; <<>> DiG 9.12.2-P1 <<>> @202.12.27.33 +dnssec 0.pfsense.pool.ntp.org.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52472
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 9, ADDITIONAL: 13
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;0.pfsense.pool.ntp.org.                IN      A

;; AUTHORITY SECTION:
org.                    172800  IN      NS      d0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.
org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    86400   IN      DS      9795 7 2 3922B31B6F3A4EA92B19EB7B52120F031FD8E05FF0B03BAFCF9F891B FE7FF8E5
org.                    86400   IN      DS      9795 7 1 364DFAB3DAF254CAB477B5675B10766DDAA24982
org.                    86400   IN      RRSIG   DS 8 1 86400 20190424170000 20190411160000 25266 . mUjeRKpkX1lXiqP1N9Dh0HDlMkObK5VdrLTbmYMajgb8muftE6nmzX1y tuyzp9lciFvSnBrjLfy+OPlxKCenMZVej6wwPIUtSlK/JDOgFBedAP4h +9iuUkrY4WzhdRbWcwHiWm6mh1FRy/3ZWPdXV1zys0fyn3WLr9/U0php rlwxQPIEH9UXmpJHeqnypN+9xmHfQnFT1RfMJCdVKumKyms1tJQVVcjg Q+G9+HtUcDw3xLsbvwVUa4mweeFcMoljpP7KM/GHRgxJj8wZ/sxYAIgY SlvXe8D75EK2YaRRCKnkSH1sOa9Gc8CxjWqfwWDfX42aWRprdEDOQWSa i5bMig==

;; ADDITIONAL SECTION:
a0.org.afilias-nst.info. 172800 IN      A       199.19.56.1
a2.org.afilias-nst.info. 172800 IN      A       199.249.112.1
b0.org.afilias-nst.org. 172800  IN      A       199.19.54.1
b2.org.afilias-nst.org. 172800  IN      A       199.249.120.1
c0.org.afilias-nst.info. 172800 IN      A       199.19.53.1
d0.org.afilias-nst.org. 172800  IN      A       199.19.57.1
a0.org.afilias-nst.info. 172800 IN      AAAA    2001:500:e::1
a2.org.afilias-nst.info. 172800 IN      AAAA    2001:500:40::1
b0.org.afilias-nst.org. 172800  IN      AAAA    2001:500:c::1
b2.org.afilias-nst.org. 172800  IN      AAAA    2001:500:48::1
c0.org.afilias-nst.info. 172800 IN      AAAA    2001:500:b::1
d0.org.afilias-nst.org. 172800  IN      AAAA    2001:500:f::1

;; Query time: 152 msec
;; SERVER: 202.12.27.33#53(202.12.27.33)
;; WHEN: Fri Apr 12 01:04:07 CEST 2019
;; MSG SIZE  rcvd: 824

If I follow the path down, this results in a.ntpns.org being the nameserver to ask.

Which does not work by name:

[2.4.4-RELEASE][admin@pfsense.home]/root: dig @a.ntpns.org +dnssec 0.pfsense.pool.ntp.org.
dig: couldn't get address for 'a.ntpns.org': not found

But by IP:

[2.4.4-RELEASE][admin@pfsense.home]/root: dig @207.171.17.42 +dnssec 0.pfsense.pool.ntp.org.

; <<>> DiG 9.12.2-P1 <<>> @207.171.17.42 +dnssec 0.pfsense.pool.ntp.org.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32168
;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;0.pfsense.pool.ntp.org.                IN      A

;; ANSWER SECTION:
0.pfsense.pool.ntp.org. 150     IN      A       5.189.146.13
0.pfsense.pool.ntp.org. 150     IN      A       37.120.184.82
0.pfsense.pool.ntp.org. 150     IN      A       176.9.102.215
0.pfsense.pool.ntp.org. 150     IN      A       87.118.124.35

;; Query time: 22 msec
;; SERVER: 207.171.17.42#53(207.171.17.42)
;; WHEN: Fri Apr 12 01:08:32 CEST 2019
;; MSG SIZE  rcvd: 192

Of course, if I knew all the IPs beforehand, I wouldn't need to use DNS ;)

A full trace does not work either:

[2.4.4-RELEASE][admin@pfsense.home]/root: dig +dnssec +trace 0.pfsense.pool.ntp.org.

; <<>> DiG 9.12.2-P1 <<>> +dnssec +trace 0.pfsense.pool.ntp.org.
;; global options: +cmd
;; connection timed out; no servers could be reached

At least I'd expect to see the list of root servers here.

Any insight? Hopefully I'm just doing it wrong...

Greetings and thanks for your help!

Gertjan · Apr 12, 2019, 7:56 AM

@scope said in unbound not resolving - dig on ssh session works:

The resolver log just show something like:

It show what you want to see ;)
If you want, inform it to show more details (like what servers it can't reach).

Btw - your last command :

dig +dnssec +trace 0.pfsense.pool.ntp.org.

runs fully for me.

Also :

dig @a.ntpns.org +dnssec 0.pfsense.pool.ntp.org.

works for me, @dig @a.ntpns.org will get resolved to "207.171.17.42" (and it's IPv6 counter part)

Although you gave good info, I can't see (from here) why 'some' NS servers can't be contacted.

What are (floating) firewall rules ? NAT specials ? packages ?

As you said : the Resolvers works fine right after setup. Nothing need to be done to make it work better.
I used pppoe myself for years, many do still today. That couldn't be the issue.

scope · Apr 12, 2019, 12:48 PM

Hi,

thanks for your feedback!

Ok, did not look closely enough on the advanced tab - log level ist easy to overlook :)

Fortunately I was able to resolve my issue, though I still don't know what happened exactly.

I tried to take as much out of the equation as possible - my last try was to take even pfsense out...
So I started unbound as a docker container on one of my machines (pfsense still in use for internet gateway, though)...
That showed the exact same symptoms.

That was a serious WTF moment :|

I then googled for a long time, and tried to debug that single docker unbound instance - without success. It seemed as if something dropped packets, but I did not know what.

I eventually found a promising thread:
https://forums.freebsd.org/threads/unbound-very-slow-and-or-dns-address-could-not-be-found.57493/

That setup seemed to be similar. I as well use a Fritzbox that is tricked into working only as a dsl modem.
I tried to change the dsl protocol version as suggested in the thread - without success.

I then took a deeper look at the fritzbox again. There are various modes the fritzbox can be turned into a dsl modem.
I used the variant, where the fritzbox itself handles VLAN tag 7 for VDSL.
I found various threads, that this bridge mode suffers a drop-some-PPPoE-packets problem.

I then switched the box to full_bridge - of course I had to adjust pfsense to do the VLAN tagging itself on the WAN interface.
And guess what: it worked.

Unbound now works fine. Why the bridge mode of the fritzbox dropped just these specific DNS packets - I have absolutely no idea. Of course, I mustn't complain, since I use the box in a way it was not designed for (at least not officially) :)

So basically everything was set up correctly and I did look at the wrong end...