Problems with unbound after ISP change

Azaron · May 5, 2016, 1:05 PM

Hello,

after I upgraded to 2.3 I encountered some strange errors within my pfsense unbound dns resolver that I can't fix myself, because of lack of knowledge…
Basically there are 2 things that do no longer work, but worked before in 2.2.

1.) DNSSEC Support.
I can no longer check „Enable DNSSEC Support“. Whenever I enable it I get logentries saying „info: failed to prime trust anchor -- could not fetch DNSKEY rrset . DNSKEY IN“.
DNS queries on clients fail afterwards.

2.) Some domains / Subdomains to not resolve on Clients.
When I turn off DNSSEC unbound looks working and resolves many domainnames.
e.g. www.google.com, fedoraproject.org …
but a few like download.fedoraproject.org are not resolved.
„*** download.fedoraproject.org wurde von pfSense nicht gefunden: Server failed.“
On pfsense itself nslookup to download.fedoraproject.org works fine it shows the answer wildcard.fedoraproject.org.

I'm using google DNS Servers (8.8.8.8 and 8.8.4.4) as general DNS-Servers in my pfsense.

I tried since a whole day every combination of config that I could imagine over the webinterface but nothing solved this.
When I deactivate dns resolver and use dns forwarder as dns server all domain queries on the clients get resolved.

As this all worked before the update I assume something got broken in my unbound configuration that I can't reach over the webinterface and as I'm a bit clueless what it could be I kindly ask if someone here might have an idea.

johnpoz · May 5, 2016, 1:17 PM

"I'm using google DNS Servers (8.8.8.8 and 8.8.4.4) as general DNS-Servers in my pfsense."

Are you using forwarder mode or resolver mode in unbound? To be honest there is little reason to have anything in pfsense general setting for dns other than 127.0.0.1 so your asking the same dns service your clients are using..

You know the resolver in resolver mode has to be able to query anywhere via 53 both udp and tcp.. If your isp blocks some of this traffic than you will have all kinds of issues with resolver mode.. If your saying your having issues with dnssec I resolving stuff I would guess your isp is blocking or having issues or something between pfsense and internet is blocking or intercepting your dns traffic… Can you query the root servers directly?

Marc05 · May 5, 2016, 9:39 PM

@johnpoz:

"I'm using google DNS Servers (8.8.8.8 and 8.8.4.4) as general DNS-Servers in my pfsense."

Are you using forwarder mode or resolver mode in unbound? To be honest there is little reason to have anything in pfsense general setting for dns other than 127.0.0.1 so your asking the same dns service your clients are using..

You know the resolver in resolver mode has to be able to query anywhere via 53 both udp and tcp.. If your isp blocks some of this traffic than you will have all kinds of issues with resolver mode.. If your saying your having issues with dnssec I resolving stuff I would guess your isp is blocking or having issues or something between pfsense and internet is blocking or intercepting your dns traffic… Can you query the root servers directly?

Would you mind explaining how DNS Resolver would work if 127.0.0.1 is put under General Settings? TIA

cmb · May 5, 2016, 10:02 PM

Is your system clock really far off? Having a significantly wrong date and time on the system could be what's making DNSSEC fail.

kpa · May 6, 2016, 9:40 AM

@Marc05:

@johnpoz:

"I'm using google DNS Servers (8.8.8.8 and 8.8.4.4) as general DNS-Servers in my pfsense."

Are you using forwarder mode or resolver mode in unbound? To be honest there is little reason to have anything in pfsense general setting for dns other than 127.0.0.1 so your asking the same dns service your clients are using..

You know the resolver in resolver mode has to be able to query anywhere via 53 both udp and tcp.. If your isp blocks some of this traffic than you will have all kinds of issues with resolver mode.. If your saying your having issues with dnssec I resolving stuff I would guess your isp is blocking or having issues or something between pfsense and internet is blocking or intercepting your dns traffic… Can you query the root servers directly?

Would you mind explaining how DNS Resolver would work if 127.0.0.1 is put under General Settings? TIA

It will work if the resolver works in recursive resolution mode where it does not depend on forwarders. It would also work in forwarding mode but then you need to make sure the forwarders in unbound.conf are set to the real forwarders and not 127.0.0.1. I'm not sure if checking the "Do not use the DNS Forwarder as a DNS server for the firewall" option also prevents the unbound resolver from using the forwarders set in System->General Setup.

hda · May 6, 2016, 9:33 AM

@Marc05:

Would you mind explaining how DNS Resolver would work if 127.0.0.1 is put under General Settings? TIA

For DNS Resolver/Unbound, you do not need to config-set any on [System > General Setup > DNS Server Settings]. All empty…

Azaron · May 6, 2016, 9:21 PM

@cmb:

Is your system clock really far off? Having a significantly wrong date and time on the system could be what's making DNSSEC fail.

No, I just verified this once again. The time within my pfsense is perfectly sync with atomic clocks.

@johnpoz:

Are you using forwarder mode or resolver mode in unbound? To be honest there is little reason to have anything in pfsense general setting for dns other than 127.0.0.1 so your asking the same dns service your clients are using..

You know the resolver in resolver mode has to be able to query anywhere via 53 both udp and tcp.. If your isp blocks some of this traffic than you will have all kinds of issues with resolver mode.. If your saying your having issues with dnssec I resolving stuff I would guess your isp is blocking or having issues or something between pfsense and internet is blocking or intercepting your dns traffic… Can you query the root servers directly?

I tried both, first using resolver mode and afterwards forwarding mode. Both variants to not work for my clients in resolving some of the dnsnames.
Honestly of course it could be my ISP blocking something, but shouldn't forwarding mode still work with unbound?
Shouldn't forwarding mode in unbound give the same result as the DNS Forwarder (dnsmasq)?
As said DNS Forwarder (dnsmasq) is working!
(and unbound was working in default configuration before upgrade as well btw.)

[edit]
just tested on the shell to dig the root servers and they answer with the expected resolves.
[/edit]

@hda:

@Marc05:

Would you mind explaining how DNS Resolver would work if 127.0.0.1 is put under General Settings? TIA

For DNS Resolver/Unbound, you do not need to config-set any on [System > General Setup > DNS Server Settings]. All empty…

This was new to me. Thanks for this Information.
However this shouldn't be the reason for my problem, as the pfsense box itself is using all 3 dns servers during the dnslookup and get the result for download.fedoraproject.org while my networkclients fail to resolv this. see attached jpg as well.

cmb · May 7, 2016, 3:17 AM

What are your clients using for their DNS? Your screenshot there shows Unbound's working fine, which is the same thing your clients would get in response.

The only other thing that might impact whether clients can resolve via Unbound is its access control. Your LAN interface subnets are automatically added as allowed, but maybe your LANs are misconfigured as WANs? Interfaces>LAN, must be no gateway set there.

You'll see what's allowed in /var/unbound/access_lists.conf

Azaron · May 7, 2016, 11:56 AM

@cmb:

What are your clients using for their DNS? Your screenshot there shows Unbound's working fine, which is the same thing your clients would get in response.

The only other thing that might impact whether clients can resolve via Unbound is its access control. Your LAN interface subnets are automatically added as allowed, but maybe your LANs are misconfigured as WANs? Interfaces>LAN, must be no gateway set there.

You'll see what's allowed in /var/unbound/access_lists.conf

cat /var/unbound/access_lists.conf
access-control: 127.0.0.1/32 allow
access-control: ::1 allow
access-control: 127.0.0.0/8 allow
access-control: 192.168.42.0/24 allow
access-control: 172.16.42.0/24 allow
access-control: 172.17.42.0/24 allow

My Clients have DHCP activated and get
default gateway = <ip of="" pfsense="">DHCP-Server = <ip of="" pfsense="">DNS-Server = <ip of="" pfsense="">As said in the original posting this all was working once before the update to 2.3 and the clients can resolve domains, primary domains but not all subdomains with "DNS Resolver (unbound)" (regardless if forwarding is active or not while "DNS Forwarder (dnsmasq)" ist working without problems for my pfsense AND my Clients .
I assume there is something really wrong in my unbound processing of subdomains like download.fedoraproject.org which gets resolved as CNAME wildcard.fedoraproject.org in

dig @a.root-servers.net download.fedoraproject.org

; <<>> DiG 9.10.3-P4 <<>> @a.root-servers.net download.fedoraproject.org
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54264
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;download.fedoraproject.org. IN A

;; ANSWER SECTION:
download.fedoraproject.org. 299 IN CNAME wildcard.fedoraproject.org.
wildcard.fedoraproject.org. 59 IN A 5.175.150.50
wildcard.fedoraproject.org. 59 IN A 152.19.134.198
wildcard.fedoraproject.org. 59 IN A 140.211.169.206
wildcard.fedoraproject.org. 59 IN A 209.132.181.15
wildcard.fedoraproject.org. 59 IN A 209.132.181.16
wildcard.fedoraproject.org. 59 IN A 67.219.144.68
wildcard.fedoraproject.org. 59 IN A 213.175.193.206
wildcard.fedoraproject.org. 59 IN A 140.211.169.196
wildcard.fedoraproject.org. 59 IN A 152.19.134.142

;; Query time: 71 msec
;; SERVER: 198.41.0.4#53(198.41.0.4)
;; WHEN: Sat May 07 13:56:48 CEST 2016
;; MSG SIZE rcvd: 222</ip></ip></ip>

Azaron · May 7, 2016, 12:02 PM

ohh forgot to answer your last question.
See my LAN interface config attached. There is no magic, a static IP and no gateway set.

johnpoz · May 7, 2016, 12:31 PM

" but not all subdomains with "DNS Resolver (unbound)"

What is your specific example of something that is not working.. And show this not working from the client doing a query to pfsense.. If your saying pfsense can resolve it, then your clients using pfsense would resolve it.. Unless your clients are not using pfsense for dns. Do they happen to have more than 1 dns set??

Possible reasons for stuff to fail in unbound while it works with a forwarder, is what your getting from the forwarder is OLD and cached, while currently the authoritative name server for what your looking for is either down or not reachable by you.

If you have dnssec enabled, its quite possible there is something wrong with the dnssec setup for this domain.

But without actual examples of what is not working, saying something doesn't resolve is never going to get to the root of the problem.. What doesn't resolve..

"download.fedoraproject.org while my networkclients fail to resolv"

Not sure where you doing queries too.. But that answer you show is not a ROOT SERVER answer… root servers do not do recursive... So if you ask a root server for something.. The only thing its going to give you back is name servers you need to go ask next..

This is a query to your root server..


> dig @a.root-servers.net download.fedoraproject.org

; <<>> DiG 9.10.4 <<>> @a.root-servers.net download.fedoraproject.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31491
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 13
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;download.fedoraproject.org.    IN      A

;; AUTHORITY SECTION:
org.                    172800  IN      NS      d0.org.afilias-nst.org.
org.                    172800  IN      NS      a0.org.afilias-nst.info.
org.                    172800  IN      NS      c0.org.afilias-nst.info.
org.                    172800  IN      NS      a2.org.afilias-nst.info.
org.                    172800  IN      NS      b0.org.afilias-nst.org.
org.                    172800  IN      NS      b2.org.afilias-nst.org.

;; ADDITIONAL SECTION:
d0.org.afilias-nst.org. 172800  IN      A       199.19.57.1
d0.org.afilias-nst.org. 172800  IN      AAAA    2001:500:f::1
a0.org.afilias-nst.info. 172800 IN      A       199.19.56.1
a0.org.afilias-nst.info. 172800 IN      AAAA    2001:500:e::1
c0.org.afilias-nst.info. 172800 IN      A       199.19.53.1
c0.org.afilias-nst.info. 172800 IN      AAAA    2001:500:b::1
a2.org.afilias-nst.info. 172800 IN      A       199.249.112.1
a2.org.afilias-nst.info. 172800 IN      AAAA    2001:500:40::1
b0.org.afilias-nst.org. 172800  IN      A       199.19.54.1
b0.org.afilias-nst.org. 172800  IN      AAAA    2001:500:c::1
b2.org.afilias-nst.org. 172800  IN      A       199.249.120.1
b2.org.afilias-nst.org. 172800  IN      AAAA    2001:500:48::1

;; Query time: 109 msec
;; SERVER: 198.41.0.4#53(198.41.0.4)
;; WHEN: Sat May 07 07:32:03 Central Daylight Time 2016
;; MSG SIZE  rcvd: 457

You getting back cname and the result of that cname tells me your not talking to the server you think your talking too.. Most likely your ISP is intercepting your dns queries..

Azaron · May 7, 2016, 12:58 PM

Ok, maybe you havent' understood everything all together as I haven't put all the infos into one single post. Tho I thought I had made clear what's happening…

Attached find what I mean with 3 simple screenshots.

1.) dnslookup within webinterface of pfsense while DNS Resolver is aktive ==> download.fedoraproject.org gets resolved = OK. /me happy
2.) dig to root servers from pfsense shell ==> working. /me happy
3.) nslookup from my Client-PC ==> not working. /me sad.

in addition, as other questions here asked for it 2 more screenshots showing my LAN-Interface config and the config of unbound itself.

Azaron · May 7, 2016, 1:13 PM

Just had to install dig on the windows client as well.
Now I can show you the full dig on pfsense as dns failing as well.


c:\Temp\BIND9.10.4.x64>nslookup download.fedoraproject.org
Server:         192.168.42.1
Address:        192.168.42.1#53

** server can't find download.fedoraproject.org: SERVFAIL

c:\Temp\BIND9.10.4.x64>dig @pfsense.grapes.home download.fedoraproject.org

; <<>> DiG 9.10.4 <<>> @pfsense.grapes.home download.fedoraproject.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 51175
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;download.fedoraproject.org.    IN      A

;; Query time: 1344 msec
;; SERVER: 192.168.42.1#53(192.168.42.1)
;; WHEN: Sat May 07 15:14:13 Mitteleuropõische Sommerzeit 2016
;; MSG SIZE  rcvd: 55

c:\Temp\BIND9.10.4.x64>

johnpoz · May 7, 2016, 1:29 PM

"2.) dig to root servers from pfsense shell ==> working. /me happy"

What you show as dig to root - is WRONG AND NOT what a root server would answer… Root servers DO NOT DO RECURSIVE QUERIES... They will never give you such a response - EVER!!!

As to your client getting SERVFAIL from doing a query to pfsense for download.fedoraproject.org lets see good query.. Prob due to fact your unbound is not able to do valid queries to the roots - since again what you show as a query to root is NOT a query to root...

See your attached query, and then an actual valid query from a root server..

ROOT servers will only every tell you the next NS server to go to - they would NEVER answer in such a way as what you posted.

Azaron · May 7, 2016, 2:11 PM

Ahh OK thanks for clarification.

I'm not expert in DNS when it comes to root Servers and stuff so thanks for pointing me to the error.

For me this is solved then, as if I understand you right, I can't fix it as it looks like my ISP has changed something right in the same time frame as I updated my pfsense leaving unbound in a condition no longer working as before.

As DNS Forwarder works for me I stick with that DNS Solution.

cmb · May 7, 2016, 10:59 PM

Yeah your ISP started intercepting DNS it appears. Enabling forwarding mode in Unbound will get you the same behavior as dnsmasq and avoid the root hijacking your ISP is apparently doing.

johnpoz · May 8, 2016, 11:29 AM

If your ok with your isp intercepting dns queries - sure use forwarder mode… I would be in a freaking uproar and on the phone with them, or finding a new isp..