Periodic since 2.2 pages load blank, certs invalid
-
Yup.
Unbound starts returning SERVFAIL for random domains after querying at least one or both of those hostnames. I'm doing it one more time at unbound log level 5.
This sounds like the manifestation of the issue with DNSSEC enabled. At some point, you may want to try it again with DNSSEC disabled; you should then see all domains being resolved to a hostile IP, bad certs for https, etc. like reported in the beginning of this thread. Not sure if this would help with the diagnostics.
Yes, DNSSEC was the change I made that made it go from the bad domain resolutions to the failure to resolve at all.
I've had both issues and DNSSEC was the difference between which one I got.
-
Uncheck DNSSEC, Save, Apply…. There is most certainly something here....
$ dig ns3.csof.net ; <<>> DiG 9.8.3-P1 <<>> ns3.csof.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1411 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 3 ;; QUESTION SECTION: ;ns3.csof.net. IN A ;; ANSWER SECTION: ns3.csof.net. 600 IN A 195.22.26.199 ;; AUTHORITY SECTION: csof.net. 1792 IN NS ns62.domaincontrol.com. csof.net. 1792 IN NS ns61.domaincontrol.com. csof.net. 1792 IN NS ns4.csof.net. csof.net. 1792 IN NS ns2.csof.net. csof.net. 1792 IN NS ns1.csof.net. ;; ADDITIONAL SECTION: ns4.csof.net. 1792 IN A 54.72.8.183 ns2.csof.net. 1793 IN A 212.6.183.201 ns1.csof.net. 1792 IN A 54.77.72.254 ;; Query time: 29 msec ;; SERVER: 192.168.223.1#53(192.168.223.1) ;; WHEN: Mon Feb 9 18:48:25 2015 ;; MSG SIZE rcvd: 203 $ dig api-nyc01.exip.org ; <<>> DiG 9.8.3-P1 <<>> api-nyc01.exip.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14360 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;api-nyc01.exip.org. IN A ;; ANSWER SECTION: api-nyc01.exip.org. 10 IN A 195.22.26.248 ;; Query time: 206 msec ;; SERVER: 192.168.223.1#53(192.168.223.1) ;; WHEN: Mon Feb 9 18:48:35 2015 ;; MSG SIZE rcvd: 52 $ dig www.pfsense.org ; <<>> DiG 9.8.3-P1 <<>> www.pfsense.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51593 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.pfsense.org. IN A ;; ANSWER SECTION: www.pfsense.org. 10 IN A 195.22.26.248 ;; AUTHORITY SECTION: org. 172779 IN NS ns1.csof.net. org. 172779 IN NS ns2.csof.net. org. 172779 IN NS ns3.csof.net. org. 172779 IN NS ns4.csof.net. ;; Query time: 159 msec ;; SERVER: 192.168.223.1#53(192.168.223.1) ;; WHEN: Mon Feb 9 18:48:39 2015 ;; MSG SIZE rcvd: 129 -
looks like unbound is allowing itself to get polluted by this nonsense:
gridbug:etc cjl$ dig @ns1.csof.net. com. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. com. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31834 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;com. IN NS ;; ANSWER SECTION: com. 172800 IN NS ns1.csof.net. com. 172800 IN NS ns2.csof.net. com. 172800 IN NS ns3.csof.net. com. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 163 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:58:46 2015 ;; MSG SIZE rcvd: 165 gridbug:etc cjl$ dig @ns1.csof.net. . ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. . ns ; (1 server found) ;; global options: +cmd ;; connection timed out; no servers could be reached gridbug:etc cjl$ dig @ns1.csof.net. net. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. net. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23620 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;net. IN NS ;; ANSWER SECTION: net. 172800 IN NS ns1.csof.net. net. 172800 IN NS ns2.csof.net. net. 172800 IN NS ns3.csof.net. net. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 161 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:59:19 2015 ;; MSG SIZE rcvd: 162 gridbug:etc cjl$ dig @ns1.csof.net. com. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. com. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57675 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;com. IN NS ;; ANSWER SECTION: com. 172800 IN NS ns1.csof.net. com. 172800 IN NS ns2.csof.net. com. 172800 IN NS ns3.csof.net. com. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 164 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:59:23 2015 ;; MSG SIZE rcvd: 165 gridbug:etc cjl$ dig @ns1.csof.net. org. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. org. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20295 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;org. IN NS ;; ANSWER SECTION: org. 172800 IN NS ns1.csof.net. org. 172800 IN NS ns2.csof.net. org. 172800 IN NS ns3.csof.net. org. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 162 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:59:27 2015 ;; MSG SIZE rcvd: 165 gridbug:etc cjl$ dig @ns1.csof.net. edu. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. edu. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28770 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;edu. IN NS ;; ANSWER SECTION: edu. 172800 IN NS ns1.csof.net. edu. 172800 IN NS ns2.csof.net. edu. 172800 IN NS ns3.csof.net. edu. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 161 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:59:31 2015 ;; MSG SIZE rcvd: 165 gridbug:etc cjl$ dig @ns1.csof.net. gov. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. gov. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46880 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;gov. IN NS ;; ANSWER SECTION: gov. 172800 IN NS ns1.csof.net. gov. 172800 IN NS ns2.csof.net. gov. 172800 IN NS ns3.csof.net. gov. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 162 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:59:34 2015 ;; MSG SIZE rcvd: 165 gridbug:etc cjl$ dig @ns1.csof.net. mil. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. mil. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21791 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;mil. IN NS ;; ANSWER SECTION: mil. 172800 IN NS ns1.csof.net. mil. 172800 IN NS ns2.csof.net. mil. 172800 IN NS ns3.csof.net. mil. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 160 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:59:40 2015 ;; MSG SIZE rcvd: 165 gridbug:etc cjl$ dig @ns1.csof.net. us. ns ; <<>> DiG 9.8.3-P1 <<>> @ns1.csof.net. us. ns ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5930 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;us. IN NS ;; ANSWER SECTION: us. 172800 IN NS ns1.csof.net. us. 172800 IN NS ns2.csof.net. us. 172800 IN NS ns3.csof.net. us. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 161 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Mon Feb 9 18:59:43 2015 ;; MSG SIZE rcvd: 164 -
Back to dnsmasq for me until someone says something.
-
I don't know which I should be.
Upset that unbound is doing this
Relieved that everything I was seeing can be reproduced -
You should be thankful you found something that made it easily-reproducible. No need to be mad. Things happen - Look at BIND's history. And you can always just run dnsmasq in the meantime even though that's the exact opposite advice you got at first. :/
-
I'm really out of my league trying to diagnose this, but here is the output of 'unbound-control -c /var/unbound/unbound.conf lookup slashdot.org', when the issue is occuring with DNSSEC enabled:
unbound-control -c /var/unbound/unbound.conf lookup slashdot.org The following name servers are used for lookup of slashdot.org. ;rrset 85785 4 0 7 0 org. 172185 IN NS ns1.csof.net. org. 172185 IN NS ns2.csof.net. org. 172185 IN NS ns3.csof.net. org. 172185 IN NS ns4.csof.net. ;rrset 83879 2 1 11 4 org. 83879 IN DS 21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2 org. 83879 IN DS 21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0D90F01BA org. 83879 IN RRSIG DS 8 1 86400 20150219170000 20150209160000 16665 . RxYpI0BzYpGTE/PjRQdR4SZaxlvXCja3SJyx10JagTfz20gnltl4ar94GOwp8bA/ktY/7JxMoJvzCTAtcsGaTGRv04yDHr7WaydMxZuPCP9YT9Ixc+fX9IAZlSfwLCkBQgiC0mVeRiq+LmbIJhI2grJbTtvy96O9mipAqkFR42g= ;{id = 16665} ;rrset 1185 1 0 3 0 ns4.csof.net. 1185 IN A 54.72.8.183 ;rrset 197 1 0 8 0 ns3.csof.net. 197 IN A 195.22.26.199 ;rrset 1185 1 0 8 0 ns2.csof.net. 1185 IN A 212.6.183.201 ;rrset 1185 1 0 8 0 ns1.csof.net. 1185 IN A 54.77.72.254 Delegation with 4 names, of which 4 can be examined to query further addresses. It provides 4 IP addresses. 54.77.72.254 NoDNSSEC rto 344 msec, ttl 286, ping 84 var 65 rtt 344, tA 0, tAAAA 0, tother 0, EDNS 0 probed. 212.6.183.201 NoDNSSEC rto 365 msec, ttl 286, ping 69 var 74 rtt 365, tA 0, tAAAA 0, tother 0, EDNS 0 probed. 195.22.26.199 rto 96256 msec, ttl 286, ping 0 var 94 rtt 376, tA 3, tAAAA 3, tother 3, EDNS 0 assumed. 54.72.8.183 NoDNSSEC rto 315 msec, ttl 286, ping 99 var 54 rtt 315, tA 0, tAAAA 0, tother 0, EDNS 0 probed.And then after restarting unbound:
unbound-control -c /var/unbound/unbound.conf lookup slashdot.org The following name servers are used for lookup of slashdot.org. ;rrset 86374 6 0 2 0 org. 86374 IN NS a0.org.afilias-nst.info. org. 86374 IN NS a2.org.afilias-nst.info. org. 86374 IN NS b0.org.afilias-nst.org. org. 86374 IN NS b2.org.afilias-nst.org. org. 86374 IN NS c0.org.afilias-nst.info. org. 86374 IN NS d0.org.afilias-nst.org. ;rrset 86374 2 1 2 0 org. 86374 IN DS 21366 7 1 E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2 org. 86374 IN DS 21366 7 2 96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0D90F01BA org. 86374 IN RRSIG DS 8 1 86400 20150219170000 20150209160000 16665 . RxYpI0BzYpGTE/PjRQdR4SZaxlvXCja3SJyx10JagTfz20gnltl4ar94GOwp8bA/ktY/7JxMoJvzCTAtcsGaTGRv04yDHr7WaydMxZuPCP9YT9Ixc+fX9IAZlSfwLCkBQgiC0mVeRiq+LmbIJhI2grJbTtvy96O9mipAqkFR42g= ;{id = 16665} ;rrset 86374 1 0 1 0 d0.org.afilias-nst.org. 172774 IN A 199.19.57.1 ;rrset 86374 1 0 1 0 d0.org.afilias-nst.org. 172774 IN AAAA 2001:500:f::1 ;rrset 86374 1 0 1 0 c0.org.afilias-nst.info. 172774 IN A 199.19.53.1 ;rrset 86374 1 0 1 0 c0.org.afilias-nst.info. 172774 IN AAAA 2001:500:b::1 ;rrset 86374 1 0 1 0 b2.org.afilias-nst.org. 172774 IN A 199.249.120.1 ;rrset 86374 1 0 1 0 b2.org.afilias-nst.org. 172774 IN AAAA 2001:500:48::1 ;rrset 86374 1 0 1 0 b0.org.afilias-nst.org. 172774 IN A 199.19.54.1 ;rrset 86374 1 0 1 0 b0.org.afilias-nst.org. 172774 IN AAAA 2001:500:c::1 ;rrset 86374 1 0 1 0 a2.org.afilias-nst.info. 172774 IN A 199.249.112.1 ;rrset 86374 1 0 1 0 a2.org.afilias-nst.info. 172774 IN AAAA 2001:500:40::1 ;rrset 86374 1 0 1 0 a0.org.afilias-nst.info. 172774 IN A 199.19.56.1 ;rrset 86374 1 0 1 0 a0.org.afilias-nst.info. 172774 IN AAAA 2001:500:e::1 Delegation with 6 names, of which 0 can be examined to query further addresses. It provides 12 IP addresses. 2001:500:e::1 not in infra cache. 199.19.56.1 not in infra cache. 2001:500:40::1 not in infra cache. 199.249.112.1 not in infra cache. 2001:500:c::1 not in infra cache. 199.19.54.1 not in infra cache. 2001:500:48::1 not in infra cache. 199.249.120.1 rto 356 msec, ttl 874, ping 8 var 87 rtt 356, tA 0, tAAAA 0, tother 0, EDNS 0 probed. 2001:500:b::1 not in infra cache. 199.19.53.1 rto 482 msec, ttl 874, ping 22 var 115 rtt 482, tA 0, tAAAA 0, tother 0, EDNS 0 probed. 2001:500:f::1 not in infra cache. 199.19.57.1 not in infra cache. -
I'll ask again…
In Services: DNS Resolver: Advanced
Have you tried checking:
Harden Glue
Harden DNSSEC data
Unwanted Reply Threshold (10 million)
etc... ???
-
Have you tried checking:
Harden Glue
Harden DNSSEC data
These two in particular, if you don't have them enabled, enable them. I changed things to enable both those by default, and we'll add config upgrade code to turn those on for anyone who doesn't have them enabled upon upgrade to 2.2.1.
-
Yeah - I went on a voyage of discovery to find a combo that worked…
Less a matter of being smart. More a matter of random experimentation.Even reading and searching online yielded little results.
But yeah - Fixes alot.
I'm glad you guys are making it defaults.
-
Couple more of the hardened stuff here: https://forum.pfsense.org/index.php?topic=88466.msg488511#msg488511
- harden-referral-path is hardcoded to no in unbound.inc ATM :(
- harden-below-nxdomain can be set via the advanced config
(created a feature request for the two above)
- there's also this use-caps-for-id thing (patch to expose in GUI in 4205 or stick use-caps-for-id: yes to advanced config.
Another thing that makes me wonder… dnsmasq AFAICT is not compiled with DNSSEC support at all on pfSense even though it's apparently supported now. Hmmm. And of course, no matter what you are completely screwed with unsigned domains, and the DNS protocol is a piece of insecure crap.
A couple of suggestions for your own servers:
- sign your domains (ahem… why's pfsense.org not signed?!)
- if your registrar does not allow you to do so, switch registrar
- if the TLD you are using is not signed, choose a different one
- use DANE/TLSA for your critical servers at least.
Some useful tools:
-
DNSSEC Test - check your resolvers
-
DNSSEC Analyzer - check your domain for DNSSEC problems
-
Update for our situation (only dns active, harden glue settings not (yet) updated).
The situation happened again a few minutes ago, and here it is how it looked in the resolver.log (first time "csof" is visible today) :Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: response for 239.75.246.46.in-addr.arpa. PTR IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <75.246.46.in-addr.arpa.> 91.213.246.6#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns1.transitionalprotocol.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns2.transitionalprotocol.net. A IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving f.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving g.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving k.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving l.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns2.transitionalprotocol.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving h.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving i.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns1.transitionalprotocol.net. A IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving m.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving d.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: response for a.ns.portlane.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <portlane.net.>80.67.0.6#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: response for b.ns.portlane.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <portlane.net.>80.67.0.6#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: response for 125.10.113.188.in-addr.arpa. PTR IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <10.113.188.in-addr.arpa.> 80.240.240.2#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: resolving 113.188.in-addr.arpa. DS IN Feb 10 10:03:18 pf unbound: [24807:1] notice: sendto failed: Operation not permitted Feb 10 10:03:18 pf unbound: [24807:1] notice: remote address is 2001:500:13::c7d4:35 port 53 Feb 10 10:03:18 pf unbound: [24807:1] info: error sending query to auth server 2001:500:13::c7d4:35 port 53 Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.>192.52.178.30#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns2.csof.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving ns1.csof.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving e.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: resolving d.gtld-servers.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. A IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.>192.52.178.30#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:18 pf unbound: [24807:0] info: response for 125.74.1.116.in-addr.arpa. PTR IN Feb 10 10:03:18 pf unbound: [24807:0] info: reply from <1.116.in-addr.arpa.> 202.103.224.69#53 Feb 10 10:03:18 pf unbound: [24807:0] info: query response was NXDOMAIN ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.>192.26.92.30#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. A IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.>212.6.183.201#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. A IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.>192.31.80.30#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.csof.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.>192.43.172.30#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.transitionalprotocol.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.>212.6.183.201#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was nodata ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: response for 113.188.in-addr.arpa. DS IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <188.in-addr.arpa.> 199.212.0.53#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was nodata ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved not a delegation point Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved no DS. Feb 10 10:03:18 pf unbound: [24807:1] info: Verified that unsigned response is INSECURE Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. A IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.>54.77.72.254#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns1.csof.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <net.>192.26.92.30#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:18 pf unbound: [24807:1] info: response for 239.75.246.46.in-addr.arpa. PTR IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <239.75.246.46.in-addr.arpa.> 195.22.26.248#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:18 pf unbound: [24807:1] info: resolving 246.46.in-addr.arpa. DS IN Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved not a delegation point Feb 10 10:03:18 pf unbound: [24807:1] info: NSEC RRset for the referral proved no DS. Feb 10 10:03:18 pf unbound: [24807:1] info: Verified that unsigned response is INSECURE Feb 10 10:03:18 pf unbound: [24807:1] info: response for ns2.csof.net. AAAA IN Feb 10 10:03:18 pf unbound: [24807:1] info: reply from <csof.net.>208.109.255.32#53 Feb 10 10:03:18 pf unbound: [24807:1] info: query response was nodata ANSWER Feb 10 10:03:18 pf unbound: [24807:0] info: response for 166.31.215.188.in-addr.arpa. PTR IN Feb 10 10:03:18 pf unbound: [24807:0] info: reply from <31.215.188.in-addr.arpa.> 89.39.166.2#53 Feb 10 10:03:18 pf unbound: [24807:0] info: query response was THROWAWAY Feb 10 10:03:19 pf unbound: [24807:1] info: response for ns1.csof.net. AAAA IN Feb 10 10:03:19 pf unbound: [24807:1] info: reply from <csof.net.>208.109.255.32#53 Feb 10 10:03:19 pf unbound: [24807:1] info: query response was nodata ANSWER Feb 10 10:03:19 pf unbound: [24807:1] info: resolving 205.112.194.173.in-addr.arpa. PTR IN Feb 10 10:03:19 pf unbound: [24807:1] info: resolving ns2.google.com. AAAA IN Feb 10 10:03:19 pf unbound: [24807:1] info: resolving ns4.google.com. AAAA IN Feb 10 10:03:19 pf unbound: [24807:1] info: resolving ns1.google.com. AAAA IN Feb 10 10:03:19 pf unbound: [24807:1] info: response for ns1.transitionalprotocol.net. AAAA IN Feb 10 10:03:19 pf unbound: [24807:1] info: reply from <transitionalprotocol.net.>212.6.183.201#53 Feb 10 10:03:19 pf unbound: [24807:1] info: query response was nodata ANSWER Feb 10 10:03:19 pf unbound: [24807:1] info: response for 205.112.194.173.in-addr.arpa. PTR IN Feb 10 10:03:19 pf unbound: [24807:1] info: reply from <194.173.in-addr.arpa.> 216.239.36.10#53 Feb 10 10:03:19 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:20 pf unbound: [24807:1] info: resolving www.insign.ch. A IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving dns2.insign.ch. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving dns1.insign.ch. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: response for www.insign.ch. A IN Feb 10 10:03:20 pf unbound: [24807:1] info: reply from <insign.ch.>46.175.9.120#53 Feb 10 10:03:20 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:20 pf unbound: [24807:1] info: NSEC3s for the referral proved no DS. Feb 10 10:03:20 pf unbound: [24807:1] info: Verified that unsigned response is INSECURE Feb 10 10:03:20 pf unbound: [24807:1] info: resolving 64.94.172.95.in-addr.arpa. PTR IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns-b.pnap.net. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns-c.pnap.net. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns-a.pnap.net. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving 185.112.194.173.in-addr.arpa. PTR IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns3.google.com. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns4.google.com. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: resolving ns1.google.com. AAAA IN Feb 10 10:03:20 pf unbound: [24807:1] info: response for 185.112.194.173.in-addr.arpa. PTR IN Feb 10 10:03:20 pf unbound: [24807:1] info: reply from <194.173.in-addr.arpa.> 216.239.32.10#53 Feb 10 10:03:20 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:20 pf unbound: [24807:1] info: response for 64.94.172.95.in-addr.arpa. PTR IN Feb 10 10:03:20 pf unbound: [24807:1] info: reply from <94.172.95.in-addr.arpa.> 64.95.61.4#53 Feb 10 10:03:41 pf unbound: [24807:0] info: query response was REFERRAL Feb 10 10:03:41 pf unbound: [24807:0] info: resolving dns4.bigrock.in. AAAA IN Feb 10 10:03:41 pf unbound: [24807:0] info: resolving dns2.bigrock.in. AAAA IN Feb 10 10:03:41 pf unbound: [24807:0] info: resolving asia1.akam.net. AAAA IN Feb 10 10:03:41 pf unbound: [24807:0] notice: sendto failed: Operation not permitted Feb 10 10:03:41 pf unbound: [24807:0] notice: remote address is 2600:1401:2::43 port 53 Feb 10 10:03:41 pf unbound: [24807:0] info: error sending query to auth server 2600:1401:2::43 port 53 Feb 10 10:03:41 pf unbound: [24807:1] info: response for ns0.mirasystem.net. A IN Feb 10 10:03:41 pf unbound: [24807:1] info: reply from <net.>54.72.8.183#53 Feb 10 10:03:41 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:42 pf unbound: [24807:0] info: response for asia1.akam.net. AAAA IN Feb 10 10:03:42 pf unbound: [24807:0] info: reply from <akam.net.>184.85.248.67#53 Feb 10 10:03:42 pf unbound: [24807:0] info: query response was nodata ANSWER Feb 10 10:03:42 pf unbound: [24807:0] info: response for dns2.bigrock.in. AAAA IN Feb 10 10:03:42 pf unbound: [24807:0] info: reply from <bigrock.in.>2.22.230.64#53 Feb 10 10:03:42 pf unbound: [24807:0] info: query response was nodata ANSWER Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.epn.ru. A IN Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <epn.ru.>195.22.26.248#53 Feb 10 10:03:42 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns0.epn.ru. AAAA IN Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <epn.ru.>195.22.26.248#53 Feb 10 10:03:42 pf unbound: [24807:1] info: query response was nodata ANSWER Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns0.epn.ru. A IN Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <epn.ru.>195.22.26.248#53 Feb 10 10:03:42 pf unbound: [24807:1] info: query response was ANSWER Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.epn.ru. AAAA IN Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <epn.ru.>195.22.26.248#53 Feb 10 10:03:42 pf unbound: [24807:1] info: query response was nodata ANSWER Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.hn.cnc.cn. AAAA IN Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <cnc.cn.>210.52.207.2#53 Feb 10 10:03:42 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns1.hn.cnc.cn. A IN Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <cnc.cn.>210.52.207.2#53 Feb 10 10:03:42 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns2.hn.cnc.cn. A IN Feb 10 10:03:42 pf unbound: [24807:1] info: reply from <cnc.cn.>210.52.207.2#53 Feb 10 10:03:42 pf unbound: [24807:1] info: query response was REFERRAL Feb 10 10:03:42 pf unbound: [24807:1] info: response for ns2.hn.cnc.cn. AAAA IN</cnc.cn.></cnc.cn.></cnc.cn.></epn.ru.></epn.ru.></epn.ru.></epn.ru.></bigrock.in.></akam.net.></net.></insign.ch.></transitionalprotocol.net.></csof.net.></csof.net.></net.></transitionalprotocol.net.></transitionalprotocol.net.></net.></net.></transitionalprotocol.net.></net.></net.></net.></portlane.net.></portlane.net.>At the end, all requests get resolved as "195.22.26.248".
Trigger request couldn't yet be found (the log rule on port 53 for requests not going over the router returned nothing at this moment).So when it starts, every single request then go over one of these ns1.csof.net name server, and get answered as if csof.net was a root server, but with wrong data. For example:
Normal case when querying the pfsense resolver (when everything is ok) :
om@ompc:~> dig -t ns @192.168.1.100 ch. ; <<>> DiG 9.9.5-rpz2+rl.14038.05-P1 <<>> -t ns @192.168.1.100 ch. ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62805 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 14 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;ch. IN NS ;; ANSWER SECTION: ch. 85292 IN NS c.nic.ch. ch. 85292 IN NS b.nic.ch. ch. 85292 IN NS e.nic.ch. ch. 85292 IN NS d.nic.ch. ch. 85292 IN NS a.nic.ch. ch. 85292 IN NS h.nic.ch. ch. 85292 IN NS f.nic.ch. ;; ADDITIONAL SECTION: a.nic.ch. 171692 IN A 130.59.1.80 a.nic.ch. 171692 IN AAAA 2001:620::4 b.nic.ch. 171692 IN A 130.59.211.10 b.nic.ch. 171692 IN AAAA 2001:620::5 c.nic.ch. 171692 IN A 147.28.0.39 c.nic.ch. 171692 IN AAAA 2001:418:1::39 d.nic.ch. 171692 IN A 200.160.0.5 d.nic.ch. 171692 IN AAAA 2001:12ff:0:a20::5 e.nic.ch. 171692 IN A 194.0.17.1 e.nic.ch. 171692 IN AAAA 2001:678:3::1 f.nic.ch. 171692 IN A 194.146.106.10 f.nic.ch. 171692 IN AAAA 2001:67c:1010:2::53 h.nic.ch. 171692 IN A 194.42.48.120 ;; Query time: 0 msec ;; SERVER: 192.168.1.100#53(192.168.1.100) ;; WHEN: Tue Feb 10 10:55:52 CET 2015 ;; MSG SIZE rcvd: 427When querying this ns1.csof.net server :
om@ompc:~> dig -t ns @ns1.csof.net ch. ; <<>> DiG 9.9.5-rpz2+rl.14038.05-P1 <<>> -t ns @ns1.csof.net ch. ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45232 ;; flags: qr aa rd; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;ch. IN NS ;; ANSWER SECTION: ch. 172800 IN NS ns1.csof.net. ch. 172800 IN NS ns2.csof.net. ch. 172800 IN NS ns3.csof.net. ch. 172800 IN NS ns4.csof.net. ;; ADDITIONAL SECTION: ns1.csof.net. 100 IN A 54.77.72.254 ns2.csof.net. 100 IN A 212.6.183.201 ns3.csof.net. 100 IN A 195.22.26.199 ns4.csof.net. 100 IN A 54.72.8.183 ;; Query time: 54 msec ;; SERVER: 54.77.72.254#53(54.77.72.254) ;; WHEN: Tue Feb 10 10:45:26 CET 2015 ;; MSG SIZE rcvd: 164I now increased the verbosity of unbound with "unbound-control -c /var/unbound/unbound.conf verbosity 4" and waiting for another hit… If you have other suggestions in the mean time, please feel free.
-
Does that mean you have already made all the suggested changes?
-
Does that mean you have already made all the suggested changes?
Apparently not.
Update for our situation (only dns active, harden glue settings not (yet) updated).
People, by all means feel free to experiment if its your network. However, mind that if there are any unsuspecting users behind your router using your resolver, this is not the best idea around. Noone likes getting their boxes infected via malicious pages served via the subverted DNS, getting their banking info compromised etc.!
-
Does that mean you have already made all the suggested changes?
(if the question is for me) : no, as we would first would like to find out what/who is triggering this phenomena. But we'll change the setup then (today before the evening) and report again.
-
Yeah - Please do find out. Its a mystery for me.
-
Activating following settings (as suggested previously in this thread) seems to solve the issue for now, at least for the last 3 hours :
harden-glue: yes harden-dnssec-stripped: yesIn the mean time I have a 500 MB resolver.log in "verbose=5" mode to analyze.
-
With the great support of Wouter Wijngaards from the Unbound-Dev-Team this issue can be considered as closed, at least on our side.
Activating "harden-glue: yes" and "harden-dnssec-stripped: yes" was the solution, or a patch can be applied to fix this situation without enabling these options.@Wouter:
Hi Martin, Olivier,
I have found the issue, it is in unbound. You have turned off
harden-glue in config and this disabled the poison checks (that is its
purpose) and caused the poisoning. The domain is not an attack domain
I think but a domain-reseller.Turn on harden-glue: yes in unbound.conf
Also this patch may solve the problem (and you can continue to use
harden-glue: no). Note that harden-glue no allows people (not this
specific attack) to poison nameserver glue records and then poison
your cache as well, also with the patch.Index: iterator/iter_scrub.c
–- iterator/iter_scrub.c (revision 3329)
+++ iterator/iter_scrub.c (working copy)
@@ -680,7 +680,9 @@
* (we dont want its glue that was approved
* during the normalize action) /
del_addi = 1;
- } else if(!env->cfg->harden_glue) {
+ } else if(!env->cfg->harden_glue && (
+ rrset->type == LDNS_RR_TYPE_A ||
+ rrset->type == LDNS_RR_TYPE_AAAA)) {
/ store in cache! Since it is relevant
* (from normalize) it will be picked up
* from the cache to be used later */Why was harden-glue turned off? And perhaps I should change
documentation or implementation of this misfeature?Best regards,
WouterMore information from the Unbound-Users mailing list : https://unbound.nlnetlabs.nl/pipermail/unbound-users/2015-February/003768.html
Suggestion for pfSense 2.2.1 : apply this patch to unbound and/or activate the harden-options by default. I'll check later if there is anything in redmine about this issue or will create a new one accordingly.
Thanks again for your fast and helpful answers on this forum & I hope it will remain stable this way ! Now I can get back to my IPSEC issues :-)
Kind regards.PS: about the trigger :
> (...) could you see which host/device caused this issue ? Yes, the offending query was for ns2.transitionalprotocol.net A. The logs are missing lines (the logger seems to skip lines when presented at a high rate). That query was made to resolve another query for 105.73.246.46.in-addr.arpa. PTR. And this was requested by Feb 10 11:48:15 pf unbound: [68609:0] debug: udp request from ip4 192.168.1.150 port 62403 (len 16) The query for that reverse address seems innocent. -
@Wouter:
Why was harden-glue turned off?
Indeed, this is a good question, keeping in mind that unbound is enabled by default in all new 2.2 installations, I wonder how many unsuspecting people are being affected by this.
I feel that it warrants an official announcement by pfSense team with specific suggestions on how to remedy the issue… -
I'd say applying that patch AND enabling harden glue by default would probably be the safest option considering the havoc this was able to cause.
In the mean time, even with harden glue on, I'm going to keep my 0.0.0.0 overrides ;)
