Periodic since 2.2 pages load blank, certs invalid

kejianshi

Yeah - Please do find out. Its a mystery for me.

swix

Activating following settings (as suggested previously in this thread) seems to solve the issue for now, at least for the last 3 hours :


harden-glue: yes
harden-dnssec-stripped: yes

In the mean time I have a 500 MB resolver.log in "verbose=5" mode to analyze.

swix

With the great support of Wouter Wijngaards from the Unbound-Dev-Team this issue can be considered as closed, at least on our side.
Activating "harden-glue: yes" and "harden-dnssec-stripped: yes" was the solution, or a patch can be applied to fix this situation without enabling these options.

@Wouter:

Hi Martin, Olivier,

I have found the issue, it is in unbound. You have turned off
harden-glue in config and this disabled the poison checks (that is its
purpose) and caused the poisoning. The domain is not an attack domain
I think but a domain-reseller.

Turn on harden-glue: yes in unbound.conf

Also this patch may solve the problem (and you can continue to use
harden-glue: no). Note that harden-glue no allows people (not this
specific attack) to poison nameserver glue records and then poison
your cache as well, also with the patch.

Index: iterator/iter_scrub.c

–- iterator/iter_scrub.c (revision 3329)
+++ iterator/iter_scrub.c (working copy)
@@ -680,7 +680,9 @@
* (we dont want its glue that was approved
* during the normalize action) /
del_addi = 1;
- } else if(!env->cfg->harden_glue) {
+ } else if(!env->cfg->harden_glue && (
+ rrset->type == LDNS_RR_TYPE_A ||
+ rrset->type == LDNS_RR_TYPE_AAAA)) {
/ store in cache! Since it is relevant
* (from normalize) it will be picked up
* from the cache to be used later */

Why was harden-glue turned off? And perhaps I should change
documentation or implementation of this misfeature?

Best regards,
Wouter

More information from the Unbound-Users mailing list : https://unbound.nlnetlabs.nl/pipermail/unbound-users/2015-February/003768.html

Suggestion for pfSense 2.2.1 : apply this patch to unbound and/or activate the harden-options by default. I'll check later if there is anything in redmine about this issue or will create a new one accordingly.

Thanks again for your fast and helpful answers on this forum & I hope it will remain stable this way ! Now I can get back to my IPSEC issues :-)
Kind regards.

PS: about the trigger :


> (...) could you see which host/device caused this issue ?

Yes, the offending query was for ns2.transitionalprotocol.net A.
The logs are missing lines (the logger seems to skip lines when
presented at a high rate).  That query was made to resolve another
query for 105.73.246.46.in-addr.arpa. PTR.
And this was requested by
Feb 10 11:48:15 pf unbound: [68609:0] debug: udp request from ip4
192.168.1.150 port 62403 (len 16)

The query for that reverse address seems innocent.

dgcom

@swix:

@Wouter:

Why was harden-glue turned off?

Indeed, this is a good question, keeping in mind that unbound is enabled by default in all new 2.2 installations, I wonder how many unsuspecting people are being affected by this.
I feel that it warrants an official announcement by pfSense team with specific suggestions on how to remedy the issue…

Trel

I'd say applying that patch AND enabling harden glue by default would probably be the safest option considering the havoc this was able to cause.

In the mean time, even with harden glue on, I'm going to keep my 0.0.0.0 overrides ;)

swix

@swix:

Suggestion for pfSense 2.2.1 : apply this patch to unbound and/or activate the harden-options by default. I'll check later if there is anything in redmine about this issue or will create a new one accordingly.

Voilà : https://redmine.pfsense.org/issues/4402

doktornotor

harden-glue is already on by default in RELENG_2_2:

https://redmine.pfsense.org/projects/pfsense/repository/revisions/ef120e878558f84ed14369a484c0938ccd1b6db5

(The iter_scrub.c patch still useful though…)

0x10C

I just started getting this problem today for the first time since I installed the 2.2-RELEASE.

I was also using the 8.8.8.8 and 8.8.4.4 DNS from Google. I've changed to using OpenDNS now which seems to have stopped the problem?

The thread is really long I read the first few pages, should I activate the Harden Glue feature? - Just a worried noobie.

doktornotor

Harden Glue - yes, that definitely should be enabled. Otherwise, OpenDNS does not support DNSSEC at all, so all those other DNSSEC-related hardening features are kinda useless till you switch back to Google, or just disable the forwarding altogether.

0x10C

Ok so I've turned Hardened Glue on, what else should I enable?

heper

@cmb:

@kejianshi:

Have you tried checking:

Harden Glue

Harden DNSSEC data

These two in particular, if you don't have them enabled, enable them. I changed things to enable both those by default, and we'll add config upgrade code to turn those on for anyone who doesn't have them enabled upon upgrade to 2.2.1.

0x10C

@heper:

@cmb:

@kejianshi:

Have you tried checking:

Harden Glue

Harden DNSSEC data

These two in particular, if you don't have them enabled, enable them. I changed things to enable both those by default, and we'll add config upgrade code to turn those on for anyone who doesn't have them enabled upon upgrade to 2.2.1.

Okay I've enabled both of those. They will work okay with OpenDNS? - Thanks :)

doktornotor

@0x10C:

They will work okay with OpenDNS? - Thanks :)

Goto…

0x10C

@doktornotor:

@0x10C:

They will work okay with OpenDNS? - Thanks :)

Goto…

I read that but I just wanted to confirm because I wasn't sure if you mistyped what you said due to the way it was worded.

cmb

@0x10C:

Okay I've enabled both of those. They will work okay with OpenDNS?

Yes

kejianshi

Just to be clear, with harden glue I see no issue, but I'm not sure that opendns supports DNSSEC, so if forwarding from opendns with DNSSEC enabled, might get "nothing".

I'm not running opendns here, but seems like it didn't support DNSSEC in the past. Not sure about now.

(For me at least, trying to use DNSSEC with non-DNSSEC capable servers results in DNS failure to resolve)

doktornotor

Obviously, as already stated multiple times here, hardening features that make use of DNSSEC will do absolutely nothing useful when you are forwarding DNS queries to OpenDNS or any other open DNS server that does not support DNSSEC at all. If you want DNSSEC, stop using OpenDNS.

swix

Final followup here for me (I hope): Unbound 1.5.2rc1 has just been released, http://www.unbound.net/pipermail/unbound-users/2015-February/003774.html

Interesting part of the release notes in our case:

This release fixes a DNSSEC validation issue when an upstream server
with different trust anchors introduces unsigned records in messages.
Harden-glue when turned off allows potentially poisonous records in
the cache in the hopes of that enabling DNS resolution for 'impossible
to resolve' domains, it is fixed to have 'less cache poisoning',
quotes added because it is by definition not secure to turn off
harden-glue. New features are that "inform" can be used to see which
IPs lookup a domain, and unbound-control can use named unix pipes.

According to Chris in Redmine, this should be fixed in 2.2.1.