Periodic since 2.2 pages load blank, certs invalid

Derelict

Ok. So what are the DNS servers configured on the client?

swix

Ok. So what are the DNS servers configured on the client?

Set via DHCP, simply the router's ip address:


   option domain-name-servers 192.168.1.100;

Derelict

And you've actually verified that's the case on the client?

doktornotor

@swix:

Thanks for your update Trel, we're still searching here, but enabling DNSSEC does not stop the issue.

It does NOT stop the issue on domains that are not signed, no. Also, it will NOT prevent the DNS hijack if your clients are NOT using pfSense or another DNSSEC-enabled resolver, even if the zones are signed. It will prevent resolving domains to malicious crap for the rest.

Block/redirect all DNS queries on LAN to pfSense
Find and reimage infected crap.

Trel

@doktornotor:

Find and reimage infected crap.

I agree, but I would like to point out the way it affects pfsense/unbound is not a good thing at all.

A lookup on a completely isolated network segment made unbound start giving bad resolutions to ALL network segments when other DNS servers were permitted, and when they were blocked, unbound simply stopped replying.

That's not really the best outcome for a single computer looking up a bad domain.

swix

@Derelict:

And you've actually verified that's the case on the client?

Yes, + also did tests with dig @192.168.1.100 and directly via the pfsense shell. "poisoned" ip in every case (after a few seconds after the beginning of a new occurence of the issue).

@doktornotor:

It does NOT stop the issue on domains that are not signed, no. Also, it will NOT prevent the DNS hijack if your clients are NOT using pfSense or another DNSSEC-enabled resolver, even if the zones are signed. It will prevent resolving domains to malicious crap for the rest.

Yep, I supposed that too, but sometimes there are collateral effets to such settings.

@doktornotor:

Block/redirect all DNS queries on LAN to pfSense

Find and reimage infected crap.

It will continue tomorrow, now it is calm again, as everybody left the office :) But even if it is related to one malicious host on the LAN, it shouldn't be able to break the unbound resolver so easily…

Thanks again for all your feedbacks and until tomorrow!

Derelict

If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

You need to be looking at these queries from the root back and see where things go wrong.

Very intriguing.

doktornotor

@Derelict:

If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

Derelict

Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

I would put this on LAN:

pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log

Put that above your normal pass rule. If everything is as you say, it should log nothing.

On pfSense 2.2 you should be able to set the dest to ! This Firewall (self).

doktornotor

@Derelict:

Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

Pretty sure I could get these guys involved in investigating the issue here (they've also written the Knot DNS server so I'm rather convinced they are familiar with DNS :P) – however that'd require either remote access or at least uncensored traffic captures. Not example.com -- totally useless.

Trel

@doktornotor:

@Derelict:

If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

Using Comcast with a modem only (not a gateway in bridged mode). Here's the block rule.
With these settings, if I try to look up the domain I get this scenario

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)

I understand that an infected machine should not be on the network, but if a mere typical DNS lookup can cause this much havoc, then something is really wrong.

restricted_dns.gif_thumb

swix

@Derelict:

Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.
Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

It wasn't obfuscated, it really looked like that… (also with other domains, juste replace example.com by anything)

@Derelict:

I would put this on LAN:
pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log
Put that above your normal pass rule. If everything is as you say, it should log nothing.

Ok, thanks, will setup this.

@doktornotor:

Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

I would be really happy to know the cause, it is really strange that Trel is having a similar problem with the very same target IP "195.22.26.248", especially from different countries/ISP's. The only recent change to our infrastructure was upgrading to pfSense 2.2 at the beginning of January, otherwise nothing special. But I'll setup some network monitoring tools later this week.

Best regards

Derelict

@Trel:

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)

A search of redmine does not show that as an open issue. Have you reported it?

2chemlud

@Trel:

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)[

[/quote]

I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

Derelict

I'm not in a position to test this at the moment. Tonight.

Trel

@2chemlud:

@Trel:

When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
-A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
(persists until unbound service is restarted)

I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

When you say interruptions, could those have been unbound not responding?
Someone did mention that one of the times I was unable to restart the service manually (as I was not available) it began working again after 45-50 minutes.

Either way though, as soon as I overrode the DNS for those sites, it's never happened again.

@Derelict:

I'm not in a position to test this at the moment. Tonight.

If you're going to test, try accessing and resolving

api-nyc01.exip.org

and

ns3.csof.net

Derelict

A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything

So I can test it properly, to what domains is this referring?

Derelict

@Trel:

If you're going to test, try accessing and resolving
api-nyc01.exip.org
and
ns3.csof.net

What about it?

dig . ns

Pick a root server at random:

dig @e.root-servers.net ns3.csof.net

Get a list of gtld servers. Pick one at random:

dig @e.gtld-servers.net ns3.csof.net

Pertinent info:


;; AUTHORITY SECTION:
csof.net.		172800	IN	NS	ns61.domaincontrol.com.
csof.net.		172800	IN	NS	ns62.domaincontrol.com.

;; ADDITIONAL SECTION:
ns61.domaincontrol.com.	172800	IN	A	216.69.185.32
ns62.domaincontrol.com.	172800	IN	A	208.109.255.32

Pick one of those:

dig @216.69.185.32 ns3.csof.net


;; QUESTION SECTION:
;ns3.csof.net.			IN	A

;; ANSWER SECTION:
ns3.csof.net.		600	IN	A	*** 195.22.26.199 ***

Their name servers either want that name to resolve to 195.22.26.199 or are giving bogus information or are otherwise hacked. What, exactly, would you expect unbound to do to fix that?

Trel

Those domains aren't the problem.

The problem is what happens AFTER looking them up.

I can see there's some failure to communicate here.

Problem 1:
Without DNSSEC on and with other DNS servers allowed, a few minutes after looking up those domains, looking up google.com will return something in the 195.22.x range (persisting until I restarted unbound, or possibly 45 minutes to an hour)

At this point I switched to DNSSEC and blocked all outgoing DNS except to the firewall.

Problem 2:
With DNSSEC enabled and only unbound able to resolve, a few minutes after looking up those domains, lookup up google.com will return nothing the result will be blank as if the domain didn't exist (persisting until I restarted unbound, or possibly 45 minutes to an hour)

At this point, I put DNS overrides in for those domains setting them to 0.0.0.0 so they would not be able to be looked up at all.

At this point, the symptoms have stopped.

Derelict

Ok I just let unbound look them up.