Filterdns stops working
-
Problem persists even in Pfsense 2.4.4
-
What problem ?
Look carefully at the log that @hcww provided. Thousands of line all one the moment (Oct 23 23:15:21) because he was asking to resolve an URL that return many, many (no even more : more then 5000 entries at ones ) IP addresses.
That will bring down a this task for sure. Probably the entire system.One might try to get an hold on every IP address that the big ones** use, but, looking at the log and you'll find out that that is not a good idea. Known subject btw.
** Google, Facebook, Twitter, etc
-
@muhammad-waqas said in Filterdns stops working:
Problem persists even in Pfsense 2.4.4
Luckily it got resolved in 2.4.4p1, so update before you complain.
-
We're running 2.4.4-RELEASE-p2 (amd64), but the issue is still there for us. Over the last two weeks I've had two occurances of strange issues, people being unable to connect and such, and it turned out SOME of the aliasses weren't resolved. So far the only thing that helps is killing filterdns as people suggested.
-
@rgijsen said in Filterdns stops working:
and it turned out SOME of the aliasses weren't resolved
What hosts ?
What happened with the DNS at that moment ? (logs ..)
Default Resolver DNS - or a "have it handled by someone else" ? -
@gertjan said in Filterdns stops working:
@rgijsen said in Filterdns stops working:
and it turned out SOME of the aliasses weren't resolved
What hosts ?
What happened with the DNS at that moment ? (logs ..)
Default Resolver DNS - or a "have it handled by someone else" ?As far as I can tell now, only internal hosts in both cases. However, the default resolver are two internal AD controllers, hence acting as DNS servers as well. They resolve the internal names and using root-hints or if that fails forwarders, they resolve external names. Those two are our main DNS servers. If BOTH of them would be unreachable / not responding, I'd be completely down.
Unfortunately, I've don't have enough backlog in resolver.log to see the issue I had seemingly yesterday. So I'll have to increase my log file space. Would Status --> System --> LogsSettings --> Log file size (Bytes) include logs like resolver.log?
-
In your case : check log of your AD controller.
The request for a local host was received ? The answer was send back ? -
@gertjan We don't have DNS debug-logging enabled by default, as usually we don't have any need for it. However, I'm pretty confident it was up and running, at least one node (although I don't have any reason to believe even one of them had issues). Our monitoring system (Zabbix) relies on DNS heavily. If DNS wouldn't repsond, pretty much all hosts would be not available to our monitoring systems. And of course without proper DNS nothing really works.
pfSense has an interface in the net where both AD/DNS servers are, so no routing involved. The thing is, this probably started yesterday (sunday) in the afternoon; but at least the issue was there this morning. Both DNS were up and running, no issues, but some names were not resolved by pfSense. Only when I killed filterdns and started a new instance by saving an existing alias, it started resolving again, without doing any other changes on the DNS machines or anything else. This makes me believe there was an issue with pfSense / filterdns.
I'll increase log size, so I have some more backlog in the future.
[edit]
My alias-sesolving time is 300s (default). I just found that when I add a new host based on DNS (a host residing on internet this time), even after 15 minutes the table for that alias isn't updated. Also resolver.log doesn't show any activity. -
@rgijsen
I'm onto something now. I can currently reproduce some of the issue I think. When I add 'specific' hosts to an alias, they DO get resolved by our DNS:2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Rcv <pfsense IP> a463 Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)
2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Rcv <pfsense IP> 519a Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)
2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Snd <pfsense IP> a463 R Q [8081 DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)
2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Snd <pfsense IP> 519a R Q [8085 A DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)
This is an external host, i.e. a DNS that needs to be externaly resolved by our DNS servers. That seems to work fine, however the host does NOT end up in the table for that alias. When I add another DNS, same domain, so hosted at the same DNS on internet, that works fine. I tried others like www.tweakers.net, www.nos.nl or bbc.co.uk I have the same success loggings in my DNS debug log, and they DO end up in the alias table as well. At first I though the issue was with hosts that are already in a table somewhere, but that doesn't seem to be the case. Most internal names I tried now don't end up in that table either.
pfSense Resolver log:
Feb 18 12:47:14 filterdns Adding host <Host that gets added to the alias> (I just added that one in the alias)
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <Host that gets added to the alias>
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <host that does NOT end up in table> (I just added that one in the alias as well)
Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: www.ict-net.nlThe host that does NOT end up in table here, is by the way successfully added to some other aliasses, where it works just as expected. But for this alias I am missing the 'Adding host' in the pfSense log.
I expect something fishy is going on. I obviously don't want my unmasked logs online, but if there's a better way to show you what's actually happening I'd be glad to do so.
[edit]
One more addition:
I tried creating a new alias, with the same three hosts as in the alias I used above. Here NONE of them end up in the table, after waiting for about 20 minutes, while in the alias used above two out of three (and the same two every time, no matter what order I put them in) work.
I again killed filterdns, restarted it and poof - the tables immediately got filled. So it seems filterdns is partially functional - some hosts get added, some aren't. -
Seems to be https://redmine.pfsense.org/issues/9296
I've reverted my test-environment to 2.4.4-Release and the issue is not there.
