Filterdns stops working



  • this is annoying, it happened again this night. yesterday after 11pm no more updates.
    for the moment i will add a cronjob to restart filterdns every hour.

    ![2018-01-06 13_14_43-pfSenseStatus_ System Logs_ System_ DNS Resolver.png](/public/imported_attachments/1/2018-01-06 13_14_43-pfSenseStatus_ System Logs_ System_ DNS Resolver.png)
    ![2018-01-06 13_14_43-pfSenseStatus_ System Logs_ System_ DNS Resolver.png_thumb](/public/imported_attachments/1/2018-01-06 13_14_43-pfSenseStatus_ System Logs_ System_ DNS Resolver.png_thumb)



  • I have the exact same issue on 2.4.2-RELEASE-p1. I really hope someone finds a fix to this issue.

    maybe the cronjob is what i need for now?



  • @m0nji:

    this is annoying, it happened again this night. yesterday after 11pm no more updates.
    for the moment i will add a cronjob to restart filterdns every hour.

    Can you share your cron job with me? I'm not that familiar with cronjob's and i have the same issue. would like a temp fix for now until they fix the bug.



  • @Doboy:

    Can you share your cron job with me? I'm not that familiar with cronjob's and i have the same issue. would like a temp fix for now until they fix the bug.

    sure….
    look at the picture below

    Content of cron_filterdns.sh

    
    #!/bin/sh
    pkill filterdns
    /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 240 -c /var/etc/filterdns.conf -d 1
    
    

    ![2018-01-10 12_12_26-Services_ Cron_ Edit.png](/public/imported_attachments/1/2018-01-10 12_12_26-Services_ Cron_ Edit.png)
    ![2018-01-10 12_12_26-Services_ Cron_ Edit.png_thumb](/public/imported_attachments/1/2018-01-10 12_12_26-Services_ Cron_ Edit.png_thumb)


  • Rebel Alliance Developer Netgate

    When the problem happens, does /var/run/filterdns.pid contain a valid PID for filterdns? (Check the "ps uxaww" output to find the filterdns pid)

    If you do "killall -HUP filterdns" do the entries resolve again, or are they still missing?

    The code in place now tries not to restart filterdns unless it absolutely has to, but perhaps there is some issue when it's running for prolonged periods were it gets confused or fails to update when expected.



  • i will disable my cronjob and give you feedback when it happens again @jimp



  • a little update from me:
    after the reboot 8 days ago everything still works (with the modification of the aliases).



  • and again, no more update after 11pm.
    "killall -HUP filterdns" has no effect at all.






  • ok, problem is back on my system :'(

    @jimp:

    When the problem happens, does /var/run/filterdns.pid contain a valid PID for filterdns? (Check the "ps uxaww" output to find the filterdns pid)

    yes, its the pid of the running filterdns-process.

    @jimp:

    If you do "killall -HUP filterdns" do the entries resolve again, or are they still missing?

    it doesnt change anything on my machine.

    one thing changed:
    the old entries were not deleted this time. but they are not updated.
    new entries are not resolved but they are added to the filterdns.conf.

    also the mentioned cron-job from m0nji doesnt seem to be a good idea, since the "pkill filterdns" doesn't end the filterdns-process (at least not on my pf). that means only more and more instances of filterdns will start.

    is there any way to really kill that process so i can start a fresh instance?



  • I have the same problem!
    No filterDNS in the system->DNS Resolver logs after a week or some days.



  • Somebody with a solution?



  • not really a solution, only a workaround:
    run "killall -9 filterdns" in the shell and then "/usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1" (or save and apply an existing alias). you could also put them in a cron as already mentioned.

    since the error happened to me again, i checked the resolver.log again to see if there is any information what the reason could be. even with debug level 3 there is no clue.
    the last working entries are some normal adding and clearing entries and some information about some static entries.
    the next time filterdns should run, it starts with a "Received signal Hangup(1)" entry and only one entry gets deleted and the static entries are listed.
    after that every time filterdns should run (automatically or after a manual save&apply of an alias), only the hangup-message is in log.

    ps: seems this thread is about the same problem. maybe both threads should be merged.



  • I can confirm, same issue is happening with me. It seems to be it, it started happening after pfSense upgrade in Sept-Nov 2017. I am using development snapshot from 10th of January, issue still persists.

    am using Policy Based Routing (PBR) and heavily rely on a lot of aliases: it took time to realize that tables of IP addresses (referring hostname based aliases) are not updated.

    So temporary workaround so far is same, what you have suggested:

    killall -9 filterdns
    rm /var/pid/filterdns.pid (not sure if it correct path, just writing from my head)

    and then start filterdns process again (or refresh aliases).

    In fact, starting filterdns (with proper arguments) sometimes did not help, I had to kill the process again and refresh (edit-save-apply) one of aliases lists.


  • Rebel Alliance Developer Netgate

    Do we have any reliable and predictable way to trigger this issue? Any specific alias contents that cause it? Is there a set interval at which the problem occurs? Is there some other event that causes it to fail?



  • I'm not having any issue with filterdns,  but I'm using it, it resolves a couple of (very static) URL's to IPv4 and IPv6.

    When you guys kill filterdns, restart it like this :

    /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 240 -c /var/etc/filterdns.conf -d 7
    

    or even

    /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 240 -c /var/etc/filterdns.conf -d 7 -f
    

    "-d 7" will produce massive logging is the DNS log. Something might show up.
    "-f" will keep it in the foreground, so keep your console access open for the time being. Ctrl-C will end it.

    Try also chancing the interval "-i 240" (4 minutes) to "i -600" (every 10 minutes) to give it more time.

    Btw : "filterdns" is a pretty simple FreeBSD package (program), you'll find it here : https://github.com/pfsense/FreeBSD-ports/blob/devel/net/filterdns/files/filterdns.c
    It doesn't do much, and it depends on one important thing : DNS should be working.
    Also, it injects modifications into pf tables.
    Knowing that all spawned threads (as many as there are tables) are relaunched every "-i xxx" seconds at the same time, is it possible that "pf " gets "overrun" ?



  • @jimp:

    Do we have any reliable and predictable way to trigger this issue? Any specific alias contents that cause it? Is there a set interval at which the problem occurs? Is there some other event that causes it to fail?

    Nope, no trigger (at least i haven't found one).
    Nope, i changed my aliases earlier in this thread.
    Nope, no specific interval on my system. Sometimes its some days, sometimes weeks.
    Nope, i checked the other logfiles. I found no other action that was between last working and first not working filterdns.

    I will increase the interval and debuglevel as Gertjan suggests. Maybe then we find some clues.



  • seems the entries in the alias are not the reason.
    i put the exact fdqn, ips and networks into aliases with the same name on my pfsense at home (more than one month ago).
    i had no problems at home but some here at work in that time.



  • Just chiming in that we're experiencing this issue too. Tried the kill and restart in verbose mode, but the logs just contained the "clearing entry x from pf table z on host y" messages, and then nothing further. I've put a cron job in place to do this nightly, as it's effecting our ability for our web apps to send mail via Amazon SES.

    For what it's worth, this happens to us about every 2 weeks, it seems, but I've been unable to dig up any correlation to anything.



  • Will the bug finally be resolved with 2.4.4?

    The bug report is open for 8 months now. Although wie tried to mitigate the problems when it occured for the first time for us in November, we still had problems every time filterdns stopped.

    If you're affected from the bug the problems will in many cases be critical, as either access will be allowed when it should be blocked or if you're using a whitelist approach systems or services will break because of the connection problems.



  • Can anyone affected by this please try increasing their debuglevel for filterdns using this commit in System Patches?
    https://github.com/luckman212/pfsense/commit/72834bf677bdbd1cf78f6772b79abe4b3eaa8235

    After that you can follow the logs via console/ssh

    clog -f /var/log/resolver.log
    

    Related redmine: https://redmine.pfsense.org/issues/8758



  • Dear All

    i am affected with same problem
    it happens every day approx.
    i must kill filterdns service and restart to make worked again
    i increased the debug level for filterdns
    and i attached the resolver log file
    best regards,
    0_1540362451825_resolver.txt



  • Problem persists even in Pfsense 2.4.4



  • What problem ?
    Look carefully at the log that @hcww provided. Thousands of line all one the moment (Oct 23 23:15:21) because he was asking to resolve an URL that return many, many (no even more : more then 5000 entries at ones ) IP addresses.
    That will bring down a this task for sure. Probably the entire system.

    One might try to get an hold on every IP address that the big ones** use, but, looking at the log and you'll find out that that is not a good idea. Known subject btw.

    ** Google, Facebook, Twitter, etc


  • Banned

    @muhammad-waqas said in Filterdns stops working:

    Problem persists even in Pfsense 2.4.4

    Luckily it got resolved in 2.4.4p1, so update before you complain.



  • We're running 2.4.4-RELEASE-p2 (amd64), but the issue is still there for us. Over the last two weeks I've had two occurances of strange issues, people being unable to connect and such, and it turned out SOME of the aliasses weren't resolved. So far the only thing that helps is killing filterdns as people suggested.



  • @rgijsen said in Filterdns stops working:

    and it turned out SOME of the aliasses weren't resolved

    What hosts ?
    What happened with the DNS at that moment ? (logs ..)
    Default Resolver DNS - or a "have it handled by someone else" ?



  • @gertjan said in Filterdns stops working:

    @rgijsen said in Filterdns stops working:

    and it turned out SOME of the aliasses weren't resolved

    What hosts ?
    What happened with the DNS at that moment ? (logs ..)
    Default Resolver DNS - or a "have it handled by someone else" ?

    As far as I can tell now, only internal hosts in both cases. However, the default resolver are two internal AD controllers, hence acting as DNS servers as well. They resolve the internal names and using root-hints or if that fails forwarders, they resolve external names. Those two are our main DNS servers. If BOTH of them would be unreachable / not responding, I'd be completely down.

    Unfortunately, I've don't have enough backlog in resolver.log to see the issue I had seemingly yesterday. So I'll have to increase my log file space. Would Status --> System --> LogsSettings --> Log file size (Bytes) include logs like resolver.log?



  • In your case : check log of your AD controller.
    The request for a local host was received ? The answer was send back ?



  • @gertjan We don't have DNS debug-logging enabled by default, as usually we don't have any need for it. However, I'm pretty confident it was up and running, at least one node (although I don't have any reason to believe even one of them had issues). Our monitoring system (Zabbix) relies on DNS heavily. If DNS wouldn't repsond, pretty much all hosts would be not available to our monitoring systems. And of course without proper DNS nothing really works.

    pfSense has an interface in the net where both AD/DNS servers are, so no routing involved. The thing is, this probably started yesterday (sunday) in the afternoon; but at least the issue was there this morning. Both DNS were up and running, no issues, but some names were not resolved by pfSense. Only when I killed filterdns and started a new instance by saving an existing alias, it started resolving again, without doing any other changes on the DNS machines or anything else. This makes me believe there was an issue with pfSense / filterdns.

    I'll increase log size, so I have some more backlog in the future.

    [edit]
    My alias-sesolving time is 300s (default). I just found that when I add a new host based on DNS (a host residing on internet this time), even after 15 minutes the table for that alias isn't updated. Also resolver.log doesn't show any activity.



  • @rgijsen
    I'm onto something now. I can currently reproduce some of the issue I think. When I add 'specific' hosts to an alias, they DO get resolved by our DNS:

    2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Rcv <pfsense IP> a463 Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

    2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Rcv <pfsense IP> 519a Q [0001 D NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

    2/18/2019 12:39:54 PM 1B40 PACKET 000001A857BE1DC0 UDP Snd <pfsense IP> a463 R Q [8081 DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

    2/18/2019 12:39:54 PM 1B3C PACKET 000001A858859CC0 UDP Snd <pfsense IP> 519a R Q [8085 A DR NOERROR] AAAA (8)host(7)i'm(2)resolving(0)

    This is an external host, i.e. a DNS that needs to be externaly resolved by our DNS servers. That seems to work fine, however the host does NOT end up in the table for that alias. When I add another DNS, same domain, so hosted at the same DNS on internet, that works fine. I tried others like www.tweakers.net, www.nos.nl or bbc.co.uk I have the same success loggings in my DNS debug log, and they DO end up in the alias table as well. At first I though the issue was with hosts that are already in a table somewhere, but that doesn't seem to be the case. Most internal names I tried now don't end up in that table either.

    pfSense Resolver log:
    Feb 18 12:47:14 filterdns Adding host <Host that gets added to the alias> (I just added that one in the alias)
    Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <Host that gets added to the alias>
    Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: <host that does NOT end up in table> (I just added that one in the alias as well)
    Feb 18 12:47:14 filterdns Adding Action: pf table: B_it_webserver host: www.ict-net.nl

    The host that does NOT end up in table here, is by the way successfully added to some other aliasses, where it works just as expected. But for this alias I am missing the 'Adding host' in the pfSense log.

    I expect something fishy is going on. I obviously don't want my unmasked logs online, but if there's a better way to show you what's actually happening I'd be glad to do so.

    [edit]
    One more addition:
    I tried creating a new alias, with the same three hosts as in the alias I used above. Here NONE of them end up in the table, after waiting for about 20 minutes, while in the alias used above two out of three (and the same two every time, no matter what order I put them in) work.
    I again killed filterdns, restarted it and poof - the tables immediately got filled. So it seems filterdns is partially functional - some hosts get added, some aren't.



  • Seems to be https://redmine.pfsense.org/issues/9296

    I've reverted my test-environment to 2.4.4-Release and the issue is not there.


Log in to reply