TinyDNS service not able to start in v2.2.3-RELEASE



  • Title says it all.  No errors in any of the logs, have removed squid/dansguardian etc, reinstalled the package.  It was working fine under 1.2.3 for some years, upgraded to 2.2.3 and now it won't start.  config.xml content appears fine, can go into settings etc and all seems well.  It's currently configured to be bound only to the loopback interface on 127.0.0.1 and we port-forward from the WAN IPs, (it's part of our public-facing authoritative DNS for various domains).  Just for troubleshooting purposes also tried using 127.0.0.2 on the loopback instead, binding directly to the interface (both public and private just to see if it would at least start), disabling the DNS forwarder, etc.

    Everything looks perfectly fine, just that it won't start!

    Only possible clue is that if you make a change to the settings there is a warning at the top about needing svscan running as well, and it behaves the same way - no errors logged etc.

    Log files appear to have correct permissions, other misc stuff is being logged fine, kind of stumped here - any suggestions?



  • So nobody uses this plugin any more…? 
    We moved away from BIND because of security issues and performance, but if it's better now we could switch back - is BIND good again?


  • Banned

    Unlike all the DJBware dead code, Bind is very alive and kicking, and those security issues might have been relevant somewhere around v5 or so. Not with current code.



  • Would it make sense to pull the tinyDNS package off the package list then?  Who's in charge of blessing/rejecting packages these days…?


  • Banned

    Dunno, recently submitted some unrelated fix for tinydns, it starts just fine. Perhaps you have some other DNS server running on port 53 already. Plus, I don't believe in upgrades from prehistoric versions.



  • Well, that sounds more positive anyway.  :P

    Agree on the not upgrading, and tinyDNS is running on older 2.x deployments we have, this seems to be new to 2.2.3(upgraded).  May try it on a clean system at some point, in my copious free time.

    But…

    BIND installed and ran just fine, easier to switch than try to debug.  :)

    Now I'm off to find the easiest way to bulk import all these records into BIND!



  • Hi,

    Don't know if this will help in your case, but I fixed it by doing a:

    pkg install daemontools
    

    on the console after having installed the tinydns package.

    Seems like the package just forgets installing the prerequisite: the tinydns service cannot be started because the 'svscan' binary is missing (part of the daemontools package).

    Regards,
    Marc



  • @ecce:

    pkg install daemontools
    

    Seems like the package just forgets installing the prerequisite: the tinydns service cannot be started because the 'svscan' binary is missing (part of the daemontools package).

    OMFG, that's all it was!  Thank you so much!  Never occurred to me to check if the daemontools package could be completely missing…  :-[
    It appears that the tinyDNS installer creates the svscan service entry but doesn't check to see if there's actually anybody home.

    So tinyDNS needs a bug report about not checking dependencies.

    But, should daemontools already be there?  Should there also be a bug report against the pfSense base install for not having it?



  • @Itwerx:

    But, should daemontools already be there?  Should there also be a bug report against the pfSense base install for not having it?

    No, shouldn't be in the base. The TinyDNS install should have checked for prerequisites.



  • Well, it looks like there's at least one more tinyDNS issue going on.  It has every appearance of running (and properly responding to requests etc), but there's a persistent memory leak.

    The "supervise axfrdns" process keeps creating child procs which go zombie every few seconds.  This is resulting in ~1MB of memory being taken out of the Free pool (and never restored) repeatedly until all RAM/swap is exhausted.

    This implies that while tinyDNS appears to be working it may in fact be crashing repeatedly…



  • Digging a bit further, I find that I can run tinydns by itself (not as a child of svscan) with no trouble.  It responds to queries, does not crash and there are no readily obvious memory leaks.  So this is strictly an issue with svscan.

    Which makes me wonder if the simple fix above of installing svscan manually is actually a bad idea?  TinyDNS certainly has a different config in the pfSense environment, pulling in a bog standard FreeBSD copy of svscan might not actually work?  (At least under 2.2.3?)



  • Ugh, this is nasty - lots of references to bogus locations and symlinks to patch them into working order.  I don't even think it's a pfSense package issue, I think it's a Dr Dan issue.  (And really, who the heck feels the need to write their own service monitor for their stuff?  If it's so hairy that any of the standard methods won't work then maybe it's not ready for production use in the real world…)

    Back to BIND we go!



  • Ugh, now the other tinyDNS (different cluster, working fine) won't let me do a zone transfer from BIND.

    Anybody done a tinyDNS -> BIND zone transfer before?  I've got the IPs allowed etc, maybe a key issue…


  • Banned

    @dotdash:

    @Itwerx:

    But, should daemontools already be there?  Should there also be a bug report against the pfSense base install for not having it?

    No, shouldn't be in the base. The TinyDNS install should have checked for prerequisites.

    This is not the problem. The problem is the PBI übershit with its craptastic wannabe "self-contained" chroots and stupid paths.

    https://redmine.pfsense.org/issues/4555

    @Itwerx:

    Anybody done a tinyDNS -> BIND zone transfer before?  I've got the IPs allowed etc, maybe a key issue…

    Why the heck would anyone do any such thing? Want to transfer zones to bind? Then install bind!



  • Yes, some of this is definitely PBI related.  I added a few notes to the Redmine ticket (thanks for that!)

    tinyDNS is removed from the system in question and BIND is installed.  What I'm trying to do is get zone transfers into BIND from a different pfSense box which is still running tinyDNS, (with no problems).



  • I did try massaging the tinyDNS UI output into BIND zone file format and putting that into the zone's custom text field, but it seems to ignore them.  Guess I'll have to write a script to spit out XML for the config file.  Ugh…



  • @Itwerx:

    Which makes me wonder if the simple fix above of installing svscan manually is actually a bad idea?  TinyDNS certainly has a different config in the pfSense environment, pulling in a bog standard FreeBSD copy of svscan might not actually work?  (At least under 2.2.3?)

    Hi,
    now that I have read this and something about PBI, I too started digging deeper. The tinydns actually seems to come with an own version of svscan living in: /usr/pbi/djbdns-amd64/local/bin
    So I did a "pkg remove daemontools" and edited /usr/local/etc/rc.d/svscan.sh replacing all calls to daemontools binaries with the correct paths and added the directory to the search path in the script. Here's my diff:

    11c11
    < command="/usr/local/bin/svscan"
    ---
    > command="/usr/pbi/djbdns-amd64/local/bin/svscan"
    27,28c27,28
    <         PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    <         /usr/sbin/daemon -f /bin/sh -c "$command $svscan_servicedir 2>&1 | /usr/local/bin/multilog t $logdir" > /dev/null
    ---
    >         PATH=/usr/pbi/djbdns-amd64/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    >         /usr/sbin/daemon -f /bin/sh -c "$command $svscan_servicedir 2>&1 | /usr/pbi/djbdns-amd64/local/bin/multilog t $logdir" > /dev/null
    34c34
    <         find -L "$svscan_servicedir" -mindepth 1 -maxdepth 2 -type d \( \! -path "$svscan_servicedir/*/*" -or -name 'log' \) -print0 | xargs -0 /usr/local/bin/svc -dx
    ---
    >         find -L "$svscan_servicedir" -mindepth 1 -maxdepth 2 -type d \( \! -path "$svscan_servicedir/*/*" -or -name 'log' \) -print0 | xargs -0 /usr/pbi/djbdns-amd64/local/bin/svc -dx
    
    

    This also seems to fix the "missing svscan". I am not sure if this fixes the memory leaks or the axfr issues you described.

    Since I have also read the other comments on this thread and also came across this http://samiam.org/blog/20110103.html and this http://pjp.dgplug.org/djbdns/index.html (a still-maintained linux-only fork of djbdns), and since all together this gives me the impression that the tinydns package has no maintainer on pfsense and that it seems to be only luck how long it will continue to work, I also started looking into alternatives.

    I used to use tinydns for its failover IP capabilities - to redirect clients to the correct IP in multi-WAN failover scenarios and for having a DNS failover with multiple web servers (no, BGP is not an option ;)). Any ideas on how to do this with pfsense in a different way?
    For now I have only found this solution: http://www.freebsdwiki.net/index.php/BIND%2C_dynamic_DNS%2C_failover_A_records

    Sorry for hijacking this thread in the last paragraph.

    Marc



  • Dagnabbit, you're tempting me to give it another go!  The monitoring and failover is a big plus for us as well.  :/


  • Banned



  • https://github.com/pfsense/pfsense-packages/pull/909

    Did you have a chance to give it any review or was the github just to get it in there for further development…?


  • Banned

    @Itwerx:

    https://github.com/pfsense/pfsense-packages/pull/909

    Did you have a chance to give it any review or was the github just to get it in there for further development…?

    It's been "sitting" there for 2 days. Give it some time to get reviewed and merged…



  • It's been "sitting" there for 2 days. Give it some time to get reviewed and merged…

    Yes, of course, wasn't impugning the process.  Just figured if it had been given further review and was considered a valid patch, (not still missing paths or whatever), then I would go ahead and test it out and report back my results.  (Conversely if it's known to be incomplete than I'd wait.)  - Thanks!


  • Banned

    I don't plan on changing anything there unless required. If you want to test it, probably the best way is to paste relevant parts into http(s)://<pfs_ip_or_fqdn>/exec.php

    
    	$pf_version=substr(trim(file_get_contents("/etc/version")),0,3);
    	if ($pf_version == "2.2") {
    		$svscan_path = "/usr/pbi/djbdns-" . php_uname("m") . "/local/bin/svscan";
    		$svc_path = "/usr/pbi/djbdns-" . php_uname("m") . "/local/bin/svc";
    		$tinydns_path = "/usr/pbi/djbdns-" . php_uname("m") . "/local/bin";
    	} else if ($pf_version == "2.1") {
    		$svscan_path = "/usr/pbi/djbdns-" . php_uname("m") . "/bin/svscan";
    		$svc_path = "/usr/pbi/djbdns-" . php_uname("m") . "/bin/svc";
    		$tinydns_path = "/usr/pbi/djbdns-" . php_uname("m") . "/bin";
    	} else {
    		$svscan_path = "/usr/local/bin/svscan";
    		$svc_path = "/usr/local/bin/svc";
    		$tinydns_path = "/usr/local/bin";
    	}
    	$svscan = <<<eod<br>#!/bin/sh
    # PROVIDE: svscan
    # REQUIRE: LOGIN
    # KEYWORD: FreeBSD
    . /etc/rc.subr
    name="svscan"
    rcvar="\${name}_enable"
    command="{$svscan_path}"
    svscan_enable=\${svscan_enable-"YES"}
    svscan_servicedir=\${svscan_servicedir-"{$g['varrun_path']}/service"}
    logdir="/var/log/svscan"
    start_cmd="svscan_start"
    stop_postcmd="svscan_stop_post"
    load_rc_config \$name
    required_dirs="\${svscan_servicedir}"
    svscan_start () {
            echo "Starting svscan."
            mkdir -p \$logdir
            /usr/bin/env \
            PATH={$tinydns_path}:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
            /usr/sbin/daemon -f /bin/sh -c "\$command \$svscan_servicedir 2>&1 | /usr/local/bin/multilog t \$logdir" > /dev/null
    EOD;
    if ($enableipmonitoring) {
    	$svscan .= <<<eod<br>/usr/local/bin/minicron {$refreshinterval} {$g['varrun_path']}/ping_hosts.pid "/usr/local/bin/ping_hosts.sh"
    EOD;
    }
    $svscan .= <<<eod<br>}
    svscan_stop_post () {
            echo "Stopping svscan."
            find -L "\$svscan_servicedir" -mindepth 1 -maxdepth 2 -type d \( \! -path "\$svscan_servicedir/*/*" -or -name 'log' \) -print0 | xargs -0 {$svc_path} -dx
    		/bin/pkill -F /var/run/ping_hosts.pid
    }
    run_rc_command "\$1"
    EOD;
    
    	fwrite($fd, $svscan);
    	fclose($fd);</eod<br></eod<br></eod<br> 
    

    After that, make a backup and replace /usr/local/pkg/tinydns.inc with the fixed one.</pfs_ip_or_fqdn>



  • Thank you, Doktor!

    For sending a patch in the right direction, and for posting a How-To how we can try it.

    Worked out with minor additions in your above posted code:

    • an additional line break here:
           /usr/local/bin/minicron {$refreshinterval} {$g['varrun_path']}/ping_hosts.pid "/usr/local/bin/ping_hosts.sh"
    EOD;
    }
    $svscan .= <<<eod<br>--->>>line break here<<<---
    }
    svscan_stop_post () {</eod<br>
    
    • an additional line before fwrite (had to instant-teach myself a little bit php for this :)):
    $fd=fopen("/usr/local/etc/rc.d/svscan.sh","w");
    
    • and after that edit /usr/local/etc/rc.d/svscan.sh and remove all ^M chars at the end of the line. (yes, I know…windoze)

    Again, thanks!
    Marc


  • Banned

    @ecce:

    and after that edit /usr/local/etc/rc.d/svscan.sh and remove all ^M chars at the end of the line. (yes, I know…windoze)

    
    sed -i "s/.$//g" /some/file
    
    


  • Am I correct in assuming this snippet of code is sufficiently self-contained that it can go pretty much anywhere in exec.php, (not literally of course, being careful not to break existing syntax)…?


  • Banned

    PR merged, no longer needed to mess with anything manually.



  • Thanks Doc, that was fast!

    I did a reinstall on the package, and while I don't see any of the new svscan stuff in the exec.php I am happy to report that it  seems to be running properly now.  Woohoo!  Much rockage!  :P



  • Yes, can confirm that - installed it on a system which didn't have anything to do with tinydns before, configured basic settings, started svscan and that was it!

    Yay!
    ;D


  • Banned

    Thanks for testing. ;)