Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    So many filterdns instances…

    2.1 Snapshot Feedback and Problems - RETIRED
    10
    57
    18.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      eri--
      last edited by

      I am working on cleaning those up.

      1 Reply Last reply Reply Quote 0
      • P
        phil.davis
        last edited by

        I upgraded 1 system to:
        2.1-BETA1 (i386)
        built on Wed Jan 2 16:40:07 EST 2013
        FreeBSD 8.3-RELEASE-p5

        In the system log, every 5 minutes, is:

        Jan  3 08:16:35 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:21:35 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:26:35 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:31:35 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:36:35 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:41:35 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:46:36 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:51:36 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 08:56:36 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 09:01:36 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 09:06:36 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  3 09:11:36 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known

        /var/etc/filterdns.conf just has a list of dynDNS IP names that are used in an alias (actual names changed in the text below):

        pf name-1.dyndns-ip.com INF_inet_ips
pf name-2.dyndns-ip.com INF_inet_ips
pf name-3.dyndns-ip.com INF_inet_ips
pf name-4.dyndns-ip.com INF_inet_ips
pf name-5.dyndns-ip.com INF_inet_ips
pf name-6.dyndns-ip.com INF_inet_ips
pf name-7.dyndns-ip.com INF_inet_ips
pf name-8.dyndns-ip.com INF_inet_ips
pf name-9.dyndns-ip.com INF_inet_ips
pf name-10.dyndns-ip.com INF_inet_ips
pf name-11.dyndns-ip.com INF_inet_ips

        Systems on 31 Dec 2012 snapshots are not getting this in the system log.
        I guess something in the recent filterdns code changes that Ermal is working on is processing a blank line somewhere?

        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

        1 Reply Last reply Reply Quote 0
        • E
          eri--
          last edited by

          Upgrade to today snapshot(Jan 3) it should be better maybe you caught a snap with intermediate changes.

          Also if you run top -H you should see the hostnames on each tread run for them.

          1 Reply Last reply Reply Quote 0
          • P
            phil.davis
            last edited by

            A later/Jan 3 snap is not up yet. I will upgrade and report back when Jan 3 snap appears.

            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

            1 Reply Last reply Reply Quote 0
            • P
              phil.davis
              last edited by

              2.1-BETA1 (i386)
              built on Thu Jan 3 02:32:11 EST 2013
              FreeBSD 8.3-RELEASE-p5
              still has the same failed looking up "(null)" message every 5 minutes.
              There is another snap up now 06:39 - I'll load that now and see…

              As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
              If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

              1 Reply Last reply Reply Quote 0
              • P
                phil.davis
                last edited by

                2.1-BETA1 (i386)
                built on Thu Jan 3 19:04:10 EST 2013
                FreeBSD 8.3-RELEASE-p5

                Jan  4 08:51:17 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  4 08:56:17 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  4 09:01:18 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
Jan  4 09:06:18 imp-rt-01 filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known

                This message is still logged every 5 minutes.

                As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                1 Reply Last reply Reply Quote 0
                • P
                  phil.davis
                  last edited by

                  I checked in Diagnostics:Tables.

                  1. On a system that is running Mon Dec 31 12:20:48 EST 2012 snap (before the recent filterdns changes), my INF_iinet_ips table is long - it has the current 11 IP addresses that go with the 11 names in the table, and also has lots of old IP addresses that were dynamically allocated in the past.
                    (I think the recent filterdns changes will now be clearing up old entries)

                  2. On the system running Thu Jan 3 19:04:10 EST 2013 snap, there are exactly 11 IP addresses in the table, but they are out-of-date compared to the addresses I get with nslookup from my desktop. I rebooted and the 11 IP addresses are now current (so filterdns must be looking them up OK when it starts). I will monitor the table and see if the addresses go out-of-date over time.

                  filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known

                  still in syslog every 5 minutes.

                  As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                  If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                  1 Reply Last reply Reply Quote 0
                  • E
                    eri--
                    last edited by

                    Should be corrected with tomorrow snapshot.

                    1 Reply Last reply Reply Quote 0
                    • P
                      phil.davis
                      last edited by

                      2.1-BETA1 (i386)
                      built on Fri Jan 4 17:38:46 EST 2013
                      FreeBSD 8.3-RELEASE-p5
                      Alix 32-bit nanoBSD
                      filterdns starts at bootup and successfully fills gets the current IP addresses for the 11 names in my alias table.
                      5 minutes later it dies (when it wakes up to check again, I suppose), with this in syslog:

                      kernel: pid 24638 (filterdns), uid 0: exited on signal 11

                      ps ax | grep filterdns
                      reveals that there is no filterdns process any more.
                      I rebooted, and the same behaviour is repeatable.

                      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                      1 Reply Last reply Reply Quote 0
                      • C
                        cybercare
                        last edited by

                        You probably need to try todays snap as he said yesterday it would be in todays and you are still listing a jan 4 snap.

                        1 Reply Last reply Reply Quote 0
                        • D
                          dhatz
                          last edited by

                          Seems that running latest snapshot filterdns still has some issues

                          clog system.log | tail

                          Jan  6 00:58:26 fw php: : Creating rrd update script
                          Jan  6 00:58:28 fw php: : Forcefully reloading IPsec racoon daemon
                          Jan  6 00:58:28 fw php: : Restarting/Starting all packages.
                          Jan  6 00:58:30 fw dhclient[17095]: DHCPREQUEST on em0 to x.y.z.w port 67
                          Jan  6 00:58:30 fw dhclient[17095]: DHCPACK from x.y.z.w
                          Jan  6 00:58:30 fw dhclient: RENEW
                          Jan  6 00:58:30 fw dhclient: Creating resolv.conf
                          Jan  6 00:58:30 fw dhclient[17095]: bound to x.y.z.201 – renewal in 43200 seconds.
                          Jan  6 00:58:31 fw php: : Resyncing OpenVPN instances for interface WAN.
                          Jan  6 00:58:31 fw kernel: pid 50069 (filterdns), uid 0: exited on signal 11 (core dumped)
                          Jan  6 00:58:32 fw php: : IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
                          Jan  6 00:58:34 fw login: login on ttyv0 as root
                          Jan  6 00:58:36 fw check_reload_status: Updating all dyndns
                          Jan  6 00:58:36 fw check_reload_status: Restarting ipsec tunnels
                          Jan  6 00:58:36 fw check_reload_status: Restarting OpenVPN tunnels/interfaces
                          Jan  6 00:58:36 fw check_reload_status: Reloading filter
                          Jan  6 00:58:43 fw php: : IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
                          Jan  6 00:58:47 fw kernel: pid 87410 (filterdns), uid 0: exited on signal 11 (core dumped)
                          Jan  6 01:01:22 fw php: /firewall_rules.php: Successful login for user 'admin' from: 192.168.100.12
                          Jan  6 01:01:22 fw php: /firewall_rules.php: Successful login for user 'admin' from: 192.168.100.12

                          uname -a

                          FreeBSD fw.localdomain 8.3-RELEASE-p5 FreeBSD 8.3-RELEASE-p5 #1: Sat Jan  5 13:23:58 EST 2013     root@snapshots-8_3-i386.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8  i386

                          It had the same issue with previous snapshots:

                          Jan  5 00:17:03 fw kernel: pid 48375 (filterdns), uid 0: exited on signal 11 (core dumped)
                          Jan  5 03:25:34 fw kernel: pid 45341 (filterdns), uid 0: exited on signal 11 (core dumped)
                          Jan  5 03:36:13 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 03:46:55 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 03:57:37 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 04:08:19 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 04:19:01 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 04:29:44 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 04:40:26 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 04:51:08 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  5 05:02:15 fw filterdns: host_dns: failed looking up "(null)": hostname nor servname provided, or not known
                          Jan  6 00:58:31 fw kernel: pid 50069 (filterdns), uid 0: exited on signal 11 (core dumped)
                          Jan  6 00:58:47 fw kernel: pid 87410 (filterdns), uid 0: exited on signal 11 (core dumped)
                          Jan  6 01:08:57 fw kernel: pid 24930 (filterdns), uid 0: exited on signal 11 (core dumped)

                          ls -la /filterdns.core

                          -rw–-----  1 root  wheel  4661248 Jan  6 01:08 /filterdns.core

                          1 Reply Last reply Reply Quote 0
                          • P
                            phil.davis
                            last edited by

                            2.1-BETA1 (i386)
                            built on Sat Jan 5 17:06:02 EST 2013
                            FreeBSD 8.3-RELEASE-p5
                            Now I should definitely have all the recent filterdns code changes. Still have the same symptoms, the table gets the correct 11 IP addresses translated from the names at boot. 5 minutes later, filterdns dies:

                            [2.1-BETA1][admin@imp-rt-01.imp.infn]/var/log(6): clog system.log | grep filterdns
Jan  6 11:55:27 imp-rt-01 kernel: pid 27624 (filterdns), uid 0: exited on signal 11

                            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                            1 Reply Last reply Reply Quote 0
                            • E
                              eri--
                              last edited by

                              Hrm strange that you see that.
                              5 minutes is the default update interval for rechecking names.

                              I have run test here with 5 seconds and 10 second update intervals but no issues in that regard!
                              That makes still thing the snaps do not have the latest version of filterdns.

                              Can you make a md5 of your filterdns ?

                              1 Reply Last reply Reply Quote 0
                              • D
                                dhatz
                                last edited by

                                @ermal:

                                Can you make a md5 of your filterdns ?

                                MD5 (/usr/local/sbin/filterdns) = b25470f1942956d6f887ff87c99761c4

                                1 Reply Last reply Reply Quote 0
                                • P
                                  phil.davis
                                  last edited by

                                  2.1-BETA1 (i386)
                                  built on Sun Jan 6 11:15:50 EST 2013
                                  FreeBSD 8.3-RELEASE-p5

                                  MD5 (/usr/local/sbin/filterdns) = b25470f1942956d6f887ff87c99761c4

                                  5 minutes after startup:

                                  [2.1-BETA1][admin@rt-01.mydomain]/root(2): clog /var/log/system.log | grep filterdns
Jan  7 08:07:02 rt-01 kernel: pid 28781 (filterdns), uid 0: exited on signal 11

                                  As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                  If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    dhatz
                                    last edited by

                                    Just bumping up this thread, since filterdns is still exiting + dumping core (note: I had just updated to latest 2.1-BETA1 snapshot)

                                    1 Reply Last reply Reply Quote 0
                                    • P
                                      phil.davis
                                      last edited by

                                      Bump from me also, now on:
                                      2.1-BETA1 (i386)
                                      built on Sun Jan 13 19:34:21 EST 2013
                                      FreeBSD 8.3-RELEASE-p5
                                      and still getting:

                                      Jan 14 12:09:19 imp-rt-01 kernel: pid 34114 (filterdns), uid 0: exited on signal 11 (core dumped)

                                      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                      1 Reply Last reply Reply Quote 0
                                      • P
                                        phil.davis
                                        last edited by

                                        Some more information. filterdns only crashes if SIGHUP is received and it goes through the "Cleaning up previous hostnames" code:

                                        Jan 16 08:57:26 imp-rt-01 filterdns: Received signal SIGHUP(1).
Jan 16 08:57:26 imp-rt-01 filterdns: Cleaning up previous hostnames

                                        This happens as various interfaces and OpenVPN links come up during startup - filter reloads happen a few times, and are fed to filterdns. It dies with sig 11 at the next scheduled 5 minute wakeup.
                                        Something in the reload of filterdns.conf and attempted preservation of existing threads, removal of threads no longer needed, and addition of threads to monitor new IPs, is freeing memory that is still needed. In filterdns.c, merge_config calls clear_config:

                                        static void
clear_config(struct thread_list *thrlist)
{
	struct thread_data *thr;

	pthread_mutex_lock(&sig_mtx);
	while ((thr = TAILQ_FIRST(thrlist)) != NULL) {
		if (debug >= 4)
			syslog(LOG_ERR, "Cleaning up hostname %s", thr->hostname);
		TAILQ_REMOVE(thrlist, thr, next);
		if (thr->thr_pid != 0)
			pthread_cancel(thr->thr_pid);
		clear_hostname_addresses(thr);
		if (thr->hostname)
			free(thr->hostname);
		if (thr->tablename)
			free(thr->tablename);
		free(thr);
	}
	pthread_rwlock_unlock(&main_lock);
}

                                        merge_config sets thr_pid to 0 for threads that should continue on (do not need to be cancelled). But clear_config frees various data for the thread (hostname and tablename) and the thread data itself, even when the thread is not cancelled.
                                        When the thread awakes in check_hostname at the 5 minute timer, it will have lost its thr data structure - reference to it will cause sig 11.
                                        Perhaps it just needs this code for clear_config:

                                        static void
clear_config(struct thread_list *thrlist)
{
	struct thread_data *thr;

	pthread_mutex_lock(&sig_mtx);
	while ((thr = TAILQ_FIRST(thrlist)) != NULL) {
		if (debug >= 4)
			syslog(LOG_ERR, "Cleaning up hostname %s", thr->hostname);
		TAILQ_REMOVE(thrlist, thr, next);
		if (thr->thr_pid != 0) {
			pthread_cancel(thr->thr_pid);
			clear_hostname_addresses(thr);
			if (thr->hostname)
				free(thr->hostname);
			if (thr->tablename)
				free(thr->tablename);
			free(thr);
		}
	}
	pthread_rwlock_unlock(&main_lock);
}

                                        Also, "pthread_rwlock_unlock(&main_lock);" at the end seems odd. Shouldn't it be "pthread_mutex_unlock(&sig_mtx);" - to match the "pthread_mutex_lock(&sig_mtx);" at the start of the routine?
                                        @ermal: I don't have an environment to compile in, but this might give enough clues for you to track this down.

                                        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                        1 Reply Last reply Reply Quote 0
                                        • E
                                          eri--
                                          last edited by

                                          Thanks for the analysis pushed a fix.

                                          1 Reply Last reply Reply Quote 0
                                          • P
                                            phil.davis
                                            last edited by

                                            Thanks, now it doesn't crash. But somewhere in the boot process, with OpenVPN links etc coming up, it has a point where it deletes all the table entries then does not recover them again. After boot, my table that should have 11 IP addresses is empty. The log indicates entries being deleted at one point.
                                            As a side issue:

                                            syslog(LOG_WARNING, "\t DELETED %d addresses(%d) to table %s.", io.pfrio_nadd, address->sa_family, pfd->tablename);

                                            should be:

                                            syslog(LOG_WARNING, "\t DELETED %d addresses(%d) to table %s.", io.pfrio_ndel, address->sa_family, pfd->tablename);

                                            (the debug line is reporting pfrio_nadd when it needs to report pfrio_ndel)

                                            If I restart filterdns (kill it by hand, then use Diagnostics:Execute Command:PHP Execute to do:

                                            mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns.pid -i 300 -c {$g['varetc_path']}/filterdns.conf -d 10");

                                            It comes up nicely and puts all 11 IPs in the table.
                                            After this, the entries survive when I stop and start an OpenVPN client process - the log looks good.
                                            @ermal: I will PM you a log of filterdns behaviour at boot with -d 10 set.

                                            As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                            If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.