Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    So many filterdns instances…

    Scheduled Pinned Locked Moved 2.1 Snapshot Feedback and Problems - RETIRED
    57 Posts 10 Posters 20.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      eri--
      last edited by

      Thanks for the analysis pushed a fix.

      1 Reply Last reply Reply Quote 0
      • P
        phil.davis
        last edited by

        Thanks, now it doesn't crash. But somewhere in the boot process, with OpenVPN links etc coming up, it has a point where it deletes all the table entries then does not recover them again. After boot, my table that should have 11 IP addresses is empty. The log indicates entries being deleted at one point.
        As a side issue:

        syslog(LOG_WARNING, "\t DELETED %d addresses(%d) to table %s.", io.pfrio_nadd, address->sa_family, pfd->tablename);

        should be:

        syslog(LOG_WARNING, "\t DELETED %d addresses(%d) to table %s.", io.pfrio_ndel, address->sa_family, pfd->tablename);

        (the debug line is reporting pfrio_nadd when it needs to report pfrio_ndel)

        If I restart filterdns (kill it by hand, then use Diagnostics:Execute Command:PHP Execute to do:

        mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns.pid -i 300 -c {$g['varetc_path']}/filterdns.conf -d 10");

        It comes up nicely and puts all 11 IPs in the table.
        After this, the entries survive when I stop and start an OpenVPN client process - the log looks good.
        @ermal: I will PM you a log of filterdns behaviour at boot with -d 10 set.

        As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
        If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

        1 Reply Last reply Reply Quote 0
        • P
          phil.davis
          last edited by

          Also, in filterdns.c main, it:
          a) reads the config, filling in thread_list
          b) loops creating a check_hostname thread for each host
          c) inits main_lock
          d) creates the thread for merge_config

          	TAILQ_FOREACH(thr, &thread_list, next) {
		error = pthread_create(&thr->thr_pid, &attr, check_hostname, thr);
		if (error != 0) {
			if (debug >= 1)
				syslog(LOG_ERR, "Unable to create monitoring thread for host %s", thr->hostname);
		}
		pthread_set_name_np(thr->thr_pid, thr->hostname);
	}

	pthread_rwlock_init(&main_lock, NULL);
	sig_mtx = PTHREAD_MUTEX_INITIALIZER;
        sig_condvar = PTHREAD_COND_INITIALIZER;
	error = pthread_create(&sig_thr, &attr, merge_config, NULL);
	if (error != 0) {
		if (debug >= 1)
			syslog(LOG_ERR, "Unable to create signal thread %s", thr->hostname);
	}
	pthread_set_name_np(sig_thr, "signal-thread");

          But check_hostname uses main_lock. So it is possible that main_lock is not initialized when check_hostname runs the first time.
          Maybe that could cause some early accesses to thread_list to be inconsistent?
          Maybe:

          pthread_rwlock_init(&main_lock, NULL);

          should be moved earlier in main.

          As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
          If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

          1 Reply Last reply Reply Quote 0
          • E
            eri--
            last edited by

            I did make the code correct but i think the issue was mostly related to getaddrinfo code not reporting correctly the EAGAIN error.
            This made entries expire, though it does not explain why it does not reenter them.

            1 Reply Last reply Reply Quote 0
            • P
              phil.davis
              last edited by

              Upgraded to:
              2.1-BETA1 (i386)
              built on Fri Jan 18 03:21:43 EST 2013
              It puts the 11 IP address entries in the table at the start, then sometime over the next few minutes, the addresses are all deleted from the table. The problem comes from when this message is reported 11 times (site names 1 to 11):

              Jan 18 20:19:43 imp-rt-01 filterdns: Creating a new thread for host site1.dyndns-ip.com!

              It already has all 11 threads for the 11 names in the table. Then, for whatever reason, it decides to create 11 new threads. In the process, it ends up clearing out the 11 table entries and never actually putting them back.
              @ermal: I will send another full debug log.

              As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
              If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

              1 Reply Last reply Reply Quote 0
              • D
                dhatz
                last edited by

                Upgraded to today's latest snapshot, I'm still getting "exited on signal 11 (core dumped)" and I see only one filterdns process running (whereas in the past there used to be more filterdns processes – for ipsec / CP / etc)

                1 Reply Last reply Reply Quote 0
                • I
                  iamzam
                  last edited by

                  I have been following this thread because of similar problems with filterdns crash/core dumps and I have an observation:

                  My problem seems to be related to the filterdns that gets started through the vpn/ipsec stuff.

                  After updating to the latest snapshot today:

                  2.1-BETA1 (amd64)
                  built on Fri Jan 18 04:21:30 EST 2013
                  FreeBSD 8.3-RELEASE-p5

                  • I increased the filterdns debug level to 10 (in vpn.inc, line 984, '-d 10' switch) and clicked save on the VPN -> IPsec page to restart the filterdns process monitoring the vpn endpoints.

                  Here is the log output I get after this:

                  Jan 18 12:29:51 pfs check_reload_status: Syncing firewall
                  Jan 18 12:29:51 pfs filterdns: Found hostname vpn.net.loc with netmask 32.
                  Jan 18 12:29:51 pfs filterdns: found entry 10.5.0.6 for (null)
                  Jan 18 12:29:51 pfs filterdns: found entry 10.5.0.6 for (null)
                  Jan 18 12:29:51 pfs filterdns: entry 10.5.0.6 exists in table (null)
                  Jan 18 12:29:51 pfs filterdns: found entry 10.5.0.6 for (null)
                  Jan 18 12:29:51 pfs filterdns: entry 10.5.0.6 exists in table (null)
                  Jan 18 12:29:51 pfs filterdns: Found 1 entries for vpn.net.loc
                  Jan 18 12:29:51 pfs check_reload_status: Restarting ipsec tunnels
                  Jan 18 12:29:51 pfs filterdns: Ran command /usr/local/sbin/pfSctl -c "service reload ipsecdns" with exit status 0 because a dns change on hostname vpn.net.loc was detected.
                  Jan 18 12:29:53 pfs php: : IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
                  Jan 18 12:29:58 pfs php: : Could not determine VPN endpoint for 'WAN IPv4 IPsec Mobile Phase1 '
                  Jan 18 12:30:03 pfs php: : Could not determine VPN endpoint for 'WAN IPv4 IPsec Mobile Phase1 '
                  Jan 18 12:30:03 pfs filterdns: Received signal SIGHUP(1).
                  Jan 18 12:30:03 pfs kernel: pid 61925 (filterdns), uid 0: exited on signal 11 (core dumped)

                  This is probably not causing any real problems on my system because my remote vpn endpoint dns doesn't change or if it's related to the mobile ipsec phase1 not having an endpoint I am not sure how that would affect me, but I have noticed the core dump syslog messages and I have read that there can be up to three running filterdns processes (filter, vpn, captiveportal).

                  Hope this helps…

                  1 Reply Last reply Reply Quote 0
                  • E
                    eri--
                    last edited by

                    I think all this happens because a filter reload will clear the contents of the table with what the filter config sends in.
                    I changed filterdns again to force update of addresses on table when a SIGHUP happens.

                    Hopefully by monday snapshot the updated filterdns will be there.

                    1 Reply Last reply Reply Quote 0
                    • P
                      phil.davis
                      last edited by

                      2.1-BETA1 (i386)
                      built on Sat Jan 19 20:44:40 EST 2013
                      Looking good - Alix nanoBSD test system has been up 9 hours. The table that should translate 11 names to 11 IPs now has 14 IP address entries. (3 of the names have dynamically switched IP in this time.) filterdns is adding to the table and not removing old entries, but I don't really care about that (feature or bug?)

                      As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                      If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                      1 Reply Last reply Reply Quote 0
                      • B
                        bardelot
                        last edited by

                        @phil.davis:

                        2.1-BETA1 (i386)
                        built on Sat Jan 19 20:44:40 EST 2013

                        There have been a few more changes after that date, you will have to try again tomorrow or so with a newer snapshot.

                        1 Reply Last reply Reply Quote 0
                        • D
                          dhatz
                          last edited by

                          I just upgraded to latest snapshot but still get filterdns problems:

                          FreeBSD fw.localdomain 8.3-RELEASE-p5 FreeBSD 8.3-RELEASE-p5 #1: Sat Jan 19 21:12:44 EST 2013     root@snapshots-8_3-i386.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8  i386

                          MD5 (/usr/local/sbin/filterdns) = 6949816348947b7762586fe3c59b356e

                          …
                          Jan 21 00:05:28 fw kernel: pid 47308 (filterdns), uid 0: exited on signal 11 (core dumped)
                          Jan 21 00:05:29 fw check_reload_status: Restarting ipsec tunnels
                          Jan 21 00:05:30 fw login: login on ttyv0 as root
                          Jan 21 00:05:36 fw php: : IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
                          Jan 21 00:05:37 fw check_reload_status: Updating all dyndns
                          Jan 21 00:05:37 fw check_reload_status: Restarting OpenVPN tunnels/interfaces
                          Jan 21 00:05:38 fw check_reload_status: Reloading filter
                          Jan 21 00:05:40 fw kernel: pid 83611 (filterdns), uid 0: exited on signal 11 (core dumped)

                          1 Reply Last reply Reply Quote 0
                          • E
                            eri--
                            last edited by

                            dhatz that happens probably because of upgrade is not replacing the filterdns process.
                            Can you kill all you filterdns processes before running an upgrade and try again or
                            extract the archive of the upgrade and install manually the filterdns binary, it is located on usr/local/sbin iirc.

                            I am tracking even this issue of upgrade not replacing binaries at some time.

                            1 Reply Last reply Reply Quote 0
                            • D
                              dhatz
                              last edited by

                              Indeed it seems that the filterdns binary is not replaced by the upgrade process.

                              I will upgrade as soon as a new 2.1 snapshot image becomes available (currently the latest snapshot is from 19-Jan) and let you know how it goes.

                              1 Reply Last reply Reply Quote 0
                              • P
                                phil.davis
                                last edited by

                                nanobsd upgrade to 2.1-BETA1 (i386) built on Tue Jan 22 05:52:55 EST 2013 gets the version feature (1.1), but that is kind of obvious since nanoBSD is provided with a full slice. We will see what dhatz gets with a upgrade of a full install.
                                filterdns working well for me - but it does accumulate all the IP addresses known to it over time for the list of names. My table now has 15 IPs for 11 names.

                                As the Greek philosopher Isosceles used to say, "There are 3 sides to every triangle."
                                If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

                                1 Reply Last reply Reply Quote 0
                                • D
                                  dhatz
                                  last edited by

                                  @ermal:

                                  dhatz that happens probably because of upgrade is not replacing the filterdns process.
                                  Can you kill all you filterdns processes before running an upgrade and try again or
                                  I am tracking even this issue of upgrade not replacing binaries at some time.

                                  Just a quick reminder that doing an upgrade still won't replace the old filterdns binary.

                                  Btw I have tried killing all filterdns processes before running an upgrade (and verified they had been killed just before starting the upgrade procedure).

                                  1 Reply Last reply Reply Quote 0
                                  • E
                                    eri--
                                    last edited by

                                    Is the issue fixed for you dhatz?

                                    1 Reply Last reply Reply Quote 0
                                    • D
                                      dhatz
                                      last edited by

                                      Ermal, I upgraded the filterdns binary to
                                      MD5 (/usr/local/sbin/filterdns) = af355106eef6aff00d9e6653cca696eb

                                      However it seems that the new filterdns needs too much memory at system startup, causing errors like:

                                      swap_pager_getswapspace(16): failed
                                      swap_pager_getswapspace(12): failed
                                      swap_pager_getswapspace(6): failed
                                      swap_pager_getswapspace(16): failed
                                      swap_pager_getswapspace(12): failed
                                      swap_pager_getswapspace(9): failed

                                      and it dies shortly after…

                                      I've been running the latest pfsense-2.1 snap in a 256MB VM for the past ~10 months and never had this type of problem before.

                                      1 Reply Last reply Reply Quote 0
                                      • E
                                        eri--
                                        last edited by

                                        Do you have a long list of aliases in the one where you have the hostname?

                                        1 Reply Last reply Reply Quote 0
                                        • AhnHELA
                                          AhnHEL
                                          last edited by

                                          @dhatz:

                                          However it seems that the new filterdns needs too much memory at system startup, causing errors like:

                                          swap_pager_getswapspace(16): failed
                                          swap_pager_getswapspace(12): failed
                                          swap_pager_getswapspace(6): failed
                                          swap_pager_getswapspace(16): failed
                                          swap_pager_getswapspace(12): failed
                                          swap_pager_getswapspace(9): failed

                                          and it dies shortly after…

                                          I'm seeing this too on my Atom box but not on my Whitebox.  Same snap from yesterday and both amd64.  Filterdns eats up all my memory and then uses up all the swap space before it dies.

                                          AhnHEL (Angel)

                                          1 Reply Last reply Reply Quote 0
                                          • D
                                            dhatz
                                            last edited by

                                            @ermal:

                                            Do you have a long list of aliases in the one where you have the hostname?

                                            I have

                                            • two (2) aliases in fw -> aliases
                                              www_google_com
                                              www_paypal_com
                                              (note: this was just for testing)

                                            • one (1) hostname in IPsec

                                            • no "allowed hostnames" in CP

                                            In the past (until ~2 months ago) these settings resulted into two filterdns processes: one for fw-aliases and one for ipsec, none for CP.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.