Excessive "Call Home" DNS queries after update pfSense CE 2.8.0

[binfree]

This is more of an annoyance than anything. There shouldn't be any functional effect with the behavior described below.

After updating to 2.8.0 I'm seeing a lot of calling home action coming from pfSense - this is after turning OFF check for webUI updates.

Logging courtesy of AdGuard Home - pfSense is the busiest client on the network, with 10-30 queries every 60 seconds.

However often queries happened with 2.7.2, they were few enough to not be of notice in the logs when compared to other clients.

[tinfoilmatt]

@binfree Just conveniently flipped-on pfBlockerNG's DNS Reply Logging setting to confirm—and I'm not seeing that here on 2.8 CE.

Perhaps you could say a little more about your local DNS configuration, including pfSense host and where/how else services are being hosted/served on your LAN?

[johnpoz]

@binfree do you leave the webgui open? I don't see a reason why pfsense would look for those unless the gui is open to be honest.

And normally pfsense would ask itself first, which they should be cached.. for the length of the ttl - some of those are short, I show that pkg01-atx with a 300 second ttl. So seems odd that even if pfsense wanted to talk to it every minute that you see queries until the ttl expired.

Are you handing out a short ttl to pfsense?

[binfree]

@tinfoilmatt said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

Perhaps you could say a little more about your local DNS configuration, including pfSense host and where/how else services are being hosted/served on your LAN?

That's the strange thing. Same as before with 2.7.2

Upstream public DNS defined in pfSense General settings:

DHCP (Kea) hands out my AdGuard Home IP for DNS:

AdGuard Home is queried by any device on the network needing DNS and does caching. Upstreams to Unbound (running on pfSense) for all non-cached queries. No changes to TTL are made by AGH.

You can see the results being served from the cache:

Lastly, Unbound forwards to the first-mentioned public DNS when the lookups aren't private addresses.

Unbound custom options:

server:
private-domain: "myunraid.net"
private-domain: "plex.direct"
local-zone: "use-application-dns.net" always_nxdomain

@johnpoz said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

@binfree do you leave the webgui open? I don't see a reason why pfsense would look for those unless the gui is open to be honest.

No, the WebUI is generally not open unless I'm looking at something in particular, then I close the tab/window.

Is it possible that with 2.7.2 pfSense's own queries were using 127.0.0.1 directly and not going via Unbound?

Unbound listens on ports 53 and 853, so if pf previously used localhost and not Unbound for its own queries, I wouldn't see the queries recorded in AGH. Just speculation for a possible difference.

[binfree]

While writing the reply above, the queries continued to be made and logged. I made a single configuration change:

Specifically turning off the first option here:

And now it's been over 10 minutes with no DNS queries from pfSense. This is doing my head in. :)

It's 16:54 as I paste this:

[tinfoilmatt]

@binfree said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

Upstream public DNS defined in pfSense General settings:

If you're relying on the DNS Resolver (Unbound) to utilize DoT, dns.quad9.net should be used as the hostname for both of Quad9's nameservers and cloudflare-dns.com for both of Cloudflare's.

[SteveITS]

@binfree the cert expiration notice is an email from pfSense.

So, what on pfSense is pointing to Adguard for DNS? Or am I misunderstanding?

[binfree]

@SteveITS said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

@binfree the cert expiration notice is an email from pfSense.

Yes. And it should have no bearing on this issue. It's just something I wanted to turn off. And writing settings to pfSense (at this time) suddenly and unexpectedly coincided with all DNS queries coming from pfSense stopping. Problem solved - but who knows why.

So, what on pfSense is pointing to Adguard for DNS? Or am I misunderstanding?

Just the DNS address setting in DHCP. The place it needs to be set to have DHCP provide it to all clients on the network.

[binfree]

@tinfoilmatt said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

If you're relying on the DNS Resolver (Unbound) to utilize DoT, dns.quad9.net should be used as the hostname for both of Quad9's nameservers and cloudflare-dns.com for both of Cloudflare's.

Can't remember why I forgot to change the IPs for quad9. The dns11 is correct, I just forgot to change the other entries - for DNSSEC/ECS.

Followed this for Cloudflare: DNS over TLS

[SteveITS]

@binfree said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

So, what on pfSense is pointing to Adguard for DNS? Or am I misunderstanding?

Just the DNS address setting in DHCP. The place it needs to be set to have DHCP provide it to all clients on the network.

But you’re saying pfSense itself is querying AdGuard? Even though it’s not configured to use it?

[binfree]

@SteveITS said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

But you’re saying pfSense itself is querying AdGuard? Even though it’s not configured to use it?

I should probably read the documentation on this as the use of the word "or" makes the result undefined when you have multiple options configured. Not that the setting should have anything to do with the DNS spam I mentioned.

After reading the docs, pfSense should not be using the DNS server defined in DHCP - this is the only place AGH IP is defined as I recall. It should use only the resolver which uses the DNS servers listed above as it's in forwarding mode.

And if I check the resolver status, it only knows about the upstream DNS entries:

So no idea how/why pfSense was sending requests to AGH. Must be a bug somewhere.

[SteveITS]

@binfree said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

pfSense should not be using the DNS server defined in DHCP

Right, that was the gist of my point/question.

Are there any NAT forwards to AdGuard?

@binfree said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

the use of the word "or"

"Resolver or Forwarder" are the two options for providing DNS in pfSense. Resolver = unbound, Forwarder = dnsmasq. The dropdown allows, for instance, not using "itself" (127.0.0.1 is one of the two) and only using DNS from Settings > General.

You can also check Diagnostics > DNS Lookup.

[binfree]

@SteveITS said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

@binfree said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

pfSense should not be using the DNS server defined in DHCP

Are there any NAT forwards to AdGuard?

No, nothing.

@binfree said in Excessive "Call Home" DNS queries after update pfSense CE 2.8.0:

"Resolver or Forwarder" are the two options for providing DNS in pfSense.

In addition of course to "none" which uses only the upstream in General directly without Unbound or DNSMasq.

This is working now as per the documentation. I'm using Unbound in forward mode and no pfSense queries ever show up in AdGuard anymore - which is how it should always be.

You can also check Diagnostics > DNS Lookup.

Thanks. Same as the testing I did from the terminal on pfSense, which confirms 127* and the defined upstreams. Nothing indicating anywhere that it should ever go to AGH's address.

[binfree]

So here's another unexplained appearance of the pfSense IP in AdGuard:

Some 2000 queries, starting last night apparently. None are actually sourced from pfSense, they're all from my iPhone - the ones in the screenshot are most definitely from the FlipBoard news aggregation app.

In pfSense logs I see these entries with my iPhone's MAC:

I don't have a clue where 192.168.1.102 comes from. AFAIK there's nothing on my network using that range. Nothing I've ever set up anyway. The info on Apple Bonjour sleep proxy doesn't seem to cover this scenario. And I already have the system tunable net.link.ether.inet.log_arp_movements set to disabled anyway.

Then there's this other set of flip-flops which are all over the log, coming from a x86 machine running Unraid that only has a single NIC connected to the network with the MAC 60:be:b4:17:30:bc (S-Bluetech, limited). I don't know of anything on the network with the other MAC. The only MACs even starting with 54 are virtual from QEMU.

[binfree]

Uuuuuugh. On a long drive earlier today had a thought pop into my head...

Tailscale

There's a (much more) than zero chance that the iPhone traffic I saw coming from pfSense is tunneled traffic from the Tailnet - which advertises my AGH IP (10.8.8.10) as DNS. That's also likely how pfSense got the address to do its own queries - I just don't know why those would try hitting the Tailnet for that period of time when normally it just goes local/127.0.0.1

[SteveITS]

@binfree Would it be (in essence) NATting through pfSense?

[binfree]

More or less, SNAT I believe.

Tailscale offers a mesh-based Wireguard tunnel to access (for example) devices on your LAN without the bother of doing any router-side config.

The device you're using outside the LAN (my iPhone) connects directly to public sites without going through the tunnel, but since I'm advertising my own LAN-based DNS to the whole Tailnet (the mesh), and that DNS is not itself running Tailscale, connections to that DNS system are routed from another system that IS running Tailscale on the LAN. The system being used in this case is pfSense, as what Tailscale calls a subnet router, so its IP shows up as the source of the DNS query on the DNS (AGH) system.

Turns out there's a way for Tailscale to preserve the original IP, so I'm going to try that out.

And so... Guess what happens when you set AGH to drop connections from pfSense? :) Yeah, no DNS on your mobile - which I ran into today as I tried, and failed, to stream music after just starting my drive.