Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0

binfree

• 2.7.2 working as expected for over 6 months.
• Upgrade to 2.8.0 and immediately AGH can't connect to pfSense DNS resolver (Unbound)
• Revert to 2.7.2 and everything goes back to working.

Setup:
pfSense set to use local DNS, Unbound listening on port 53 and 853
Upstream DNS set to Quad9 + Cloudflare
AGH set to use pfSense/Unbound as upstream DNS and private resolver
All clients DNS set via DHCP to the AGH IP

So DNS goes like this:
Client -> AGH -> Unbound/pfSense -> Quad9/Cloudflare

On 2.8.0, AGH can't connect to Unbound and testing produces the error:

Screenshot 2025-05-29 at 12.19.17 PM.png

Unbound seems to be working as verified by performing DNS queries directly on the pfSense system itself (via SSH). So nslookup using 127.0.0.1 works fine in 2.7.2 and 2.8.0

Screenshot 2025-05-29 at 1.40.46 PM.png

Likewise, if I set my Macbook to use 10.8.8.8 (pfSense IP) as DNS, that will also work as expected.

It seems to be only the Adguard Home instance that won't work with the 2.8.0 pfsense update - and I can't for the life of me figure out why not. AGH is running in docker and I can't get a shell on it to test that way. I also don't know how to get more verbose messages or logs for the connection error.

Can anyone offer some advice on where to continue trouble-shooting this? In hopes of resolving the issue or at the very least getting more detailed information to help someone on either project to make the necessary changes to fix any possible bugs.

johnpoz

@binfree have you tried from another client, and actually doing a query to pfsenses IP.. which I take it is 10.8.8.8 - which is an ODD IP to use as your gateway (pfsense IP)

use your fav dns client, nslookup, dig, host, doggo, etc. etc..

how would that tls setting work, unless your adguard could resolve what IP pfsense.xee.to resolves to? And you installed a tls cert on pfsense for pfsense.xee.to trusts?

Also what is the point of using tls for dns locally? You actually concerned with someone/thing on your local network sniffing your dns traffic?

binfree

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

@binfree have you tried from another client, and actually doing a query to pfsenses IP.. which I take it is 10.8.8.8 - which is an ODD IP to use as your gateway (pfsense IP)

Re: oddity of numbering. I've never in 35 years used ".1" for my router/gateway address, so I didn't think 2024 was the year to start. :)

Yes, 10.8.8.8 is pfSense/gateway/router IP. Any query from any client directly to 10.8.8.8 as DNS works as expected.

For example:
$ dig apple.com @10.8.8.8
$ dig apple.com @10.8.8.8 -p 53

or manually setting DNS to 10.8.8.8 and then using system(s) as normal (web browser, etc.)

This was tested from MacOS, Debian and Unraid (Slackware)

use your fav dns client, nslookup, dig, host, doggo, etc. etc..

dig and nslookup tested

how would that tls setting work, unless your adguard could resolve what IP pfsense.xee.to resolves to?

It does resolve, as 10.8.8.8 (Unbound) is used as local/private upstream and for initial resolution of any TLS host. But this is a bit of a red-herring because it doesn't have an effect on this issue. It can be safely ignored and I should have removed it from the screenshot as it's been eliminated for testing.

And you installed a tls cert on pfsense for pfsense.xee.to trusts?

Yes, every server/service on my network that needs a cert has one, issued by Let's Encrypt against a real domain/tld.

Also what is the point of using tls for dns locally?

Not much point, mainly left over from before AdGuard Home was using Unbound/pfSense for upstream and was communicating directly to outside servers.

johnpoz

@binfree well in 30+ some years I have been doing this - I have never seen a .8 used as the gateway anywhere ever. .254 sure.. I use .253 myself. middle of the segment again yeah.. I don't use .254 or .1 at home either because I fire up stuff for testing all the time - and some iot devices in their infinite wisdom like to default to those IPs but prob less likely to be a concern because I don't use the common default IP ranges anyway.. But it is habit - but in work setup .1 and .254 are the common ones.

If a client can query pfsense IP for dns - then your issue is with adguard, since to unbound running on pfsense - its just another client.

Unless you have specific firewall rules that would prevent that adguard IP from talking to 53, or you have some acl in unbound pfsense that would prevent the IP from talking to unbound as well.

binfree

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

@binfree well in 30+ some years I have been doing this - I have never seen a .8 used as the gateway anywhere ever. .254 sure.. I use .253 myself. middle of the segment again yeah..

No arguments here, it's simply a tradition and slylistic choice on my part. My LAN is /23 and DHCP only serves above .9.100 so plenty of IPs to go around.

If a client can query pfsense IP for dns - then your issue is with adguard, since to unbound running on pfsense - its just another client.

This was always my thought as well, but I can't wrap my head around anything that can be causing this from the AGH side. Swapping back and force between pf 2.7.2 and 2.8.0 has everything working and then not working - no other changes.

Unless you have specific firewall rules that would prevent that adguard IP from talking to 53, or you have some acl in unbound pfsense that would prevent the IP from talking to unbound as well.

Here too via cursory inspection makes it seem like all settings are the same between 2.7.2 and 2.8.0 after upgrade. I have both installs available to swap at will and have been rebooting each to compare. I've also removed a lot of settings, plugins, etc. from 2.8.0 to no effect.

I've posted a similar report to AGH's github, but so far it hasn't been looked at. My next test is probably going to be setting up a second AGH container along with PiHole, so I can switch between all of them looking for differences in functionality.

johnpoz

@binfree if you have no firewall rules - look to the unbound ACLs by default they are auto and created to allow the segments pfsense is connected to.. do you have your manually set or maybe something is wrong there.

Does it just timeout, do you get a refused? can you from a cmd line on the adguard box do a dns query - so you can actually see if timeout or refused, etc. I take it the aguard box is on the 10.8.8 network as well?

Your saying it resolves that pfsense fqdn to 10.8.8.8 when its set for tls? but standard query to 10.8.8.8 over 53 doesn't work?

I would sniff on pfsense 10.8.8.8 interface - do you even see a query from adguard? Do you see a response sent?

binfree

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

@binfree if you have no firewall rules

No firewall rules after cleaning out the few that I do have, to make sure they're not causing it.

look to the unbound ACLs by default they are auto and created to allow the segments pfsense is connected to.. do you have your manually set or maybe something is wrong there.

Unbound -> ACL is empty

Does it just timeout, do you get a refused?

SERVFAIL reported back from AGH at a shell - timeout verified from logs.

Screenshot 2025-05-29 at 12.22.46 PM.png

2025/05/31 10:11:47.263334 ERROR response received addr=10.8.8.8:53 proto=udp status="exchanging with 10.8.8.8:53 over udp: read udp 10.8.8.10:53688->10.8.8.8:53: i/o timeout"

can you from a cmd line on the adguard box do a dns query

so you can actually see if timeout or refused, etc. I take it the aguard box is on the 10.8.8 network as well?

AGH is running in a docker container on Unraid. I can't get a shell from inside tyhe container, and from the host, if I dig and specify pfSense IP, as previous from other machines, it works.

Your saying it resolves that pfsense fqdn to 10.8.8.8 when its set for tls? but standard query to 10.8.8.8 over 53 doesn't work?

When pfSense is on 2.8.0, no nothing works. Only on 2.7.2 it all works.

johnpoz

@binfree thought your IP was 10.8.8.8 - why is your command trying to talk to 10.8.8.10 ?

That looks like a servfail trying to do a ptr lookup.. Which should but work - but do something more common like

nslookup www.google.com

servfail is not a could not connect, that is the server telling you it failed to do that lookup for some reason.

binfree

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

servfail is not a could not connect, that is the server telling you it failed to do that lookup for some reason.

Yes, you're correct. I got the details from the log and it's timing out.

nslookup or any other tool that uses DNS never works - always times out regardless of domain.

If bypassing AGH and going direct to pfSense IP, all DNS works. So with dig you can @IP to do this. DNS config on any client can be manually modified to include pfSense IP, etc.

binfree

I've since also installed PiHole and verified it too fails with pfSense 2.8.0 (as upstream), while working fine with 2.7.2

Screenshot 2025-05-31 at 11.38.25 AM.png

Screenshot 2025-05-31 at 11.39.04 AM.png

Screenshot 2025-05-31 at 11.41.39 AM.png

The PiHole docker has a working shell and I can run queries from it, unlike with AGH.

From the PiHole docker container, I can't reach pfSense 2.8.0 - timeout on ping

Screenshot 2025-05-31 at 11.56.17 AM.png

I thought it might be a docker-specific issue as I can ping from a Debian LXC, another VM, Unraid, my Mac, etc. But ping from another random docker container also seems to work as expected.

So so far, it's AdGuard Home and PiHole dockers I can't get sorted.

johnpoz

@binfree well your never going to get dns if you can not even ping.

So sniff on pfsense - while you do this.. Do you see a ping come into pfsense?

How do you have these dockers setup - because they are normally on a different network, 17.17 none of my dockers are on actual network.. You would have to setup macvlan or host network for..

example - all of my dockers are on the bridge network

ash-4.4# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "5cf977ea258a9886865b2a88c9bf76bf9b61928a41353bd5b31fda4a415fc3d7",
        "Created": "2025-05-20T23:04:45.831584868-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }

is pfsense a physical box or some other VM?

How is it you have been in IT for 35+ years and you can't troubleshoot basic connectivity?:

binfree

@johnpoz

I've modified the AGH container and can now get to its shell.

pfSense 2.8.0 (and 2.7.2) receives icmp echos and issues replies. To any/all clients, including AGH.

The AGH machine never sees the replies from pfSense 2.8.0 while 2.7.2 works as expected.

You're assuming I haven't already done all this. And there's no need to assume that while I have (well over) 35 years IT experience, that it's ever been my day job. It's rare that I get stumped, but more rare that something seemingly as mundane as this type of update causes this kind of issue.

Containers for black holes running on macvlan network, everything's on the same subnet.

pfSense running in a VM - or bare metal, dedicated ports, as I can also swap between those different installs. No difference here.

johnpoz

@binfree said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

The AGH machine never sees the replies from pfSense 2.8.0 while 2.7.2 works as expected.

well that is nonsense... If pfsense sent the reply something not seeing it sure isn't on pfsense.. Is it sending it to the correct mac.

How could you think its something pfsense is doing - if you see pfsense put the reply on the wire?

If you have done all of this already - why wasn't it in your first post?

Your saying pfsense replies to the ping, you see it via a sniff.. But doesn't work for device X, but works fine for device Y.. Come on dude - and your saying its pfsense 2.8.. How does that make any sense at all - come on..

Change the IP of the, change the mac of your docker.. If your using macvlan change the mac of the container.

binfree

I'm very grateful for the assistance and I don't expect anyone else to do a deep dive on this nor instant replies. At the same time, I have other priorities that take time and I can't necessarily do all the testing at once.

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

well that is nonsense... If pfsense sent the reply something not seeing it sure isn't on pfsense..

Here's the thing: When you have a system producing consistent results and you change one component and see different results, then change back and see the original results. And this A-B testing can be replicated at-will and repeatedly, it's likely that the changed component has the "fault." It's certainly where one should concentrate efforts looking for the cause of the issue. Too many years working in QA.

And when I use the word "fault" it's not that something is necessarily broken - it may very well be that something has been fixed, and a bug or other issue in the previous release(es) are responsible for the whole system working (for whatever reason). So I'm not discounting anything here.

Is it sending it to the correct mac.

I didn't mention this in my previous reply. As I was in the middle of something else and about to go out the door, I did a quick capture but didn't log at a level with MAC.

So now that I have the time today, I've captured with a more verbose output and sure enough... two different addresses for the AGH container showing up

Your saying pfsense replies to the ping, you see it via a sniff.. But doesn't work for device X, but works fine for device Y.. Come on dude - and your saying its pfsense 2.8.. How does that make any sense at all - come on..

But this is exactly what's happening. The container is showing the different addresses, while my MacBook Air and presumably other systems that can connect show consistent addresses.

Change the IP of the, change the mac of your docker.. If your using macvlan change the mac of the container.

Done and done previously when I tested PiHole, but I'll run captures on that now, as I didn't when I last fired up that other container.

Here's what pfSense captured running the two different versions, As you can see, the MAC discrepancy only happens when running pfSense 2.8.0. Do I have any idea why? No, I haven't a clue.

2.8.0
------------------------
AdGuard Home
    10.8.8.10 > 10.8.8.8: ICMP echo request, id 36, seq 18, length 64
10:48:59.686726 52:54:00:62:24:30 > 02:42:0a:08:08:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.10: ICMP echo reply, id 36, seq 18, length 64
10:49:00.320368 02:42:0a:08:08:0a > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 22606, offset 0, flags [DF], proto UDP (17), length 86)


MacBook Air Eth
    10.8.8.88 > 10.8.8.8: ICMP echo request, id 30894, seq 6, length 64
10:54:27.615982 52:54:00:62:24:30 > c8:4d:44:26:a2:a9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 49691, offset 0, flags [none], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.88: ICMP echo reply, id 30894, seq 6, length 64
10:54:27.785264 c8:4d:44:26:a2:a9 > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 234: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 220)

------------------------

2.7.2
------------------------
AdGuard Home
    10.8.8.10 > 10.8.8.8: ICMP echo request, id 43, seq 11, length 64
15:00:25.490315 52:54:00:62:24:30 > 02:42:0a:08:08:0a, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.10: ICMP echo reply, id 43, seq 11, length 64
15:00:26.490509 02:42:0a:08:08:0a > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 56187, offset 0, flags [DF], proto ICMP (1), length 84)

MacBook Air Eth
    10.8.8.88 > 10.8.8.8: ICMP echo request, id 30948, seq 16, length 64
14:59:17.979686 52:54:00:62:24:30 > c8:4d:44:26:a2:a9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 57930, offset 0, flags [none], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.88: ICMP echo reply, id 30948, seq 16, length 64
14:59:18.043717 c8:4d:44:26:a2:a9 > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 1242: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)

------------------------

binfree

I believe I've found the issue causing the weird behavior.

In looking at the MAC addresses captured, I can see that the "wrong" address is 02:42:0a:08:08:02

When using macvlan, the MAC always defaults to matching the specified IP address, but the incorrect address above doesn't.

So I then took a look at the lease table and saw this:

Screenshot 2025-06-01 at 11.13.53 AM.png

Hmm... That's not right. But it's the same wrong number. Looks like it's ignored in 2.7.2 and not ignored in 2.8.0

I made an edit to the reservation, put the correct 0A in there and... Look at that, MAC comes up to match in 2.8.0 and replies start working.

SonofA....

Now I can probably get going on the meat of this update (for me), migrating from ISC to KEA and the new if_pppoe.