Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0

johnpoz

@binfree if you have no firewall rules - look to the unbound ACLs by default they are auto and created to allow the segments pfsense is connected to.. do you have your manually set or maybe something is wrong there.

Does it just timeout, do you get a refused? can you from a cmd line on the adguard box do a dns query - so you can actually see if timeout or refused, etc. I take it the aguard box is on the 10.8.8 network as well?

Your saying it resolves that pfsense fqdn to 10.8.8.8 when its set for tls? but standard query to 10.8.8.8 over 53 doesn't work?

I would sniff on pfsense 10.8.8.8 interface - do you even see a query from adguard? Do you see a response sent?

binfree

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

@binfree if you have no firewall rules

No firewall rules after cleaning out the few that I do have, to make sure they're not causing it.

look to the unbound ACLs by default they are auto and created to allow the segments pfsense is connected to.. do you have your manually set or maybe something is wrong there.

Unbound -> ACL is empty

Does it just timeout, do you get a refused?

SERVFAIL reported back from AGH at a shell - timeout verified from logs.

Screenshot 2025-05-29 at 12.22.46 PM.png

2025/05/31 10:11:47.263334 ERROR response received addr=10.8.8.8:53 proto=udp status="exchanging with 10.8.8.8:53 over udp: read udp 10.8.8.10:53688->10.8.8.8:53: i/o timeout"

can you from a cmd line on the adguard box do a dns query

so you can actually see if timeout or refused, etc. I take it the aguard box is on the 10.8.8 network as well?

AGH is running in a docker container on Unraid. I can't get a shell from inside tyhe container, and from the host, if I dig and specify pfSense IP, as previous from other machines, it works.

Your saying it resolves that pfsense fqdn to 10.8.8.8 when its set for tls? but standard query to 10.8.8.8 over 53 doesn't work?

When pfSense is on 2.8.0, no nothing works. Only on 2.7.2 it all works.

johnpoz

@binfree thought your IP was 10.8.8.8 - why is your command trying to talk to 10.8.8.10 ?

That looks like a servfail trying to do a ptr lookup.. Which should but work - but do something more common like

nslookup www.google.com

servfail is not a could not connect, that is the server telling you it failed to do that lookup for some reason.

binfree

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

servfail is not a could not connect, that is the server telling you it failed to do that lookup for some reason.

Yes, you're correct. I got the details from the log and it's timing out.

nslookup or any other tool that uses DNS never works - always times out regardless of domain.

If bypassing AGH and going direct to pfSense IP, all DNS works. So with dig you can @IP to do this. DNS config on any client can be manually modified to include pfSense IP, etc.

binfree

I've since also installed PiHole and verified it too fails with pfSense 2.8.0 (as upstream), while working fine with 2.7.2

Screenshot 2025-05-31 at 11.38.25 AM.png

Screenshot 2025-05-31 at 11.39.04 AM.png

Screenshot 2025-05-31 at 11.41.39 AM.png

The PiHole docker has a working shell and I can run queries from it, unlike with AGH.

From the PiHole docker container, I can't reach pfSense 2.8.0 - timeout on ping

Screenshot 2025-05-31 at 11.56.17 AM.png

I thought it might be a docker-specific issue as I can ping from a Debian LXC, another VM, Unraid, my Mac, etc. But ping from another random docker container also seems to work as expected.

So so far, it's AdGuard Home and PiHole dockers I can't get sorted.

johnpoz

@binfree well your never going to get dns if you can not even ping.

So sniff on pfsense - while you do this.. Do you see a ping come into pfsense?

How do you have these dockers setup - because they are normally on a different network, 17.17 none of my dockers are on actual network.. You would have to setup macvlan or host network for..

example - all of my dockers are on the bridge network

ash-4.4# docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "5cf977ea258a9886865b2a88c9bf76bf9b61928a41353bd5b31fda4a415fc3d7",
        "Created": "2025-05-20T23:04:45.831584868-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }

is pfsense a physical box or some other VM?

How is it you have been in IT for 35+ years and you can't troubleshoot basic connectivity?:

binfree

@johnpoz

I've modified the AGH container and can now get to its shell.

pfSense 2.8.0 (and 2.7.2) receives icmp echos and issues replies. To any/all clients, including AGH.

The AGH machine never sees the replies from pfSense 2.8.0 while 2.7.2 works as expected.

You're assuming I haven't already done all this. And there's no need to assume that while I have (well over) 35 years IT experience, that it's ever been my day job. It's rare that I get stumped, but more rare that something seemingly as mundane as this type of update causes this kind of issue.

Containers for black holes running on macvlan network, everything's on the same subnet.

pfSense running in a VM - or bare metal, dedicated ports, as I can also swap between those different installs. No difference here.

johnpoz

@binfree said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

The AGH machine never sees the replies from pfSense 2.8.0 while 2.7.2 works as expected.

well that is nonsense... If pfsense sent the reply something not seeing it sure isn't on pfsense.. Is it sending it to the correct mac.

How could you think its something pfsense is doing - if you see pfsense put the reply on the wire?

If you have done all of this already - why wasn't it in your first post?

Your saying pfsense replies to the ping, you see it via a sniff.. But doesn't work for device X, but works fine for device Y.. Come on dude - and your saying its pfsense 2.8.. How does that make any sense at all - come on..

Change the IP of the, change the mac of your docker.. If your using macvlan change the mac of the container.

binfree

I'm very grateful for the assistance and I don't expect anyone else to do a deep dive on this nor instant replies. At the same time, I have other priorities that take time and I can't necessarily do all the testing at once.

@johnpoz said in Adguard Home can't connect to Unbound after upgrade to pfSense 2.8.0:

well that is nonsense... If pfsense sent the reply something not seeing it sure isn't on pfsense..

Here's the thing: When you have a system producing consistent results and you change one component and see different results, then change back and see the original results. And this A-B testing can be replicated at-will and repeatedly, it's likely that the changed component has the "fault." It's certainly where one should concentrate efforts looking for the cause of the issue. Too many years working in QA.

And when I use the word "fault" it's not that something is necessarily broken - it may very well be that something has been fixed, and a bug or other issue in the previous release(es) are responsible for the whole system working (for whatever reason). So I'm not discounting anything here.

Is it sending it to the correct mac.

I didn't mention this in my previous reply. As I was in the middle of something else and about to go out the door, I did a quick capture but didn't log at a level with MAC.

So now that I have the time today, I've captured with a more verbose output and sure enough... two different addresses for the AGH container showing up

Your saying pfsense replies to the ping, you see it via a sniff.. But doesn't work for device X, but works fine for device Y.. Come on dude - and your saying its pfsense 2.8.. How does that make any sense at all - come on..

But this is exactly what's happening. The container is showing the different addresses, while my MacBook Air and presumably other systems that can connect show consistent addresses.

Change the IP of the, change the mac of your docker.. If your using macvlan change the mac of the container.

Done and done previously when I tested PiHole, but I'll run captures on that now, as I didn't when I last fired up that other container.

Here's what pfSense captured running the two different versions, As you can see, the MAC discrepancy only happens when running pfSense 2.8.0. Do I have any idea why? No, I haven't a clue.

2.8.0
------------------------
AdGuard Home
    10.8.8.10 > 10.8.8.8: ICMP echo request, id 36, seq 18, length 64
10:48:59.686726 52:54:00:62:24:30 > 02:42:0a:08:08:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.10: ICMP echo reply, id 36, seq 18, length 64
10:49:00.320368 02:42:0a:08:08:0a > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 22606, offset 0, flags [DF], proto UDP (17), length 86)


MacBook Air Eth
    10.8.8.88 > 10.8.8.8: ICMP echo request, id 30894, seq 6, length 64
10:54:27.615982 52:54:00:62:24:30 > c8:4d:44:26:a2:a9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 49691, offset 0, flags [none], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.88: ICMP echo reply, id 30894, seq 6, length 64
10:54:27.785264 c8:4d:44:26:a2:a9 > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 234: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 220)

------------------------

2.7.2
------------------------
AdGuard Home
    10.8.8.10 > 10.8.8.8: ICMP echo request, id 43, seq 11, length 64
15:00:25.490315 52:54:00:62:24:30 > 02:42:0a:08:08:0a, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.10: ICMP echo reply, id 43, seq 11, length 64
15:00:26.490509 02:42:0a:08:08:0a > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 56187, offset 0, flags [DF], proto ICMP (1), length 84)

MacBook Air Eth
    10.8.8.88 > 10.8.8.8: ICMP echo request, id 30948, seq 16, length 64
14:59:17.979686 52:54:00:62:24:30 > c8:4d:44:26:a2:a9, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 57930, offset 0, flags [none], proto ICMP (1), length 84)
    10.8.8.8 > 10.8.8.88: ICMP echo reply, id 30948, seq 16, length 64
14:59:18.043717 c8:4d:44:26:a2:a9 > 52:54:00:62:24:30, ethertype IPv4 (0x0800), length 1242: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)

------------------------

binfree

I believe I've found the issue causing the weird behavior.

In looking at the MAC addresses captured, I can see that the "wrong" address is 02:42:0a:08:08:02

When using macvlan, the MAC always defaults to matching the specified IP address, but the incorrect address above doesn't.

So I then took a look at the lease table and saw this:

Screenshot 2025-06-01 at 11.13.53 AM.png

Hmm... That's not right. But it's the same wrong number. Looks like it's ignored in 2.7.2 and not ignored in 2.8.0

I made an edit to the reservation, put the correct 0A in there and... Look at that, MAC comes up to match in 2.8.0 and replies start working.

SonofA....

Now I can probably get going on the meat of this update (for me), migrating from ISC to KEA and the new if_pppoe.