Unbound/DNS resolver with IPv6 unreliable finally solved

strandte

When I do the search:

find / -name access_list.conf

The name seems to be with an "s" on list -> access_lists.conf

When I search the config /usr/local/etc/unbound/unbound.conf I do not find any reference to access_lists.conf.
When I search the /var/unbound/unbound.conf I do see the reference to access_lists.conf.

What are the difference between the two config files?

I also see that access_lists.conf changes to my own rules when I check the "Disable Auto-added Access Control"

So when I use manual access control the:

access-control: ::1 allow_snoop

is not included in access_lists.conf

Maybe the access line:

access-control: ::1/128 allow

isn't evaluated since the "access-control: ::1 allow_snoop" comes first in the list in access_lists.conf when auto rules is chosen?

This should indicate that ::1 has not been forgotten, but that is not what I experience.

Gertjan

@strandte said in Unbound/DNS resolver with IPv6 unreliable finally solved:

When I do the search:

find / -name access_list.conf

All unbound related settings files are here
/var/unbound/

If you use the general 'search all' command, you might find the same file else where. These are not used.
Run the magic command :

ps aux | grep 'unbound' 
...
unbound 64814   0.0  2.8 142788 114284  -  Ss   10:04       1:25.10 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
...

Now you know where the actually unbound.conf file is, and all other files it includes, like the access_list.conf file.

@strandte said in Unbound/DNS resolver with IPv6 unreliable finally solved:

When I search the config /usr/local/etc/unbound/unbound.conf I do not find any reference to access_lists.conf.
When I search the /var/unbound/unbound.conf I do see the reference to access_lists.conf.

What are the difference between the two config files?

pfSense is based upon FreeBSD, but isn't FreeBSD.
pfSense uses FreeBSD packages, and when you install them, they can place config file somewhere under (example) /usr/local/.... but these are rarely used by pfSense.
All (most) of the processes that are sued by pfSense have their config lives kept /var/.....

About these :
access-control: ::1 allow_snoop
access-control: ::1/128 allow

I couldn't tell you what the difference is between allow_snoop and allow or why ::1/128 and ::1.
For me, these two, as 127.0.0.1, are only be used / reached by processes running on pfSense itself that need some host name to be resolved, like the pfSense package upgrade checker.

strandte

It says in the web gui what the differences are between the Allow and allow_snoop:

Allow: Allow queries from hosts within the netblock defined below.

Allow Snoop: Allow recursive and nonrecursive access from hosts within the netblock defined below. Used for cache snooping and ideally should only be configured for the administrative host.

I will start testing with the allow_snoop before or after the allow in my manual access list. Then we can see if this is the root problem.

Gertjan

@strandte

allow_ or allow_snoop, thats one thing.
But what does is mean :

access-control: ::1 allow_snoop
access-control: ::1/128 allow

as ::1 and ::1/128 are the same for me.
So, allow_snoop gets set on ::1 and then overridden by 'allow' ?

Here you can see how the access_lists.conf file gets created :
/etc/inc/unbound.inc

First, "127.0.0.1/32 allow_snoop" gets thrown in and then "::1 allow_snoop".
You and I don't chose the 'allowed_snoop' from the GUI here, it's hard coded.

Then, all your local known interfaces, and this includes
127.0.0.0/8 allow
and
::1/128 allow

and as said : these are the same for me.

Note that the 'allow' here is the one I set up here :

I wonder what happens if I delete these two lines :

strandte

It is possible to configure manually the "Allow_snoop", by choosing "Allow_snoop" under "Action" in the web gui of unbound under "Access Lists". The sequence of rules shown in the lower part of that web page are the same as the sequence of the rules in the access_lists.conf file. I'm currently testing to see if putting the snoop rule first or last has any influence on the end result, but so far I can't say that it seems to have any effect.
What I see is that after doing a change in the configuration the resolver will work for some minutes more, then be unresponsive for some minutes and then come back. I wonder if it is pfBlockers large DNSBL lists which need to be loaded before unbound can take care of resolving again?
After this down period of some minutes it again seems to be stable no matter if the snoop is first or last. The only thing I'm not able to reproduce is to make the rule in access_lists.conf 100% similar to the auto created rule:

Auto created it looks like this:

access-control: ::1 allow_snoop

but when I manually create it I can't make it in any other way than this (mask needs to be selected, and if you do not select it will be auto created):

access-control: ::1/128 allow_snoop

I guess that should be the same, if it isn't a bug which makes trouble for the auto rule?

Gertjan

@strandte said in Unbound/DNS resolver with IPv6 unreliable finally solved:

It is possible to configure manually the "Allow_snoop", by choosing "Allow_snoop" u

Noop.

I selected some random "Refuse Nonlocal" :

this creates :

access-control: 127.0.0.1/32 allow_snoop
access-control: ::1 allow_snoop
access-control: 127.0.0.0/8 allow 
access-control: 192.168.1.0/24 allow 
access-control: 192.168.2.0/24 allow 
access-control: 192.168.3.0/24 allow 
access-control: 192.168.100.0/24 allow 
access-control: 2a01:dead:beef:a6e2::/64 allow 
access-control: ::1/128 allow 
#Local
access-control: fc00::/7 refuse_non_local
access-control: fe80::/64 refuse_non_local
access-control: 10.0.0.0/24 refuse_non_local
access-control: ::ffff:0:0/96 refuse_non_local
access-control: 192.168.4.0/24 refuse_non_local
access-control: 192.168.3.0/24 refuse_non_local
access-control: 2a01:dead:beef:a6e2::/64 refuse_non_local

so everything before
#Local
didn't change.

strandte

Are you sure you have disabled the auto rules?
Services_ DNS Resolver_ Advanced Settings.png
The access_lists.conf does not look like that in my case with auto rules disabled.

Gertjan

@strandte

When I check this :

( which I don't have checked right now )

I have to create my own access list .... so more chances to f##k up.
I'm a "leave it to default" guy

tinfoilmatt

@Gertjan said in Unbound/DNS resolver with IPv6 unreliable finally solved:

I wonder what happens if I delete these two lines :

a9c12224-4af3-4fde-8015-2265b6b91de5-image.png

I would delete ::1/128 allow, and add the /128 CIDR notation to the ::1 allow_snoop entry manually—and leave 127.0.0.1/32 allow_snoop as is.

But I agree that neither may be necessary as my auto-generated /var/unbound/access_lists.conf contains only the ACLs I've defined via the webGUI. No loopback addresses are present.

strandte

This post is deleted!

Gertjan

@tinfoilmatt said in Unbound/DNS resolver with IPv6 unreliable finally solved:

127.0.0.1/128

Isn't that a 'syntax error' ?
127.0.0.1/32 is as far as it goes.

strandte

I tried to add the:

access-control: ::1/128 allow_snoop

to my manual access list over the weekend. The result was that both the primary and the secondary firewall had a unresponcive unbond service on sunday. Today I have removed the access rule above. We will see how this goes.

Does anybody know what this rule is for?

strandte

Yes, 127.0.0.1/128 is wrong, and 127.0.0.1/32 is correct, but I see that the auto rule allow 127.0.0.0/8. Is that necessary? In case it is which other IP addresses in the 127.0.0.0/8 are in use?

Gertjan

@strandte said in Unbound/DNS resolver with IPv6 unreliable finally solved:

but I see that the auto rule allow 127.0.0.0/8. Is that necessary? In case it is which other IP addresses in the 127.0.0.0/8 are in use?

127.0.0/8 is a bit large, true.

Execute for example

sockstat -4 | grep '127'

to see who is using 127.a.b.c

strandte

I can't see any othe address in the 127.0.0.0/8 used other than 127.0.0.1, so I would assume it would be ok to change out 127.0.0.0/8 with 127.0.0.1/32.

Gertjan

@strandte

Sure.
Will it make any difference ?
Not sure.

w0w

@strandte said in Unbound/DNS resolver with IPv6 unreliable finally solved:

After I setup monitoring I found out that the DNS resolver on the pfSense boxes often stopped for a while and then automatically started to respond to queries again, and that the problem seemed to be more pronounced for resolving via the IPv6 addresses of the pfSense boxes. Often the unbound stopped responding to queries done via the IPv6 address, but still responded to queries done via the IPv4 address. After a while both became unresponsive. When this was the case restarting the service made it respond to queries again, but I think it might also would have started again at some time if I had not done anything. When the service had stopped this would not be the case of cause.

I honestly don't think that the unbound control settings are related to this issue. Unless access control for unbound simply prevents its endless restarts and refreshes, which, in turn, solves one problem but clearly causes a thousand others. In fact, unbound was rock-stable for me on 24.11 and earlier. But it "broke" on the 23.05 beta because pfSense suddenly decided that now, every time it receives configuration packets (RA info) from the ISP, it needs to refresh and update all related settings, including unbound, even if no changes are detected in those settings received. When I started digging into this issue, I was surprised to see just how many requests there were to stop and restart the service — sometimes ending with it stopping and not starting again. Ideally, with proper Python module integration, everything should be much more stable, but sometimes it is not.

tinfoilmatt

@Gertjan said in Unbound/DNS resolver with IPv6 unreliable finally solved:

Isn't that a 'syntax error' ?

Yes, typo. Post edited. Thanks for pointing out.