Unbound - hangs regularly after upgrade to 2.5.1

LRS

Hello!

After upgrade to 2.5.1 (about two weeks ago). Unbound regularly stops working. I cannot stop/restart form web nor from command line. I did not found anything suspicious in logs in standard log level. I've switched logs to debug mode and there are thousands entries like:

May 11 11:40:30 SERWER110 unbound[59135]: [59135:3] debug: reuse_move_writewait_away item a12-131.akagtm.org.

Everything else looks working normally.

Where to start diagnosis?

Gertjan

@lrs said in Unbound - hangs regularly after upgrade to 2.5.1:

Where to start

Like this :

All DNS settings to default.
That includes : no more forwarding.
But Uncheck "DHCP Registration".

Btw :

How do you start or stop unbound from the command line ??
How do you know if unbound is actually running ??

LRS

All DNS settings to default.
That includes : no more forwarding.

That's production system not possible till saturday night :)

But Uncheck "DHCP Registration".

Done.

How do you start or stop unbound from the command line?

service unbound onerestart 
or
… stop/start

Both on:
– web Diagnostics → Command Line
– SSH → (magic number to get command line)

How do you know if unbound is actually running?

I know when it's not running – no domain names resolving on my personal computer :)
(PfSense IP is set as the one and only one DNS server in network configuration).
But on the web interface it shows as running.

Gertjan

@lrs said in Unbound - hangs regularly after upgrade to 2.5.1:

service unbound onerestart

Hummm. Thanks

I'm using that 'serveice' command on other OS's already all my live.
I never thought that it could work for pfSense.
I learned something here !!

I know when it's not running – no domain names resolving on my personal computer :)

That's not a solid proof.
It could be running, but ACL's exclude some networks.
Or it just doesn't listen to an interface /IP network.
Or you're forwarding from your PC to something else - but not pfSense.

I use these :

ps ax | grep 'unbound'

Check if the 'pid' is the same as the integer in unbound's pid file;

talk to it from the pfSense command line :

dig @127.0.0.1 test.com +trace

This command uses 127.0.0.1 (pfSense's localhost) - port 53 of course.

LRS

I know when it's not running – no domain names resolving on my personal computer :)

That's not a solid proof.

Well, not only on my. Rest of the machines in my network cannot too. Until restart of PfSense.

It could be running,

It cannot be stopped/restarted from command line or web interface – for sure somethings wrong…

but ACL's exclude some networks.
Or it just doesn't listen to an interface /IP network.

It should not happends during normal operation…

Or you're forwarding from your PC to something else - but not pfSense.

No… See above – all machines cannot resolve names.

Check if the 'pid' is the same as the integer in unbound's pid file;

Yes, it is.

dig @127.0.0.1 test.com +trace

Answer looks as it should, at least for my knowledge… Will try when it hangs again :)

Gertjan

@lrs said in Unbound - hangs regularly after upgrade to 2.5.1:

Will try when it hangs again

Just to be sure :

Do a dig from th command line.
Do a dig (or nslookup) from one of your PC's.

Sure thing : if your PC uses '192.168.1.1' == pfSEnse, it s using unbound.
I see from a PC :

C:\Users\gwkro>nslookup
Serveur par défaut :   pfsense.my-local-mess.net
Address:  2001:470:dead:beef:5c0:2::1

> test.com
Serveur :   pfsense.my-local-mess.net
Address:  2001:470:dead:beef:2::1

Réponse ne faisant pas autorité :
Nom :    test.com
Address:  69.172.200.235

My default network protocols shifted from IPv4 to IPv6.
The "2001:470:dead:beef:5c0:2::1" is equivalent to '192.168.1.1' == pfSense.

LRS

Just to be sure :

Yes, …110 it's PfSense box.

Gertjan

There is a second player here :

Your "PC" (not a Windows PC for sure) uses its own DNS request collector, which is listening on 127.0.0.53 port 53
And then the DNS is forwarded to the IP of pfSense on 192.168.0.1 or 192.168.0.254 ..... why somewhere in the middle of a network like 110 ? It's possible, of course.

LRS

There is a second player here :
Your "PC" (not a Windows PC for sure) uses its own DNS request collector,
which is listening on 127.0.0.53 port 53

Yes and no, that's Ubuntu's DNS service. It's OK.

And then the DNS is forwarded to the IP of pfSense

Yes, and it works this way. But problems (when happens) are not only on my PC, they are on all network (Linux/Win/Android/etc.)

why somewhere in the middle of a network like 110

Long story, dating back to year ~1997 :)

gearhead2020

I have two SG-5100s and two SG-4860s. I did an upgrade from 2.5 to 21.02.2-RELEASE on both SG-4860s and one of the SG-5100's.

I am now seeing this same unbound DNS resolver crash issue on both SG-5100s (even the one that I did not upgrade) and one of the SG-4860s.

As for the second SG-4860 that I upgraded, apparently the upgrade to 21.02.2-RELEASE was not as successful as it initially appeared because as of this morning it will not even boot due to not being able to find a critical system file. I had rebooted this second SG-4860 several times successfully after the update, but not from being powered off. I expect to be contacting Netgate Support shortly.

LRS

Looks that my system work stable now.

The only change I've done was:

But Uncheck "DHCP Registration".

gearhead2020

@lrs Under "Services > DNS Resolver > General Settings", the checkbox next to "DHCP Registration" was already unchecked on my routers.