Netgate Firewall Freeze, WAN Interruption

daro

From time to time, about every month or two, my Netgate 4200 firewall (pfsense+ up to date 26.03.1) freezes.

Installed Packets
Installierte Pakete.png

Running Servers on SBC's

SAMBA and Syncthing
Mailserver, SOGo
Musicserver and Bepasty
FritzBox WLAN

Problemdiscription
I am new to selfhosting and started 2,5 years ago. Since almost two years I experience the loss of Internet connection on WAN. That means, I can reach the services on the servers from LAN, but I am not able to go outside (Internet) from LAN. And since all our clients are connected with WireGuard, we are not able to reach the services on our servers from outside our home. I can reach the GUI on the firewall from LAN, but I'm not able to reboot with the command reboot (diagnostics tab). I'm only able to reboot from the console (diagnostics tab) with /etc/rc.reboot. After the reboot everything works again. But in case of vacation, I'm not able to reach the firewall at all from outside. We don't need most of our services when away, but the mail server must stay connected.
Until now I wasn't able to figure out what causes this behavior. I did do a fresh install, but it didn't solve the problem. I configured a weekly reboot, but it has no effect on this problem.
And since I do not have any exotic configurations or tons of rules, why does this happened out of the blue? Why does WAN goes down until a reset? The problem is my firewall, I tested in these occasions, if the interruption could be on my ISP or whatever, but no.
And since I do not understand too much of all this stuff, I don't know what to do and need help.

Edit #1: It happened also before I installed ACME, HAProxy and Cron

stephenw10

First thing to check is what is logged in the system log when this happens?

Do you have multiple gateways configured? If you do do you have the default gateway set to the WAN?

Bob.Dig

@daro said in Netgate Firewall Freeze, WAN Interruption:

I am new to selfhosting and started 2,5 years ago.

Brand new I would say.

GeorgeCZ58

What is your version? I have multiple strange issues with netgates on 26.03.1 . In fact it is enough to make some flaps on interfaces and services starts to reboot and then fail. Stephen I think it is related.

daro

Thank you! I have one WAN Gateway, which is the default one (PORT1WAN_DHCP (default)), and the three LAN Gateways with the different servers connected to them.
My pfsense+ version is 26.03.1. I can't remember with which version the problem the first time occurred.

And, I noticed that two servers rebooted at 2 in the morning and short after this time my mail server started not being able to update maps and lua. I have FreedomBox servers and they sometimes reboot automatically.

I did try to look up the logs, but I wasn't able to see anything strange in my opinion.

stephenw10

@daro said in Netgate Firewall Freeze, WAN Interruption:

and the three LAN Gateways with the different servers connected to them.

So three LAN side routers with other subnets behind them?

Either way if you have internal gateways defined you need to set the default gateway to WAN specifically. If it's set to automatic then it may switch the default route to one of the internal gateways if the WAN sees any outage. And it will not switch back until you reboot.

daro

My installation is very simple.
Firewall with static WAN IP.
On each LAN port is a small server and on one port a simple switch for the additional access point (FritzBox WLAN).

Bildschirmfoto vom 2026-08-03 15-30-17.png

Bildschirmfoto vom 2026-08-03 15-48-25.png

When I say LAN gatewy, this is what I mean:
Bildschirmfoto vom 2026-08-03 15-29-46.png

On each LAN port I have a range for DHCP:
Bildschirmfoto vom 2026-08-03 15-40-35.png

The three servers are configured with a static internal IP like e.g. 192.168.170.45 outside of the DHCP address pool which goes from 100 - 200.

Edit: now I found the LAN Gateway
Bildschirmfoto vom 2026-08-03 16-17-27.png

daro

@stephenw10 said in Netgate Firewall Freeze, WAN Interruption:

If it's set to automatic then it may switch the default route to one of the internal gateways if the WAN sees any outage. And it will not switch back until you reboot.

Bildschirmfoto vom 2026-08-03 16-23-51.png

Should it be like this (I don't need IPv6)?

GeorgeCZ58

And what type of DHCP are you using? Kea?

daro

@GeorgeCZ58 Yes, KeaDHCP

stephenw10

@daro said in Netgate Firewall Freeze, WAN Interruption:

Should it be like this

How many gateways do you see listed there? I wouldn't expect any LAN side gateways in that setup unless the hosts on those VLANs are hypervisors with several hosted subnets perhaps.

If you only have the PORT1WAN_DHCP gateway listed there then it shouldn't matter. But setting it to PORT1WAN_DHCP specifically won't hurt.

daro

@stephenw10 This is the only one listed there. I just can select Automatic, PORT1WAN_DHCP and none.
My system is as "vanilla" as possible.

stephenw10

Ok so you actually only have one gateway. In that case it's not a bad default route causing the outage.

So I'd go back to my initial comment; what does the system log show during this event?

daro

@stephenw10

Aug 3 07:49:06 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (123 occurrences), euid 0, rgid 0, jail 0

The problem was during the night. My system logs only go back to 6:38.

Bildschirmfoto vom 2026-08-03 17-22-47.png

... or do you need something else?

I rebooted the firewall at around 9 o'clock. Before this time all the system logs look like this.

stephenw10

Ok well that shows the firewall stopped servicing DNS requests. You can see the queue exhaustions for port 53.

So check the resolver logs for that time for any errors.

Also you can show a lot more than the default 250 log lines if you use the filter options.

GeorgeCZ58

This is xactly same issue like in my case. Hurray..I am happy that I am not alone. OK. Stephen, it seems this happen only because of what happened before - seems like PHP get into troubles (I dont know why) and based on that other services starts to fail, and also unbound then..

daro

@stephenw10

Aug 2 20:00:06 	php 	96630 	NOTICE [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Aug 2 21:00:00 	php 	24113 	NOTICE [pfBlockerNG] Starting cron process.
Aug 2 21:00:03 	php 	24113 	NOTICE [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Aug 2 22:00:00 	php 	10759 	NOTICE [pfBlockerNG] Starting cron process.
Aug 2 22:00:04 	php 	10759 	NOTICE [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Aug 2 23:00:00 	php 	28075 	NOTICE [pfBlockerNG] Starting cron process.
Aug 2 23:00:07 	php 	28075 	NOTICE [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Aug 3 00:00:00 	php 	29669 	NOTICE [pfBlockerNG] Starting cron process.
Aug 3 00:00:19 	php 	29669 	NOTICE [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Aug 3 01:00:00 	php 	48444 	NOTICE [pfBlockerNG] Starting cron process.
Aug 3 01:00:05 	php 	48444 	NOTICE [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Aug 3 02:00:00 	php 	56338 	NOTICE [pfBlockerNG] Starting cron process.
Aug 3 02:00:05 	php 	56338 	NOTICE [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Aug 3 02:00:17 	check_reload_status 	657 	Linkup starting igc1
Aug 3 02:00:17 	kernel 		igc1: link state changed to DOWN
Aug 3 02:00:18 	php-fpm 	608 	NOTICE Hotplug event detected for PORT3LAN (opt1) static IP address (4: 192.168.178.1)
Aug 3 02:00:18 	php-fpm 	608 	NOTICE DEVD Ethernet detached event for opt1
Aug 3 02:00:19 	check_reload_status 	657 	Reloading filter
Aug 3 02:00:19 	check_reload_status 	657 	Reloading filter
Aug 3 02:00:19 	check_reload_status 	657 	Linkup starting igc1
Aug 3 02:00:20 	kernel 		igc1: link state changed to UP
Aug 3 02:00:20 	php-fpm 	609 	NOTICE Hotplug event detected for PORT3LAN (opt1) static IP address (4: 192.168.178.1)
Aug 3 02:00:20 	php-fpm 	609 	NOTICE DEVD Ethernet attached event for opt1
Aug 3 02:00:20 	php-fpm 	609 	NOTICE HOTPLUG: Triggering address refresh on opt1 (igc1)
Aug 3 02:00:20 	check_reload_status 	657 	rc.newwanip starting igc1
Aug 3 02:00:21 	check_reload_status 	657 	Reloading filter
Aug 3 02:00:21 	check_reload_status 	657 	Reloading filter
Aug 3 02:00:22 	php-fpm 	608 	NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6'
Aug 3 02:01:19 	check_reload_status 	657 	Linkup starting igc1
Aug 3 02:01:19 	kernel 		igc1: link state changed to DOWN
Aug 3 02:01:20 	php-fpm 	609 	NOTICE Hotplug event detected for PORT3LAN (opt1) static IP address (4: 192.168.178.1)
Aug 3 02:01:20 	php-fpm 	609 	NOTICE DEVD Ethernet detached event for opt1
Aug 3 02:01:22 	check_reload_status 	657 	Linkup starting igc1
Aug 3 02:01:22 	kernel 		igc1: link state changed to UP
Aug 3 02:25:25 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (1 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:26:29 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (103 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:27:47 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (91 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:28:50 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (82 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:29:53 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (85 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:30:56 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (118 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:31:59 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (100 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:33:02 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (87 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:34:06 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (133 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:35:19 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (192 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:36:19 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (150 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:37:19 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (134 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:38:28 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (123 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:39:30 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (125 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:40:34 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (102 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:41:36 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (83 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:42:39 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (132 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:43:42 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (119 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:44:46 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (105 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:45:49 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (77 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:46:52 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (99 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:47:55 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (114 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:48:58 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (55 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:50:01 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (62 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:51:04 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (73 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:52:09 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (50 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:53:11 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (62 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:54:15 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (51 occurrences), euid 0, rgid 0, jail 0
Aug 3 02:55:17 	kernel 		sonewconn: pcb 0xfffff8010e148a80 (0.0.0.0:53 (proto 6)): Listen queue overflow: 193 already in queue awaiting acceptance (53 occurrences), euid 0, rgid 0, jail 0

Resolver

Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 1.000000 2.000000 52
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: server stats for thread 1: 15604 queries, 5142 answers from cache, 10462 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: server stats for thread 1: requestlist max 22 avg 0.521315 exceeded 0 jostled 0
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: average recursion processing time 0.080314 sec
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: histogram of recursion processing times
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: [25%]=0.010306 median[50%]=0.0322637 [75%]=0.0992535
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: lower(secs) upper(secs) recursions
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.000000 0.000001 813
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.000512 0.001024 1
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.001024 0.002048 349
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.002048 0.004096 630
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.004096 0.008192 510
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.008192 0.016384 1180
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.016384 0.032768 1787
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.032768 0.065536 1576
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.065536 0.131072 1898
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.131072 0.262144 1101
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.262144 0.524288 411
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.524288 1.000000 110
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 1.000000 2.000000 64
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: server stats for thread 2: 11131 queries, 3358 answers from cache, 7773 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: server stats for thread 2: requestlist max 53 avg 0.598868 exceeded 0 jostled 0
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: average recursion processing time 0.084852 sec
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: histogram of recursion processing times
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: [25%]=0.0110885 median[50%]=0.0328747 [75%]=0.0998483
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: lower(secs) upper(secs) recursions
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.000000 0.000001 535
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.000256 0.000512 1
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.000512 0.001024 4
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.001024 0.002048 274
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.002048 0.004096 410
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.004096 0.008192 385
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.008192 0.016384 922
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.016384 0.032768 1335
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.032768 0.065536 1228
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.065536 0.131072 1358
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.131072 0.262144 770
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.262144 0.524288 369
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.524288 1.000000 90
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 1.000000 2.000000 59
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: server stats for thread 3: 10871 queries, 3383 answers from cache, 7488 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: server stats for thread 3: requestlist max 27 avg 0.528312 exceeded 0 jostled 0
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: average recursion processing time 0.071459 sec
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: histogram of recursion processing times
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: [25%]=0.010308 median[50%]=0.030129 [75%]=0.0824528
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: lower(secs) upper(secs) recursions
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.000000 0.000001 540
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.000512 0.001024 1
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.001024 0.002048 293
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.002048 0.004096 410
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.004096 0.008192 382
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.008192 0.016384 934
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.016384 0.032768 1400
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.032768 0.065536 1356
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.065536 0.131072 1107
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.131072 0.262144 692
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.262144 0.524288 247
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 0.524288 1.000000 66
Aug 3 00:00:09 	unbound 	61048 	[61048:0] info: 1.000000 2.000000 41
Aug 3 00:00:11 	unbound 	63737 	[63737:0] notice: init module 0: validator
Aug 3 00:00:11 	unbound 	63737 	[63737:0] notice: init module 1: iterator
Aug 3 00:00:11 	unbound 	63737 	[63737:0] info: start of service (unbound 1.25.1).
Aug 3 00:00:13 	unbound 	63737 	[63737:1] info: generate keytag query _ta-4f66-9728. NULL IN
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: service stopped (unbound 1.25.1).
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 0: 75 queries, 29 answers from cache, 46 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 0: requestlist max 1 avg 0.0434783 exceeded 0 jostled 0
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: average recursion processing time 0.133633 sec
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: histogram of recursion processing times
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: [25%]=0.007168 median[50%]=0.0846507 [75%]=0.171122
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: lower(secs) upper(secs) recursions
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.000000 0.000001 4
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.001024 0.002048 1
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.002048 0.004096 4
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.004096 0.008192 3
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.008192 0.016384 1
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.032768 0.065536 6
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.065536 0.131072 12
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.131072 0.262144 9
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.262144 0.524288 3
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.524288 1.000000 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 1: 127 queries, 45 answers from cache, 82 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 1: requestlist max 2 avg 0.0365854 exceeded 0 jostled 0
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: average recursion processing time 0.115955 sec
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: histogram of recursion processing times
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: [25%]=0.0131982 median[50%]=0.0756185 [75%]=0.127291
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: lower(secs) upper(secs) recursions
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.000000 0.000001 5
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.001024 0.002048 3
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.002048 0.004096 5
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.004096 0.008192 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.008192 0.016384 9
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.016384 0.032768 6
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.032768 0.065536 7
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.065536 0.131072 26
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.131072 0.262144 11
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.262144 0.524288 6
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.524288 1.000000 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 2: 66 queries, 25 answers from cache, 41 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 2: requestlist max 0 avg 0 exceeded 0 jostled 0
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: average recursion processing time 0.230383 sec
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: histogram of recursion processing times
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: [25%]=0.0177493 median[50%]=0.0863884 [75%]=0.251221
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: lower(secs) upper(secs) recursions
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.000000 0.000001 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.001024 0.002048 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.002048 0.004096 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.004096 0.008192 1
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.008192 0.016384 3
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.016384 0.032768 3
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.032768 0.065536 4
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.065536 0.131072 11
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.131072 0.262144 3
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.262144 0.524288 6
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 1.000000 2.000000 4
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 3: 63 queries, 36 answers from cache, 27 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: server stats for thread 3: requestlist max 0 avg 0 exceeded 0 jostled 0
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: average recursion processing time 0.190791 sec
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: histogram of recursion processing times
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: [25%]=0.006144 median[50%]=0.08192 [75%]=0.196608
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: lower(secs) upper(secs) recursions
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.000000 0.000001 3
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.001024 0.002048 1
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.002048 0.004096 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.004096 0.008192 1
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.008192 0.016384 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.016384 0.032768 1
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.032768 0.065536 2
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.065536 0.131072 4
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.131072 0.262144 7
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 0.262144 0.524288 1
Aug 3 02:00:18 	unbound 	63737 	[63737:0] info: 1.000000 2.000000 2
Aug 3 02:00:20 	unbound 	74787 	[74787:0] notice: init module 0: validator
Aug 3 02:00:20 	unbound 	74787 	[74787:0] notice: init module 1: iterator
Aug 3 02:00:20 	unbound 	74787 	[74787:0] info: start of service (unbound 1.25.1).
Aug 3 02:00:22 	unbound 	74787 	[74787:0] info: service stopped (unbound 1.25.1).
Aug 3 09:02:30 	unbound 	81729 	[81729:0] notice: init module 0: validator

It looks that around 2 in the morning something happened. Could it be related to the servers? I configured DNS queries to the address of the LAN interface like 192.168.180.1.

stephenw10

Hmm, OK. At least that seems pretty clear. Whatever is attached to PORT3 (opt1) bounced the link and that restarted Unbound. But for some reason it looks like Unbound just stopped and didn't restart.

Was unbound shown as running when you checked later? Did you reboot after that?

What is actually connected to PORT3 and was that rebooted deliberately?

daro

@stephenw10
Sorry, I didn't check if unbound was running, I just rebooted this morning and the system was running again. Port3 (igc1) is SAMBA and Syncthing running on that server.

Aug 3 19:48:08 	unbound 	65883 	[65883:0] info: service stopped (unbound 1.25.1). 
Aug 3 19:48:10 	unbound 	90461 	[90461:0] info: start of service (unbound 1.25.1).

What could trigger this problem through the server?

The reboot is a system event. FreedomBox (debian trixie) is doing this from time to time intentionally at 2 in the morning.

Mission-Ghost

@GeorgeCZ58 it's weird you have a hotplug event at 2am unless someone or something was unplugging the network cable...or it's defective and it's losing connection due to thermal or other dimensional changes.

I had a problem a couple of days ago switching my 4200 running 26.03.1 to a new core switch. The DNS resolver is pretty intolerant and did not handle the situation well and did not like the cables being disconnected and reconnected. It failed to restart as yours seems to have done.

I made the following changes which I have not tested as solving the problem, but here's hoping:

From my change log:

Changed DNS Resolver (listening) interfaces to LAN,
VLANs and localhost only, and outgoing interfaces to
WANs only to address resolver stopping during
port flapping such as when network cables are
disconnected and reconnected rapidly.

These changes were from All incoming and All outgoing interfaces selected.

But it appears you have a defective cable, cable-end or other issue moving or disconnecting PORT3LAN (opt1) in the middle of the night and/or at random times. This also needs fixing, in addition to possibly making changes like I have done.