Unbound is still crashing, at least once daily.

mtarbox

Netgate 2100/22.05 No VLANS, two openVPN clients, pfBlockerNG-devel
Unbound has been crashing at least once daily, usually after the midnight updates, but with the occasional oddball times as well.
Prior to the 22.05 update, I did modify the pfblocker.inc file, and included the space as it was crashing far more often. Post 22.05 update, the crashes have been less.
Our system runs 24/7 as we have ambulances and crews that are out at all times.

Some of the log from last night.

Jul 13 00:00:01 php 74992 [pfBlockerNG] Starting cron process.
Jul 13 00:01:31 kernel pid 81621 (unbound), jid 0, uid 59, was killed: out of swap space
Jul 13 00:02:00 sshguard 53848 Exiting on signal.
Jul 13 00:02:00 sshguard 76348 Now monitoring attacks.
Jul 13 00:02:01 php 75169 servicewatchdog_cron.php: Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)
Jul 13 00:02:14 php 77530 notify_monitor.php: Message sent to pfsense@.org OK
Jul 13 00:06:20 kernel pid 9301 (unbound), jid 0, uid 59, was killed: out of swap space
Jul 13 00:07:01 php 84917 servicewatchdog_cron.php: Service Watchdog detected service unbound stopped. Restarting unbound (DNS Resolver)
Jul 13 00:07:14 php 86000 notify_monitor.php: Message sent to pfsense@.org OK

What else can I look at to submit to the forums for a possible solution?

I did follow the below forum, but it didn't solve my problem as I don't have DHCP mappings checked.

Re: [Unbound crashes daily]('out of swap space')

SteveITS

@mtarbox "out of swap space" indicates it is running out of memory. What is the memory usage on your router? What packages are installed besides pfBlockerNG? Do you have pfBlockerNG loading lots of feeds? Can you see what's using the memory in Diagnostics/Activity?

"Message sent to pfsense@.org" looks like an invalid email address was entered to receive notifications?

mtarbox

@steveits
the email address is modified. I get daily reports from pfSense just fine.
currently memory is at 12% of 3397mb.
edit post update is now at 30% of 3397mb.

The screen snip is from when I just forced an update.

SteveITS

@mtarbox Unless that's during the update that looks like a lot of CPU usage for pfB. Did you hear about this correction?

https://forum.netgate.com/topic/173072/high-cpu-usage-after-upgrading-to-22-05/16

That OISD list is 30.7 MB so could take a long time and a lot of memory to process.

mtarbox

@steveits said in Unbound is still crashing, at least once daily.:

https://forum.netgate.com/topic/173072/high-cpu-usage-after-upgrading-to-22-05/16

I did the modification to pfblocker.inc before I upgraded to 22.05. Of course I forgot the space, saw no changes, and quickly figured out I missed the space.

Overall, I need something to block PRON. Supposed medical "professionals" will push limits, plus we have Junior Members here.

UT1 was turning this 2100 into a slug, and I just saw OISD do the same.

SteveITS

@mtarbox said in Unbound is still crashing, at least once daily.:

I did the modification to pfblocker.inc before I upgraded to 22.05

Did you check after? I believe if you don't uninstall/reinstall the package pfSense will do so for you...

mtarbox

@steveits
$r = explode(' ', $result, 2);
looks legit.

mtarbox

Okay, I disabled OISD, enabled one of Chad Mayfields lists. I forced an update cycle and the memory is sitting at 10%, instead of 30%.
I will check the logs tomorrow morning, and see if those changes made any impact, and if Service Watchdog had to restart anything on the overnight.

@SteveITS thank you for the assistance so far. Obviously networking is not in my job title.

mtarbox

Everything is looking much better now. No email regarding unbound stopping, service watchdog restarting, etcetera.
And this was the system load this morning. Much, MUCH better!

lohphat

@mtarbox said in Unbound is still crashing, at least once daily.:

Overall, I need something to block PRON. Supposed medical "professionals" will push limits, plus we have Junior Members here.

Cadaver lab not good enough for them?

Sheesh...

Gertjan

@mtarbox said in Unbound is still crashing, at least once daily.:

service watchdog restarting, etcetera

Be careful with that one.
To keep things close to your profession : what happens when you electro choc a patients heart when it is still beating ? Right, you stop it, and thus you're making things worse.

When ever possible, stop using the "service watchdog".