[Netgate 6100] Post upgrade to 23.05.1, error:

dono

was anyone able to troubleshoot this issue?

A reboot was sufficient to resolve this for me. I did not receive any responses, so I never supplied the dumps to anyone. A few months has passed since then and I am not sure I have them around anymore, either.

However, I do have an unresolved DHCP relay issue with the newest 23.09 release. I have reverted back to 23.05.1 until I have time to investigate further.

Best regards, Ben

stephenw10

When this happens what do you see using the CPU?

Is there any particular reason you're still running 23.01?

clawsonn

@stephenw10
Hi, next occurrence what should I check for cpu usage? System activity /diag_system_activity.php and look at what is using highest percentage of WCPU or anything else?

No particular reason still on this version other than it seemed somewhat stable until this repeated issue started several months. I was going to upgrade earlier but after hitting this issue a while ago I wanted to get more info on root cause and also noticed recent announcements of changes of the PLUS licensing and I have not the time to review and determine what are ideal next steps.

Thanks for response.

stephenw10

There have been numerous issues fixed since 23.01 was released, including some in pfctl. So I would recommend upgrading to 23.09 when you can.

You can use the System Acitivity page there of run at the CLI: top -HaSP which gives slightly more output and is easier to copy/paste.
So for example:

last pid: 40231;  load averages:  0.15,  0.14,  0.14                                            up 4+13:41:29  12:56:49
315 threads:   5 running, 291 sleeping, 19 waiting
CPU 0:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 1:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 2:  0.8% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.2% idle
CPU 3:  0.0% user,  0.0% nice,  1.5% system,  0.0% interrupt, 98.5% idle
Mem: 49M Active, 717M Inact, 415M Wired, 661M Free
ARC: 168M Total, 52M MFU, 109M MRU, 1021K Header, 5668K Other
     140M Compressed, 344M Uncompressed, 2.45:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B    64K CPU1     1 106.9H 100.00% [idle{idle: cpu1}]
   11 root        187 ki31     0B    64K CPU0     0 108.0H  99.81% [idle{idle: cpu0}]
   11 root        187 ki31     0B    64K RUN      3 107.6H  99.60% [idle{idle: cpu3}]
   11 root        187 ki31     0B    64K CPU2     2 107.6H  98.76% [idle{idle: cpu2}]
87675 root         20    0   603M   494M uwait    2 186:16   0.88% /usr/local/bin/suricata -i igb1 -D -c /usr/local/etc
87675 root         20    0   603M   494M nanslp   2  24:56   0.19% /usr/local/bin/suricata -i igb1 -D -c /usr/local/etc
40231 root         20    0    14M  4184K CPU3     3   0:00   0.17% top -HaSP
    0 root        -60    -     0B  1488K -        3  11:36   0.16% [kernel{if_config_tqg_0}]
    0 root        -64    -     0B  1488K -        0  32:59   0.16% [kernel{dummynet}]
    7 root        -16    -     0B    16K pftm     0   4:49   0.03% [pf purge]

clawsonn

Hi,

Event occurred and captured cli output which appears maybe related to unbound.

For more context the machine is pfsense 23.01-RELEASE installed on physical intel machine, no vm, with all intel ix 10gbe nics, multivlan, multiwan, and mostly floating rules. There are no packages installed. Mostly vanilla plain settings and configuration. There are no config changes or reloads that occur in between reboots and the machine continues to become nonresponsive on wan lan traffic etc. Seems bizarre that the issue seems to be recurring more and more frequently.

Here follows output of CLItop -HaSP


last pid: 5420; load averages: 18.75, 14.13, 7.25 up 1+19:11:43 19:21:21  
461 threads: 23 running, 416 sleeping, 20 waiting, 2 lock  
CPU 0: 0.0% user, 0.0% nice, 7.1% system, 92.2% interrupt, 0.8% idle  
CPU 1: 0.4% user, 0.8% nice, 58.8% system, 11.4% interrupt, 28.6% idle  
CPU 2: 0.8% user, 0.4% nice, 58.4% system, 24.7% interrupt, 15.7% idle  
CPU 3: 0.0% user, 0.0% nice, 0.4% system, 99.6% interrupt, 0.0% idle  
CPU 4: 0.4% user, 1.2% nice, 60.8% system, 26.3% interrupt, 11.4% idle  
CPU 5: 0.4% user, 0.0% nice, 71.8% system, 21.2% interrupt, 6.7% idle  
CPU 6: 0.0% user, 0.0% nice, 1.6% system, 98.4% interrupt, 0.0% idle  
CPU 7: 0.0% user, 1.2% nice, 73.7% system, 7.8% interrupt, 17.3% idle  
Mem: 133M Active, 564M Inact, 959M Wired, 29G Free  
ARC: 236M Total, 87M MFU, 94M MRU, 16K Anon, 1464K Header, 53M Other  
87M Compressed, 229M Uncompressed, 2.64:1 Ratio

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND  
0 root -64 - 0B 2128K CPU4 4 17:55 87.43% [kernel{dummynet}]  
12 root -60 - 0B 320K CPU1 1 18:34 70.39% [intr{swi1: netisr 3}]  
12 root -56 - 0B 320K RUN 6 3:01 64.55% [intr{swi1: netisr 0}]  
12 root -56 - 0B 320K RUN 0 21:03 50.66% [intr{swi1: netisr 7}]  
12 root -56 - 0B 320K RUN 3 22:09 46.85% [intr{swi1: netisr 6}]  
34217 unbound 68 0 254M 196M kqread 4 2:58 45.18% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
12 root -56 - 0B 320K RUN 3 20:33 40.01% [intr{swi1: netisr 4}]  
12 root -56 - 0B 320K CPU0 0 19:01 37.83% [intr{swi1: netisr 1}]  
34217 unbound 68 0 254M 196M kqread 7 1:12 36.53% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
12 root -56 - 0B 320K CPU3 3 18:40 36.21% [intr{swi1: netisr 5}]  
12 root -56 - 0B 320K CPU6 6 24:41 33.12% [intr{swi1: netisr 2}]  
34217 unbound 68 0 254M 196M kqread 2 1:26 32.96% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
34217 unbound 68 0 254M 196M kqread 7 1:15 31.08% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
34217 unbound 68 0 254M 196M kqread 5 1:24 30.27% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
34217 unbound 95 0 254M 196M CPU5 5 1:14 28.16% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 1 42.7H 27.89% [idle{idle: cpu1}]  
34217 unbound 68 0 254M 196M kqread 7 1:24 21.94% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 7 42.6H 18.89% [idle{idle: cpu7}]  
11 root 187 ki31 0B 128K RUN 2 42.4H 15.72% [idle{idle: cpu2}]  
11 root 187 ki31 0B 128K RUN 4 42.3H 11.90% [idle{idle: cpu4}]  
11 root 187 ki31 0B 128K RUN 5 42.6H 7.04% [idle{idle: cpu5}]  
34217 unbound -16 0 254M 196M RUN 0 1:26 4.62% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
41615 root 68 20 13M 3320K wait 4 0:50 3.25% /bin/sh /var/db/rrd/updaterrd.sh  
2 root -60 - 0B 128K WAIT 0 0:42 1.49% [clock{clock (0)}]  
11 root 187 ki31 0B 128K RUN 3 42.6H 1.48% [idle{idle: cpu3}]  
2 root -60 - 0B 128K WAIT 2 0:02 1.35% [clock{clock (2)}]  
11 root 187 ki31 0B 128K RUN 6 42.3H 1.18% [idle{idle: cpu6}]  
11 root 187 ki31 0B 128K RUN 0 42.0H 0.71% [idle{idle: cpu0}]  
66272 root 20 0 18M 7216K select 7 0:04 0.45% /usr/local/sbin/openvpn --config /var/etc/openvpn/client3/config.ovpn  
77855 root 20 0 19M 8516K select 2 5:48 0.26% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid  
20990 root 20 0 13M 3240K select 6 0:27 0.24% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf  
0 root -60 - 0B 2128K - 4 3:33 0.22% [kernel{if_config_tqg_0}]  
0 root -60 - 0B 2128K - 2 16:28 0.08% [kernel{if_io_tqg_2}]  
89583 root 20 0 17M 5120K CPU2 2 0:00 0.08% top -HaSP  
0 root -60 - 0B 2128K - 6 18:25 0.08% [kernel{if_io_tqg_6}]  
0 root -60 - 0B 2128K RUN 4 19:54 0.08% [kernel{if_io_tqg_4}]  
7 root -16 - 0B 16K *pf_id 4 0:56 0.06% [pf purge]  
0 root -60 - 0B 2128K - 0 30:16 0.06% [kernel{if_io_tqg_0}]  
0 root -60 - 0B 2128K - 7 0:02 0.05% [kernel{softirq_7}]  
42158 dhcpd 20 0 25M 15M select 2 0:45 0.05% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid ix5 ix4.201 ix4.202 ix4.203 ix4.204 ix4.205 ix4.206 ix4.207  
0 root -60 - 0B 2128K RUN 1 0:00 0.05% [kernel{softirq_1}]  
0 root -60 - 0B 2128K - 2 0:02 0.05% [kernel{softirq_2}]  
0 root -60 - 0B 2128K - 5 0:02 0.05% [kernel{softirq_5}]  
0 root -60 - 0B 2128K - 3 0:02 0.04% [kernel{softirq_3}]  
0 root -60 - 0B 2128K - 6 0:02 0.04% [kernel{softirq_6}]  
0 root -60 - 0B 2128K - 0 0:00 0.03% [kernel{softirq_0}]  
0 root -60 - 0B 2128K RUN 4 0:02 0.03% [kernel{softirq_4}]  
14881 root 20 0 31M 11M kqread 4 0:35 0.02% nginx: worker process (nginx)




last pid: 10481; load averages: 17.63, 13.49, 6.79 up 1+19:11:15 19:20:53  
461 threads: 36 running, 405 sleeping, 19 waiting, 1 lock  
CPU 0: 0.0% user, 0.0% nice, 53.5% system, 46.5% interrupt, 0.0% idle  
CPU 1: 0.8% user, 0.0% nice, 65.0% system, 1.2% interrupt, 33.1% idle  
CPU 2: 0.0% user, 0.0% nice, 3.1% system, 96.9% interrupt, 0.0% idle  
CPU 3: 0.4% user, 0.0% nice, 65.0% system, 26.0% interrupt, 8.7% idle  
CPU 4: 0.4% user, 0.0% nice, 59.1% system, 34.3% interrupt, 6.3% idle  
CPU 5: 0.4% user, 0.0% nice, 7.9% system, 90.6% interrupt, 1.2% idle  
CPU 6: 0.0% user, 0.0% nice, 5.5% system, 93.3% interrupt, 1.2% idle  
CPU 7: 0.4% user, 0.0% nice, 77.2% system, 18.1% interrupt, 4.3% idle  
Mem: 123M Active, 560M Inact, 959M Wired, 29G Free  
ARC: 236M Total, 87M MFU, 94M MRU, 16K Anon, 1464K Header, 53M Other  
87M Compressed, 229M Uncompressed, 2.64:1 Ratio

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND  
12 root -56 - 0B 320K CPU2 2 24:26 94.39% [intr{swi1: netisr 2}]  
12 root -56 - 0B 320K CPU6 6 21:51 91.68% [intr{swi1: netisr 6}]  
0 root -64 - 0B 2128K CPU3 3 17:31 89.14% [kernel{dummynet}]  
12 root -56 - 0B 320K CPU5 5 18:19 77.79% [intr{swi1: netisr 5}]  
12 root -60 - 0B 320K CPU7 7 20:18 67.42% [intr{swi1: netisr 4}]  
34217 unbound 96 0 254M 196M RUN 4 1:18 51.99% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
0 root -60 - 0B 2128K - 0 30:12 51.88% [kernel{if_io_tqg_0}]  
34217 unbound 98 0 254M 196M RUN 1 1:08 51.03% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
34217 unbound 103 0 254M 196M RUN 3 1:17 41.35% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 1 42.7H 33.14% [idle{idle: cpu1}]  
34217 unbound 68 0 254M 196M RUN 1 1:21 29.39% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
12 root -60 - 0B 320K CPU4 4 2:50 28.85% [intr{swi1: netisr 0}]  
12 root -56 - 0B 320K RUN 0 18:16 25.57% [intr{swi1: netisr 3}]  
12 root -56 - 0B 320K RUN 0 18:51 11.68% [intr{swi1: netisr 1}]  
12 root -56 - 0B 320K CPU0 0 20:52 9.42% [intr{swi1: netisr 7}]  
11 root 187 ki31 0B 128K RUN 3 42.6H 8.16% [idle{idle: cpu3}]  
34217 unbound 90 0 254M 196M RUN 5 1:03 6.71% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 4 42.3H 6.21% [idle{idle: cpu4}]  
34217 unbound 105 0 254M 196M RUN 0 1:24 4.87% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 7 42.6H 4.65% [idle{idle: cpu7}]  
34217 unbound 95 0 254M 196M RUN 6 2:48 4.63% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 6 42.3H 1.13% [idle{idle: cpu6}]  
2 root -60 - 0B 128K WAIT 0 0:42 1.12% [clock{clock (0)}]  
13949 root 20 0 21M 7788K select 3 0:31 0.99% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}  
2 root -60 - 0B 128K WAIT 1 0:01 0.72% [clock{clock (1)}]  
11 root 187 ki31 0B 128K RUN 5 42.6H 0.72% [idle{idle: cpu5}]  
34217 unbound -16 0 254M 196M RUN 2 1:08 0.71% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 2 42.4H 0.22% [idle{idle: cpu2}]  
0 root -60 - 0B 2128K RUN 6 3:33 0.18% [kernel{if_config_tqg_0}]  
20990 root 20 0 13M 3240K select 1 0:27 0.15% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf  
77855 root 20 0 19M 8516K select 4 5:47 0.13% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid  
0 root -60 - 0B 2128K RUN 4 19:54 0.11% [kernel{if_io_tqg_4}]  
11 root 187 ki31 0B 128K RUN 0 42.0H 0.08% [idle{idle: cpu0}]  
89583 root 20 0 17M 5100K CPU1 1 0:00 0.07% top -HaSP  
0 root -60 - 0B 2128K - 1 0:00 0.05% [kernel{softirq_1}]  
0 root -60 - 0B 2128K - 5 0:02 0.05% [kernel{softirq_5}]  
0 root -60 - 0B 2128K RUN 3 0:02 0.04% [kernel{softirq_3}]  
0 root -60 - 0B 2128K RUN 2 16:28 0.04% [kernel{if_io_tqg_2}]  
0 root -60 - 0B 2128K RUN 4 0:02 0.04% [kernel{softirq_4}]  
0 root -60 - 0B 2128K RUN 6 0:02 0.04% [kernel{softirq_6}]  
0 root -60 - 0B 2128K RUN 6 18:25 0.03% [kernel{if_io_tqg_6}]  
0 root -60 - 0B 2128K RUN 2 0:02 0.03% [kernel{softirq_2}]  
0 root -60 - 0B 2128K RUN 7 0:02 0.03% [kernel{softirq_7}]  
0 root -60 - 0B 2128K - 0 0:00 0.02% [kernel{softirq_0}]  
42158 dhcpd 20 0 25M 15M select 3 0:45 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid ix5 ix4.201 ix4.202 ix4.203 ix4.204 ix4.205 ix4.206 ix4.207  
454 root 52 20 13M 3084K kqread 7 0:00 0.01% /usr/local/sbin/check_reload_status  
43850 dhcpd 20 0 255M 217M select 4 0:16 0.01% /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid ix5 ix4.201 ix4.202 ix4.203 ix4.204 ix4.205 ix4.206 i  
67308 root 20 0 13M 3620K bpf 3 0:20 0.01% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filt




last pid: 6954; load averages: 17.75, 13.37, 6.67 up 1+19:11:05 19:20:43  
461 threads: 30 running, 409 sleeping, 21 waiting, 1 lock  
CPU 0: 0.0% user, 0.0% nice, 12.2% system, 87.5% interrupt, 0.4% idle  
CPU 1: 0.4% user, 0.0% nice, 38.8% system, 0.0% interrupt, 60.8% idle  
CPU 2: 0.0% user, 0.0% nice, 66.3% system, 28.2% interrupt, 5.5% idle  
CPU 3: 0.0% user, 0.0% nice, 4.7% system, 94.5% interrupt, 0.8% idle  
CPU 4: 0.0% user, 0.0% nice, 51.4% system, 9.4% interrupt, 39.2% idle  
CPU 5: 0.0% user, 0.0% nice, 0.8% system, 98.8% interrupt, 0.4% idle  
CPU 6: 0.0% user, 0.0% nice, 59.6% system, 30.6% interrupt, 9.8% idle  
CPU 7: 0.0% user, 0.0% nice, 28.2% system, 68.2% interrupt, 3.5% idle  
Mem: 122M Active, 560M Inact, 959M Wired, 29G Free  
ARC: 236M Total, 87M MFU, 94M MRU, 16K Anon, 1464K Header, 53M Other  
87M Compressed, 229M Uncompressed, 2.64:1 Ratio

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND  
12 root -60 - 0B 320K WAIT 3 24:18 96.71% [intr{swi1: netisr 2}]  
0 root -64 - 0B 2128K CPU4 4 17:21 88.41% [kernel{dummynet}]  
12 root -60 - 0B 320K CPU3 3 20:10 76.63% [intr{swi1: netisr 4}]  
34217 unbound 103 0 254M 196M RUN 7 1:14 64.12% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
12 root -56 - 0B 320K CPU5 5 21:45 60.91% [intr{swi1: netisr 6}]  
11 root 187 ki31 0B 128K RUN 1 42.7H 60.50% [idle{idle: cpu1}]  
12 root -60 - 0B 320K CPU2 2 2:44 57.86% [intr{swi1: netisr 0}]  
34217 unbound 68 0 254M 196M kqread 6 1:04 45.26% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
12 root -56 - 0B 320K RUN 0 18:49 41.29% [intr{swi1: netisr 1}]  
11 root 187 ki31 0B 128K RUN 4 42.3H 41.00% [idle{idle: cpu4}]  
34217 unbound 102 0 254M 196M CPU1 1 1:21 38.06% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
12 root -56 - 0B 320K RUN 5 18:13 37.10% [intr{swi1: netisr 5}]  
12 root -56 - 0B 320K CPU0 0 18:13 23.95% [intr{swi1: netisr 3}]  
12 root -56 - 0B 320K RUN 0 20:49 20.98% [intr{swi1: netisr 7}]  
0 root -60 - 0B 2128K - 0 30:09 11.64% [kernel{if_io_tqg_0}]  
11 root 187 ki31 0B 128K RUN 6 42.3H 10.72% [idle{idle: cpu6}]  
34217 unbound 101 0 254M 196M CPU6 6 1:18 4.74% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 7 42.6H 4.39% [idle{idle: cpu7}]  
11 root 187 ki31 0B 128K RUN 2 42.4H 4.21% [idle{idle: cpu2}]  
7 root -16 - 0B 16K *pf_id 2 0:56 3.17% [pf purge]  
2 root -60 - 0B 128K WAIT 0 0:42 1.65% [clock{clock (0)}]  
11 root 187 ki31 0B 128K RUN 3 42.6H 1.59% [idle{idle: cpu3}]  
2 root -60 - 0B 128K WAIT 2 0:02 1.37% [clock{clock (2)}]  
11 root 187 ki31 0B 128K RUN 5 42.6H 0.70% [idle{idle: cpu5}]  
34217 unbound 55 0 254M 196M RUN 5 1:05 0.43% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
34217 unbound 94 0 254M 196M RUN 5 2:46 0.31% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
11 root 187 ki31 0B 128K RUN 0 42.0H 0.29% [idle{idle: cpu0}]  
34217 unbound -16 0 254M 196M RUN 0 1:16 0.29% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}  
0 root -60 - 0B 2128K - 2 3:33 0.21% [kernel{if_config_tqg_0}]  
20990 root 20 0 13M 3240K select 6 0:27 0.12% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf  
89583 root 20 0 17M 5072K CPU7 7 0:00 0.08% top -HaSP  
0 root -60 - 0B 2128K RUN 4 19:54 0.08% [kernel{if_io_tqg_4}]  
0 root -60 - 0B 2128K RUN 2 16:28 0.05% [kernel{if_io_tqg_2}]  
0 root -60 - 0B 2128K RUN 4 0:02 0.05% [kernel{softirq_4}]  
0 root -60 - 0B 2128K RUN 5 0:02 0.04% [kernel{softirq_5}]  
0 root -60 - 0B 2128K - 6 18:25 0.04% [kernel{if_io_tqg_6}]  
0 root -60 - 0B 2128K RUN 2 0:02 0.04% [kernel{softirq_2}]  
0 root -60 - 0B 2128K RUN 3 0:02 0.04% [kernel{softirq_3}]  
0 root -60 - 0B 2128K - 7 0:02 0.03% [kernel{softirq_7}]  
0 root -60 - 0B 2128K - 1 0:00 0.03% [kernel{softirq_1}]  
0 root -60 - 0B 2128K - 0 0:00 0.03% [kernel{softirq_0}]  
0 root -60 - 0B 2128K - 6 0:02 0.03% [kernel{softirq_6}]  
42158 dhcpd 20 0 25M 15M select 2 0:45 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid ix5 ix4.201 ix4.202 ix4.203 ix4.204 ix4.205 ix4.206 ix4.207  
14881 root 20 0 31M 11M kqread 2 0:35 0.01% nginx: worker process (nginx)  
13949 root 20 0 21M 7788K select 2 0:31 0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}  
43850 dhcpd 20 0 255M 217M select 2 0:16 0.01% /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid ix5 ix4.201 ix4.202 ix4.203 ix4.204 ix4.205 ix4.206 i  
20719 root 20 0 21M 10M select 6 0:00 0.01% sshd: admin@pts/0 (sshd)  
67308 root 20 0 13M 3620K bpf 4 0:20 0.01% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid

I tried to install 'System_Patches 2.2.7_2 A package to apply and maintain custom and recommended system patches' in order to search for any related unbound patches. Unfortunately the system won't allow me to install System_Patches because:

WARNING: Current pkg repository has a new PHP major  
version. pfSense should be upgraded before  
installing any new package.

Unfortunately the machine is at a distanced remote location which does not have any hands support. I normally only perform updates and patches when I visit the location and am physically onsite to mitigate unexpected upgrade issues.

Are there any steps I can perform to try to mitigate this issue remotely without attempting to perform an upgrade? I cannot perform upgrades at this time; cannot risk the machine failing to bootup or operate. Is there any method to establish ie a cron job or automatically restart unbound service repeatedly every couple hours so that I do not need to manually login and perform remote reboots?

Or if anyone recognizes the bug are there any simple steps or change of configurations that can be performed to mitigate issue? Or if system patches were made available perhaps I could apply them manually?

Anything that could bide me time.

Thank you

clawsonn

I am reviewing resolver unbound logs to see unusual activity. I do see this repeat a couple time
Nov 29 18:23:04 unbound 34217 [34217:0] notice: Restart of unbound 1.17.1.
Nov 29 18:23:04 unbound 34217 [34217:0] notice: init module 0: validator
Nov 29 18:23:04 unbound 34217 [34217:0] notice: init module 1: iterator
Nov 29 18:23:04 unbound 34217 [34217:0] info: start of service (unbound 1.17.1).
Nov 29 18:23:05 unbound 34217 [34217:3] info: generate keytag query _ta-4f66. NULL IN

Unbound settings in 'General DNS Resolver Options':
alt text

This machine has disabled Python module and disabled DHCP Registration Register DHCP leases in the DNS Resolver.

I found another recent reported bug at https://redmine.pfsense.org/issues/14980 that reads similar experience as my machine:
"experiencing recurring problems with unbound suddenly becoming unresponsive and running at 100% CPU. Restarting the unbound service brings pfSense back to normal for a while, but then in the next one to three days, unbound freaks out again at 100% CPU until we can manually intervene. "

That bug reports "A "truss" of unbound while it is locked up shows it in what appears to be an infinite loop trying to send to a UDP socket and getting back "No buffer space available" errors."
Where/how on my pfsense can I find/verify if I am hitting same bug? Where would they be seeing the '"No buffer space available" errors' and/or infinite loop?

Thank you

SteveITS

@clawsonn said in [Netgate 6100] Post upgrade to 23.05.1, error::

WARNING: Current pkg repository has a new PHP major
version. pfSense should be upgraded before
installing any new package.

Do NOT install packages for a version/OS you don’t have, per my sig. the sanity check was added to help prevent router breakage. You are lucky it stopped you. Change to Previous Stable if that’s (23.05.1) what you have.

There is a cron package for pfSense. Just install the version for your branch.

clawsonn

@SteveITS

Got it. Thank you.

Had to set to different branch in System Update to match current system
alt text

After that was set I was able to install cron package. I also installed System_Patches but don't see https://redmine.pfsense.org/issues/14980

I finished looking over DNS Resolver System logs and don't see anything out of the ordinary logged there.

I will be checking System Logs / System / General logs to see anything out of ordinary logged or if I am getting any similar messages as https://redmine.pfsense.org/issues/14980

stephenw10

You should apply the recommended system patches for 23.01 but they can only update run-time scripts not binaries like Unbound itself.

Ultimately you should upgrade when you can.

Steve

clawsonn

@stephenw10

Hi, as an update to the community I did apply several of the system patches last year and from that point forward the system stopped exhibiting the issue. Later I also updated the system to newest version and no issues have appeared.

Thank you for your support.

pfsss

@stephenw10
hi, I've encountered the same problem.
problem

my system is 2.7.2 stable.

looking for your help !

clawsonn

After updating to recent version Netgate pfSense Plus 23.09-RELEASE (amd64) there were several weeks of stability. Nothing in the mean time has been changed in the config of this PF.

Recently again the machine had similar issue and behavior showing ' SIOCGIFGROUP: Device not configured ' message again along with some other messages.

The PF machine exhibited very similar behavior again and was no longer smoothly pushing packets through, it was significantly dropping packets and the sshing into the pf over wan or accessing the webgui over wan was extremely difficult. After logging into webgui the notifications greeted with the following (date and time removed):

I also made a post in another thread because of the other error messages displayed match the OP of that thread:
https://forum.netgate.com/topic/185386/there-were-error-s-loading-the-rules-pfctl-diocaddrulenv-device-busy/18?_=1709874330173