Nightly change in free memory

[bigsy]

This is on a Netgate 2100 (32GB storage) running 23.01RC (built on Wed Feb 08 14:19:16 UTC 2023) with pfBlockerNG-devel, WG, Tailscale, IKEv2.

At 03:00 there's a drop in free memory and the dashboard memory usage shows an increase from around 20 to 39%. With the same config on 22.05 memory usage was around 20%. I rebooted the device yesterday morning around 06:00 and the memory usage again changed at 03:00.

I'm seeing a similar change on an APU2 running the same RC build and no additional packages.

login-to-view

At the time, this is what top -aSH shows.

last pid: 72928;  load averages:  0.31,  0.41,  0.38                                                                                                      up 0+23:20:04  05:18:43
273 threads:   3 running, 247 sleeping, 23 waiting
CPU:  1.2% user,  0.0% nice,  1.5% system,  0.6% interrupt, 96.7% idle
Mem: 72M Active, 428M Inact, 1197M Wired, 1606M Free
ARC: 843M Total, 310M MFU, 460M MRU, 1667K Anon, 9591K Header, 60M Other
     696M Compressed, 1566M Uncompressed, 2.25:1 Ratio

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B    32K CPU0     0  21.6H  96.79% [idle{idle: cpu0}]
   11 root        187 ki31     0B    32K RUN      1  21.7H  95.73% [idle{idle: cpu1}]
13002 unbound      20    0   213M   182M kqread   0   2:26   1.30% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    2 root        -60    -     0B    32K WAIT     0  15:27   1.10% [clock{clock (0)}]
   12 root        -60    -     0B   336K WAIT     1   0:03   0.65% [intr{swi0: uart}]
81047 root         23    0    71M    45M nanslp   0   1:29   0.61% php_wg: WireGuard service (php_wg)
13002 unbound      20    0   213M   182M kqread   1   2:38   0.57% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
69867 root         20    0    14M  4376K CPU1     1   0:00   0.55% top -aSH
42131 root         20    0    13M  3564K bpf      1   0:55   0.47% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
87904 root         20    0    13M  2792K nanslp   1   0:22   0.16% /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP6 -B  -p /var/run/dpinger_WAN_DHC
31283 dhcpd        20    0    22M  9016K select   1   0:15   0.15% /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv
   12 root        -64    -     0B   336K WAIT     0   2:42   0.15% [intr{gic0,s45: mvneta1}]
87904 root         20    0    13M  2792K sbwait   0   0:06   0.15% /usr/local/bin/dpinger -S -r 0 -i WAN_DHCP6 -B  -p /var/run/dpinger_WAN_DHC
   12 root        -64    -     0B   336K WAIT     1   5:38   0.13% [intr{gic0,s42: mvneta0}]
   12 root        -60    -     0B   336K WAIT     1   1:53   0.07% [intr{swi1: netisr 1}]
    0 root        -16    -     0B  1072K -        0   0:12   0.06% [kernel{z_wr_iss_h_0}]
   12 root        -60    -     0B   336K WAIT     0   3:24   0.06% [intr{swi1: netisr 0}]
48573 root         20    0    12M  3056K select   0   1:08   0.06% /usr/sbin/syslogd -O rfc3164 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
    7 root        -16    -     0B    16K pftm     1   1:32   0.05% [pf purge]
    0 root        -16    -     0B  1072K -        1   0:09   0.04% [kernel{z_wr_int_h_0}]
   12 root        -64    -     0B   336K WAIT     0   1:13   0.04% [intr{gic0,s27: ahci0}]
72763 root         20    0    12M  2392K kqread   0   0:33   0.04% /usr/bin/tail_pfb -n0 -F /var/log/filter.log
22356 root         20    0    23M  8968K select   0   0:30   0.03% /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient{mpd5}
    4 root        -16    -     0B    48K -        1   0:36   0.02% [cam{doneq0}]
88071 root         20    0    13M  2796K nanslp   0   0:19   0.02% /usr/local/bin/dpinger -S -r 0 -i WAN_PPPOE -B  -p /var/run/dpinger_WAN_PPPOE~7
71309 root         20    0    21M  7196K select   1   0:24   0.02% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}
26256 dhcpd        20    0    24M    12M select   0   0:14   0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid m
73019 root         20    0    77M    50M piperd   1   0:20   0.02% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    0 root        -64    -     0B  1072K -        0   0:13   0.02% [kernel{e6000sw0 taskq}]

This is the cron schedule on the 2100.

login-to-view

[stephenw10]

The bogons table maybe if you have enabled IPv6 bogons anywhere.

Check the logs, what ran at that time.

Steve

[jimp]

Bogons would just be once per month on the first. ACME runs every day and is a more likely source. I see that as well on some of mine, but not all. And rebooting then running the ACME cron by hand doesn't seem to trigger it either.

We don't do much with the built-in periodic stuff but I've also tried running that by hand and it doesn't trigger it either. One thing it does do IIRC is a daily scrub of ZFS which also seems like a top suspect.

Looking at top -aSH it's probably ZFS ARC, and if so, it's harmless. ZFS will give up wired ARC memory as needed under memory pressure.

[bigsy]

Thanks.

rc.update_bogons.sh seems to be set to run daily at 03:01 on the Netgate 2100 and on the first of the month at 03:01 on the APU2. I have not (knowingly) altered this setting on the 2100.

ACME is installed on both devices where I see this change in memory happening (Netgate 2100 and APU2E4).

Is there a known change in the behaviour of ZFS ARC in FreeBSD 14? I didn't observe this behaviour in 22.05.

[stephenw10]

Yeah, your bogons update schedule is not the default. You can set it in Sys > Adv > Networking.
The v6 bogons file can be large updating it will allocate some ram.