[SOLVED]egrep used lots of CPU,what did it grep ?

insmod

@stephenw10 It's running 23.05 now.

stephenw10

Yes, but did it start showing the excess CPU usage after updating to 23.05? I.E is this new behaviour in 23.05?

Or perhaps this was never running anything else...

insmod

@stephenw10 After I install the pfSense, I upgrade it to Plus at the first time ...
I' not sure whether pfSense CE will be the same,but I guess the /var/db/rrd/updaterrd.sh maybe the same.
[23.05-RELEASE][root@GW.Tel]/root: cat /var/db/rrd/updaterrd.sh

#!/bin/sh

export TERM=dumb

echo $$ > /var/run/updaterrd.sh.pid
counter=1
while [ "$counter" -ne 0 ]
do

# polling traffic for interface wan pppoe0 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-traffic.rrd N:`/sbin/pfctl -vvsI -i pppoe0 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface wan pppoe0 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-packets.rrd N:`/sbin/pfctl -vvsI -i pppoe0 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling traffic for interface lan igb1 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-traffic.rrd N:`/sbin/pfctl -vvsI -i igb1 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface lan igb1 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-packets.rrd N:`/sbin/pfctl -vvsI -i igb1 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling traffic for interface opt1 pppoe1 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-traffic.rrd N:`/sbin/pfctl -vvsI -i pppoe1 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface opt1 pppoe1 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-packets.rrd N:`/sbin/pfctl -vvsI -i pppoe1 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling traffic for interface ipsec enc0 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-traffic.rrd N:`/sbin/pfctl -vvsI -i enc0 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface ipsec enc0 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-packets.rrd N:`/sbin/pfctl -vvsI -i enc0 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

#pfctl_si_out="` /sbin/pfctl -si > /tmp/pfctl_si_out `"
#pfctl_ss_out="` /sbin/pfctl -ss > /tmp/pfctl_ss_out`"
#pfrate="` cat /tmp/pfctl_si_out | egrep "inserts|removals" | awk '{ pfrate = $3 + pfrate } {print pfrate}'|tail -1 `"
#pfstates="` cat /tmp/pfctl_ss_out | egrep -v '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | wc -l|sed 's/ //g'`"
#pfnat="` cat /tmp/pfctl_ss_out | egrep '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | wc -l|sed 's/ //g' `"
#srcip="` cat /tmp/pfctl_ss_out | egrep -v '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | grep '\->' | awk '{print $3}' | awk -F: '{print $1}' | sort -u|wc -l|sed 's/ //g' `"
#dstip="` cat /tmp/pfctl_ss_out | egrep -v '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | grep '<\-' | awk '{print $3}' | awk -F: '{print $1}' | sort -u|wc -l|sed 's/ //g' `"
#/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-states.rrd N:$pfrate:$pfstates:$pfnat:$srcip:$dstip

CPU=`/usr/local/sbin/cpustats | cut -f1-4 -d':'`
PROCS=`ps uxaH | wc -l | awk '{print $1;}'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-processor.rrd N:${CPU}:${PROCS}
MEM=`/sbin/sysctl -qn vm.stats.vm.v_page_count vm.stats.vm.v_active_count vm.stats.vm.v_inactive_count vm.stats.vm.v_free_count kstat.zfs.misc.arcstats.size vm.stats.vm.v_wire_count vm.stats.vm.v_user_wire_count vm.stats.vm.v_laundry_count vfs.bufspace hw.pagesize  |  /usr/bin/awk '{getline active;getline inactive;getline free;getline cache;getline wire;getline userwire;getline laundry;getline buffers;getline pagesize;cache=(cache/pagesize);buffers=(buffers/pagesize);printf ((active/$0) * 100)":"((inactive/$0) * 100)":"((free/$0) * 100)":"((cache/$0) * 100)":"((wire - (cache + buffers))/$0 * 100)":"((userwire/$0) * 100)":"((laundry/$0) * 100)":"((buffers/$0) * 100)}'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-memory.rrd N:${MEM}
MBUF=`/usr/bin/netstat -m |  /usr/bin/awk '/mbuf clusters in use/ { gsub(/\//, ":", $1); print $1; }'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-mbuf.rrd N:${MBUF}
THERMAL_TZ0=`/sbin/sysctl -qn hw.acpi.thermal.tz0.temperature | /usr/bin/sed 's/C//'`
CPU_3=`/sbin/sysctl -qn dev.cpu.3.temperature | /usr/bin/sed 's/C//'`
CPU_2=`/sbin/sysctl -qn dev.cpu.2.temperature | /usr/bin/sed 's/C//'`
CPU_1=`/sbin/sysctl -qn dev.cpu.1.temperature | /usr/bin/sed 's/C//'`
CPU_0=`/sbin/sysctl -qn dev.cpu.0.temperature | /usr/bin/sed 's/C//'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-sensors.rrd N:$THERMAL_TZ0:$CPU_3:$CPU_2:$CPU_1:$CPU_0

# Gateway quality graphs
for sock in /var/run/dpinger_*.sock; do
        if [ ! -S "$sock" ]; then
                continue
        fi

        t=$(/usr/bin/nc -U $sock)
        if [ -z "$t" ]; then
                continue
        fi

        gw=$(echo "$t" | awk '{ print $1 }')
        delay=$(echo "$t" | awk '{ print $2 }')
        stddev=$(echo "$t" | awk '{ print $3 }')
        loss=$(echo "$t" | awk '{ print $4 }')

        if echo "$loss" | grep -Eqv '^[0-9]+$'; then
                loss="U"
        fi
        if echo "$delay" | grep -Eqv '^[0-9]+$'; then
                delay="U"
        else
                # Convert delay from microseconds to seconds
                delay=$(echo "scale=7; $delay / 1000 / 1000" | /usr/bin/bc)
        fi
        if echo "$stddev" | grep -Eqv '^[0-9]+$'; then
                stddev="U"
        else
                # Convert stddev from microseconds to seconds
                stddev=$(echo "scale=7; $stddev / 1000 / 1000" | /usr/bin/bc)
        fi

        if [ ! -f /var/db/rrd/$gw-quality.rrd ]; then
                /usr/bin/nice -n20 /usr/local/bin/rrdtool create /var/db/rrd/$gw-quality.rrd --step 60 \
                DS:loss:GAUGE:120:0:100 \
                DS:delay:GAUGE:120:0:100000 \
                DS:stddev:GAUGE:120:0:100000 \
                RRA:AVERAGE:0.5:1:1200 \
                RRA:AVERAGE:0.5:5:720 \
                RRA:AVERAGE:0.5:60:1860 \
                RRA:AVERAGE:0.5:1440:2284

                /usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/$gw-quality.rrd -t loss:delay:stddev N:U:U:U
        fi

        /usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/$gw-quality.rrd -t loss:delay:stddev N:$loss:$delay:$stddev
done
sleep 60
done

stephenw10

Ok, thanks. Let me see what I can find....

insmod

@stephenw10 I find after I reboot the pfSense,the /var/db/rrd/updaterrd.sh file will be restore :(

stephenw10

Yes it's generated based on the data sources you have.

Are you able to test some other code? A patch?

insmod

@stephenw10
Yes,I can test the patch about it.

stephenw10

Ok, let me see what I can do here...

stephenw10

Ok so replacing egrep with ripgrep seems to work fine and should be significantly faster. However I'm only able to test with a relatively small number of states.

If you're able to test it install the ripgrep pkg:

pkg install ripgrep

Then edit /etc/inc/rrd.inc and replace the 5 instances of egrep with rg

I'll keep testing that here.

Steve

insmod

@stephenw10

perl -p -i -e 's/egrep/rg/g' /var/db/rrd/updaterrd.sh /etc/inc/rrd.inc

I kill and restart the updaterrd.sh.

The rg still cost lots of CPU every minus.

[23.05-RELEASE][root@GW.Tel]/tmp: wc -l pfctl_ss_out
162234 pfctl_ss_out

stephenw10

But for a shorter time?

Did he updater script get re-written? It would if you make any changes to the graph settings.

insmod

@stephenw10

But for a shorter time?
Yes,so if I did not use netdata to monitor the system very minus,I can not find it in the "top" command.

Did he updater script get re-written? It would if you make any changes to the graph settings.
I did not "save view",just update graph.