Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [SOLVED]egrep used lots of CPU,what did it grep ?

    General pfSense Questions
    2
    24
    1.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • insmodI
      insmod @stephenw10
      last edited by insmod

      @stephenw10 I will double it in next month maybe :)
      The cpu was poor .
      CPU: Intel(R) Atom(TM) Processor E3940 @ 1.60GHz (1593.60-MHz K8-class CPU)

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Hmm, not a powerful CPU to be sure. Still surprised that it would have a problem with 125K states...

        insmodI 1 Reply Last reply Reply Quote 0
        • insmodI
          insmod @stephenw10
          last edited by insmod

          @stephenw10 I take a look at the CPU cost,system cost 15%,interrupt cost 14%,nice cost 5%.
          I guess that the cost of system was made by pf when NAT, the cost of interrupt was made by network interface when revieved each packet, the cost of nice was made by rrdtool.
          I can only reduce the cost of nice by comment out the updaterrd.sh script except wan-traffice.rrd, but I can not find a way to reduce the cost of system and interrupt, would you like to give some suggestion ?

          Thanks.

          [23.05-RELEASE][root@PF.SH]/var/db/rrd: grep nice *
Binary file system-processor.rrd matches
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-traffic.rrd N:`/sbin/pfctl -vvsI -i igb0 | awk '\
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-packets.rrd N:`/sbin/pfctl -vvsI -i igb0 | awk '\
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-traffic.rrd N:`/sbin/pfctl -vvsI -i igb1 | awk '\
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-packets.rrd N:`/sbin/pfctl -vvsI -i igb1 | awk '\
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-traffic.rrd N:`/sbin/pfctl -vvsI -i enc0 | awk '\
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-packets.rrd N:`/sbin/pfctl -vvsI -i enc0 | awk '\
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-states.rrd N:$pfrate:$pfstates:$pfnat:$srcip:$dstip
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-processor.rrd N:${CPU}:${PROCS}
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-memory.rrd N:${MEM}
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-mbuf.rrd N:${MBUF}
updaterrd.sh:/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-sensors.rrd N:$CPU_15:$CPU_14:$CPU_13:$CPU_12:$CPU_11:$CPU_10:$CPU_9:$CPU_8:$CPU_7:$CPU_6:$CPU_5:$CPU_4:$CPU_3:$CPU_2:$CPU_1:$CPU_0
updaterrd.sh:           /usr/bin/nice -n20 /usr/local/bin/rrdtool create /var/db/rrd/$gw-quality.rrd --step 60 \
updaterrd.sh:           /usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/$gw-quality.rrd -t loss:delay:stddev N:U:U:U
updaterrd.sh:   /usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/$gw-quality.rrd -t loss:delay:stddev N:$loss:$delay:$stddev

          system.png

          interrupt.png

          nice.png

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            None of those seem very high. I'm just surprised that the egrep function was having such a hard time.
            Perhaps the the file /tmp/pfctl_ss_out was corrupted or far larger than expected?

            insmodI 1 Reply Last reply Reply Quote 0
            • insmodI
              insmod @stephenw10
              last edited by insmod

              @stephenw10 Yes,pfctl_ss_out is too large to grep.I've mounted /tmp/ as memdisk , it should be zero iowait ,just the file is too large.

              I'll add another PPPOE with the same bandwidth in the pfSense with "Multi-WAN on a Stick" solution.
              I guess the CPU cost will be double in next week.

              6days.png

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                @stephenw10 said in [SOLVED]egrep used lots of CPU,what did it grep ?:

                /tmp/pfctl_ss_out

                How big is it?

                insmodI 1 Reply Last reply Reply Quote 0
                • insmodI
                  insmod @stephenw10
                  last edited by

                  @stephenw10
                  [23.05-RELEASE][root@GW.Tel]/root: pfctl -ss |wc -l
                  166370

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Hmm, also not especially large. Is this something you noticed after upgrading to 23.05?

                    insmodI 1 Reply Last reply Reply Quote 0
                    • insmodI
                      insmod @stephenw10
                      last edited by

                      @stephenw10 It's running 23.05 now.

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Yes, but did it start showing the excess CPU usage after updating to 23.05? I.E is this new behaviour in 23.05?

                        Or perhaps this was never running anything else...

                        insmodI 1 Reply Last reply Reply Quote 0
                        • insmodI
                          insmod @stephenw10
                          last edited by insmod

                          @stephenw10 After I install the pfSense, I upgrade it to Plus at the first time ...
                          I' not sure whether pfSense CE will be the same,but I guess the /var/db/rrd/updaterrd.sh maybe the same.
                          [23.05-RELEASE][root@GW.Tel]/root: cat /var/db/rrd/updaterrd.sh

                          #!/bin/sh

export TERM=dumb

echo $$ > /var/run/updaterrd.sh.pid
counter=1
while [ "$counter" -ne 0 ]
do

# polling traffic for interface wan pppoe0 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-traffic.rrd N:`/sbin/pfctl -vvsI -i pppoe0 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface wan pppoe0 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-packets.rrd N:`/sbin/pfctl -vvsI -i pppoe0 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling traffic for interface lan igb1 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-traffic.rrd N:`/sbin/pfctl -vvsI -i igb1 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface lan igb1 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/lan-packets.rrd N:`/sbin/pfctl -vvsI -i igb1 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling traffic for interface opt1 pppoe1 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-traffic.rrd N:`/sbin/pfctl -vvsI -i pppoe1 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface opt1 pppoe1 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/opt1-packets.rrd N:`/sbin/pfctl -vvsI -i pppoe1 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling traffic for interface ipsec enc0 IPv4/IPv6 counters 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-traffic.rrd N:`/sbin/pfctl -vvsI -i enc0 | awk '\
/In4\/Pass/ { b4pi = $6 };/Out4\/Pass/ { b4po = $6 };/In4\/Block/ { b4bi = $6 };/Out4\/Block/ { b4bo = $6 };\
/In6\/Pass/ { b6pi = $6 };/Out6\/Pass/ { b6po = $6 };/In6\/Block/ { b6bi = $6 };/Out6\/Block/ { b6bo = $6 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

# polling packets for interface ipsec enc0 
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/ipsec-packets.rrd N:`/sbin/pfctl -vvsI -i enc0 | awk '\
/In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
/In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'`

#pfctl_si_out="` /sbin/pfctl -si > /tmp/pfctl_si_out `"
#pfctl_ss_out="` /sbin/pfctl -ss > /tmp/pfctl_ss_out`"
#pfrate="` cat /tmp/pfctl_si_out | egrep "inserts|removals" | awk '{ pfrate = $3 + pfrate } {print pfrate}'|tail -1 `"
#pfstates="` cat /tmp/pfctl_ss_out | egrep -v '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | wc -l|sed 's/ //g'`"
#pfnat="` cat /tmp/pfctl_ss_out | egrep '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | wc -l|sed 's/ //g' `"
#srcip="` cat /tmp/pfctl_ss_out | egrep -v '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | grep '\->' | awk '{print $3}' | awk -F: '{print $1}' | sort -u|wc -l|sed 's/ //g' `"
#dstip="` cat /tmp/pfctl_ss_out | egrep -v '\(([0-9a-f:.]|[|])+\) (\->|<\-)' | grep '<\-' | awk '{print $3}' | awk -F: '{print $1}' | sort -u|wc -l|sed 's/ //g' `"
#/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-states.rrd N:$pfrate:$pfstates:$pfnat:$srcip:$dstip

CPU=`/usr/local/sbin/cpustats | cut -f1-4 -d':'`
PROCS=`ps uxaH | wc -l | awk '{print $1;}'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-processor.rrd N:${CPU}:${PROCS}
MEM=`/sbin/sysctl -qn vm.stats.vm.v_page_count vm.stats.vm.v_active_count vm.stats.vm.v_inactive_count vm.stats.vm.v_free_count kstat.zfs.misc.arcstats.size vm.stats.vm.v_wire_count vm.stats.vm.v_user_wire_count vm.stats.vm.v_laundry_count vfs.bufspace hw.pagesize  |  /usr/bin/awk '{getline active;getline inactive;getline free;getline cache;getline wire;getline userwire;getline laundry;getline buffers;getline pagesize;cache=(cache/pagesize);buffers=(buffers/pagesize);printf ((active/$0) * 100)":"((inactive/$0) * 100)":"((free/$0) * 100)":"((cache/$0) * 100)":"((wire - (cache + buffers))/$0 * 100)":"((userwire/$0) * 100)":"((laundry/$0) * 100)":"((buffers/$0) * 100)}'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-memory.rrd N:${MEM}
MBUF=`/usr/bin/netstat -m |  /usr/bin/awk '/mbuf clusters in use/ { gsub(/\//, ":", $1); print $1; }'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-mbuf.rrd N:${MBUF}
THERMAL_TZ0=`/sbin/sysctl -qn hw.acpi.thermal.tz0.temperature | /usr/bin/sed 's/C//'`
CPU_3=`/sbin/sysctl -qn dev.cpu.3.temperature | /usr/bin/sed 's/C//'`
CPU_2=`/sbin/sysctl -qn dev.cpu.2.temperature | /usr/bin/sed 's/C//'`
CPU_1=`/sbin/sysctl -qn dev.cpu.1.temperature | /usr/bin/sed 's/C//'`
CPU_0=`/sbin/sysctl -qn dev.cpu.0.temperature | /usr/bin/sed 's/C//'`
/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/system-sensors.rrd N:$THERMAL_TZ0:$CPU_3:$CPU_2:$CPU_1:$CPU_0

# Gateway quality graphs
for sock in /var/run/dpinger_*.sock; do
        if [ ! -S "$sock" ]; then
                continue
        fi

        t=$(/usr/bin/nc -U $sock)
        if [ -z "$t" ]; then
                continue
        fi

        gw=$(echo "$t" | awk '{ print $1 }')
        delay=$(echo "$t" | awk '{ print $2 }')
        stddev=$(echo "$t" | awk '{ print $3 }')
        loss=$(echo "$t" | awk '{ print $4 }')

        if echo "$loss" | grep -Eqv '^[0-9]+$'; then
                loss="U"
        fi
        if echo "$delay" | grep -Eqv '^[0-9]+$'; then
                delay="U"
        else
                # Convert delay from microseconds to seconds
                delay=$(echo "scale=7; $delay / 1000 / 1000" | /usr/bin/bc)
        fi
        if echo "$stddev" | grep -Eqv '^[0-9]+$'; then
                stddev="U"
        else
                # Convert stddev from microseconds to seconds
                stddev=$(echo "scale=7; $stddev / 1000 / 1000" | /usr/bin/bc)
        fi

        if [ ! -f /var/db/rrd/$gw-quality.rrd ]; then
                /usr/bin/nice -n20 /usr/local/bin/rrdtool create /var/db/rrd/$gw-quality.rrd --step 60 \
                DS:loss:GAUGE:120:0:100 \
                DS:delay:GAUGE:120:0:100000 \
                DS:stddev:GAUGE:120:0:100000 \
                RRA:AVERAGE:0.5:1:1200 \
                RRA:AVERAGE:0.5:5:720 \
                RRA:AVERAGE:0.5:60:1860 \
                RRA:AVERAGE:0.5:1440:2284

                /usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/$gw-quality.rrd -t loss:delay:stddev N:U:U:U
        fi

        /usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/$gw-quality.rrd -t loss:delay:stddev N:$loss:$delay:$stddev
done
sleep 60
done
                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Ok, thanks. Let me see what I can find....

                            insmodI 1 Reply Last reply Reply Quote 0
                            • insmodI
                              insmod @stephenw10
                              last edited by

                              @stephenw10 I find after I reboot the pfSense,the /var/db/rrd/updaterrd.sh file will be restore :(

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Yes it's generated based on the data sources you have.

                                Are you able to test some other code? A patch?

                                insmodI 1 Reply Last reply Reply Quote 0
                                • insmodI
                                  insmod @stephenw10
                                  last edited by

                                  @stephenw10
                                  Yes,I can test the patch about it.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Ok, let me see what I can do here...

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Ok so replacing egrep with ripgrep seems to work fine and should be significantly faster. However I'm only able to test with a relatively small number of states.

                                      If you're able to test it install the ripgrep pkg:

                                      pkg install ripgrep

                                      Then edit /etc/inc/rrd.inc and replace the 5 instances of egrep with rg

                                      I'll keep testing that here.

                                      Steve

                                      insmodI 1 Reply Last reply Reply Quote 0
                                      • insmodI
                                        insmod @stephenw10
                                        last edited by insmod

                                        @stephenw10

                                        perl -p -i -e 's/egrep/rg/g' /var/db/rrd/updaterrd.sh /etc/inc/rrd.inc

                                        I kill and restart the updaterrd.sh.

                                        The rg still cost lots of CPU every minus.

                                        [23.05-RELEASE][root@GW.Tel]/tmp: wc -l pfctl_ss_out
                                        162234 pfctl_ss_out

                                        stat.png

                                        rg.png

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          But for a shorter time?

                                          Did he updater script get re-written? It would if you make any changes to the graph settings.

                                          insmodI 1 Reply Last reply Reply Quote 0
                                          • insmodI
                                            insmod @stephenw10
                                            last edited by

                                            @stephenw10

                                            But for a shorter time?
                                            Yes,so if I did not use netdata to monitor the system very minus,I can not find it in the "top" command.

                                            Did he updater script get re-written? It would if you make any changes to the graph settings.
                                            I did not "save view",just update graph.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post