running FRR high netstate CPU usage in pf2.5
-
i try from PF2.4.5 upgrade to PF2.5.
Running FRR BGP with multiple full tables upstream, resulting in very slow operation of the PF system and high netstate CPU usage.
Therefore I cannot use the web system management interface.
-
This post is deleted! -
-
netstat -m 82825/11555/94380 mbufs in use (current/cache/total) 6697/10013/16710/1000000 mbuf clusters in use (current/cache/total/max) 6697/10001 mbuf+clusters out of packet secondary zone in use (current/cache) 10138/1191/11329/524288 4k (page size) jumbo clusters in use (current/cache/total/max) 256/47/303/524288 9k jumbo clusters in use (current/cache/total/max) 0/0/0/81817 16k jumbo clusters in use (current/cache/total/max) 76964K/28101K/105066K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayedsystat -vmstat 1 2 users Load 0.11 1.15 1.62 Jan 23 18:21 Mem usage: 15%Phy 2%Kmem VN PAGER SWAP PAGER Mem: REAL VIRTUAL in out in out Tot Share Tot Share Free count 1 Act 1818M 37060K 4394252K 55120K 13025M pages 1 All 1822M 40964K 4492640K 123800K ioflt Interrupts Proc: cow 445 total r p d s w Csw Trp Sys Int Sof Flt zfod atkbd0 1 2 134 3K 40 11K 220 406 21 ozfod hdac0 16 %ozfod ohci0 ohci 0.6%Sys 0.2%Intr 0.8%User 0.0%Nice 98.4%Idle daefr 48 cpu0:timer | | | | | | | | | | | prcfr 45 cpu1:timer > 16 totfr 44 cpu2:timer 74 dtbuf react 88 cpu3:timer Namei Name-cache Dir-cache 340337 desvn pdwak 22 igb0:rxq0 Calls hits % hits % 1976 numvn 754 pdpgs igb0:rxq1 44 44 100 396 frevn intrn igb0:rxq3 527M wire igb0:aq Disks md0 ada0 pass0 1738M act 20 igb1:rxq0 KB/t 0.00 3.40 0.00 43828K inact 21 igb1:rxq1 tps 0 5 0 0 laund 2 igb1:rxq2 MB/s 0.00 0.02 0.00 13025M free 1 igb1:rxq3 %busy 0 0 0 112M buf igb1:aq 76 igb2:rxq0 19 igb2:rxq1 24 igb2:rxq2 8 igb2:rxq3 igb2:aq 16 igb3:rxq0 igb3:rxq1 igb3:rxq2 igb3:rxq3 igb3:aq 5 ahci0:ch1 6 re0 287 -
last pid: 66007; load averages: 0.10, 0.73, 1.37 up 0+00:40:55 18:24:19 291 threads: 5 running, 253 sleeping, 33 waiting CPU: 0.4% user, 0.0% nice, 0.7% system, 0.0% interrupt, 98.9% idle Mem: 1735M Active, 43M Inact, 530M Wired, 113M Buf, 13G Free Swap: 3881M Total, 3881M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 64K CPU1 1 26:54 99.59% [idle{idle: cpu1}] 11 root 155 ki31 0B 64K CPU2 2 27:11 99.44% [idle{idle: cpu2}] 11 root 155 ki31 0B 64K CPU0 0 27:34 99.39% [idle{idle: cpu0}] 11 root 155 ki31 0B 64K RUN 3 27:04 99.33% [idle{idle: cpu3}] 77156 root 20 0 166M 62M select 2 0:50 0.36% /usr/local/sbin/zebra -d{zebra} 78356 root 20 0 1249M 1199M select 0 0:52 0.25% /usr/local/sbin/bgpd -d -M rpki{bgpd} 76398 root 20 0 17M 7448K select 2 0:06 0.18% /usr/local/sbin/openvpn --config /var/etc/openvpn/client12/config.ovpn 0 root -76 - 0B 576K - 0 0:05 0.13% [kernel{if_io_tqg_0}] 77156 root 20 0 166M 62M select 2 0:38 0.13% /usr/local/sbin/zebra -d{zebra} 73265 root 20 0 14M 4028K CPU3 3 0:00 0.11% top -aSH 1227 root 20 0 17M 7448K select 2 0:05 0.09% /usr/local/sbin/openvpn --config /var/etc/openvpn/client3/config.ovpn 82143 root 20 0 11M 2612K select 0 0:21 0.07% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf 12 root -60 - 0B 528K WAIT 1 0:04 0.07% [intr{swi4: clock (0)}] 51296 root 20 0 17M 7452K select 3 0:03 0.06% /usr/local/sbin/openvpn --config /var/etc/openvpn/client9/config.ovpn 25540 root 20 0 17M 7452K select 1 0:02 0.06% /usr/local/sbin/openvpn --config /var/etc/openvpn/client7/config.ovpn 78912 root 20 0 19M 6348K select 1 0:22 0.06% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd} 78356 root 20 0 1249M 1199M select 1 0:02 0.06% /usr/local/sbin/bgpd -d -M rpki{bgpd} 55162 root 20 0 17M 7452K select 3 0:01 0.05% /usr/local/sbin/openvpn --config /var/etc/openvpn/client1/config.ovpn 0 root -76 - 0B 576K - 0 0:01 0.05% [kernel{if_config_tqg_0}] 70446 root 20 0 17M 7452K select 2 0:01 0.05% /usr/local/sbin/openvpn --config /var/etc/openvpn/client5/config.ovpn 7177 root 20 0 14M 3056K nanslp 2 0:08 0.04% /usr/local/sbin/pcscd{pcscd} 0 root -76 - 0B 576K - 1 0:01 0.04% [kernel{if_io_tqg_1}] 38594 ladvd 20 0 12M 3316K kqread 3 0:12 0.04% ladvd: child (ladvd) 12 root -72 - 0B 528K WAIT 3 0:13 0.03% [intr{swi1: netisr 3}] 96160 dhcpd 20 0 25M 14M select 0 0:00 0.03% /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid lagg0 60202 dhcpd 20 0 23M 11M select 2 0:00 0.03% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid lagg0 77156 root 20 0 166M 62M select 2 0:06 0.03% /usr/local/sbin/zebra -d{zebra} 20 root -16 - 0B 16K pftm 3 0:09 0.02% [pf purge] 22 root -16 - 0B 48K psleep 3 0:00 0.02% [pagedaemon{dom0}] 84739 root 20 0 20M 8976K select 3 0:00 0.01% sshd: admin@pts/0 (sshd) 12 root -92 - 0B 528K WAIT 3 0:00 0.01% [intr{irq287: re0}] 21695 root 20 0 11M 2560K nanslp 2 0:00 0.01% /usr/local/bin/dpinger -S -r 0 -i FMT2_VPNV4 -B 10.10.1.2 -p /var/run/dpinger_FMT2_VPNV4~10.10.1.2~10.10.1.1.pid -u /var/run/dpinger_FMT2_VPNV4~10.10.1.2~10.10.1.1.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 80 10.10.1.1{dpinger} -
2.5.0-DEVELOPMENT (amd64)
built on Tue Jan 26 00:05:42 EST 2021 -
when install lastest new frr show :
Crash report begins. Anonymous machine information:
amd64
12.2-STABLE
FreeBSD 12.2-STABLE f6709e98f88(devel-12) pfSenseCrash report details:
PHP Errors:
[30-Jan-2021 20:12:22 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[30-Jan-2021 20:12:23 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[30-Jan-2021 20:12:24 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[30-Jan-2021 20:12:26 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0No FreeBSD crash data found.
-
last pid: 7133; load averages: 1.68, 1.22, 1.15 up 0+02:47:37 18:47:19 89 processes: 3 running, 86 sleeping CPU: 46.2% user, 0.0% nice, 5.9% system, 0.1% interrupt, 47.8% idle Mem: 890M Active, 374M Inact, 569M Wired, 109M Buf, 13G Free Swap: 3881M Total, 3881M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1905 root 1 81 0 238M 171M CPU1 1 14:30 32.31% php-fpm 42795 root 1 52 0 242M 176M piperd 3 8:42 19.29% php-fpm 8255 root 3 20 0 14M 3056K select 0 0:27 2.44% pcscd 77609 root 7 20 0 184M 73M select 0 4:23 0.83% zebra 85921 root 4 20 0 777M 719M select 0 3:37 0.56% bgpd 53451 root 1 20 0 17M 7400K select 3 0:24 0.15% openvpn 44658 root 1 20 0 19M 6448K select 1 0:58 0.07% ntpd 26876 root 1 20 0 17M 7400K select 1 0:04 0.05% openvpn 99774 root 1 20 0 13M 3608K CPU0 0 0:00 0.05% top 22183 root 1 20 0 17M 7248K select 0 0:03 0.04% openvpn 25406 root 1 20 0 17M 7408K select 2 0:04 0.04% openvpn 50489 root 1 20 0 17M 7408K select 0 0:04 0.03% openvpn 74907 root 1 20 0 17M 7408K select 2 0:04 0.03% openvpn 77939 dhcpd 1 20 0 23M 11M select 2 0:01 0.03% dhcpd 80025 dhcpd 1 20 0 25M 14M select 0 0:01 0.03% dhcpd 64676 root 1 20 0 29M 5804K select 3 0:01 0.02% watchfrr 36366 root 5 52 0 11M 2564K uwait 1 0:00 0.01% dpinger 38341 root 5 52 0 11M 2564K uwait 2 0:00 0.01% dpinger 34030 root 5 52 0 11M 2560K uwait 1 0:00 0.01% dpinger 38149 root 5 52 0 11M 2564K uwait 1 0:00 0.01% dpinger 38668 root 5 52 0 11M 2560K uwait 3 0:00 0.01% dpinger 35315 root 5 52 0 11M 2560K uwait 0 0:00 0.01% dpinger 34408 root 5 52 0 11M 2560K uwait 1 0:00 0.01% dpinger 35646 root 5 52 0 11M 2560K uwait 2 0:00 0.01% dpinger 37666 root 5 52 0 11M 2564K uwait 1 0:00 0.01% dpinger 37451 root 5 52 0 11M 2560K uwait 0 0:00 0.01% dpinger 34900 root 5 52 0 11M 2556K uwait 2 0:00 0.01% dpinger 36202 root 5 52 0 11M 2560K uwait 3 0:00 0.01% dpinger 34372 root 5 52 0 11M 2556K uwait 1 0:00 0.01% dpinger 36737 root 5 52 0 11M 2560K uwait 2 0:00 0.01% dpinger 36948 root 5 52 0 11M 2564K uwait 3 0:00 0.01% dpinger 35836 root 5 52 0 11M 2564K uwait 3 0:00 0.01% dpinger 71222 root 1 20 0 20M 9060K select 2 0:00 0.01% sshd 80032 root 1 20 0 30M 6548K select 3 0:00 0.00% staticd 18919 root 1 20 0 21M 8368K select 3 0:00 0.00% mpd5 345 root 1 20 0 101M 25M kqread 0 0:00 0.00% php-fpm -
why php high CPU?
-
I just experienced the same one one test box.
Even if I ssh in and stop FRR, there are still some netstat processes pinning the CPU. Web UI timing out.
Let's compare notes.
Is your instance bare metal or virtualized ?
My case where I am observing this is on VMware. Normal system load is well below 0.50 - its announcing 1 IPv6 /44 and receiving only default route doing nothing else (test box).
Now load is 1.80 - 2.50.
1GB RAM is used and its starting to use swap.
I upgraded to a newer RC just now. Now on
2.5.0.r.20210210.0933 the issue is still there.I see you made a bug in redmine but it was rejected. Let's try establish a case here so we can file again.
-
i have no use virtualized, i am running frr and get a lot of full routes and a lot of upstream.
-
netstat -m 166327/957758/1124085 mbufs in use (current/cache/total) 13309/9627/22936/1000000 mbuf clusters in use (current/cache/total/max) 11901/9098 mbuf+clusters out of packet secondary zone in use (current/cache) 9635/1232/10867/524288 4k (page size) jumbo clusters in use (current/cache/total/max) 256/90/346/524288 9k jumbo clusters in use (current/cache/total/max) 0/0/0/81817 16k jumbo clusters in use (current/cache/total/max) 109051K/264431K/373483K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed -
Turns out our issues are not related.
My instance the issue is that somehow FRR 7.5 doesn't respect my prefix-list filter for inbound. I had it set to
10 permit ::/0
20 deny allto only receive default route from upstream. Since upgrading this is ignored.
show ipv6 route bgpShows the entire routing table being accepted which obviously causes issues with a lab VM that has 1GB of RAM.
OP runs on powerful bare metal so the VMware thing was a red herring.
My instance that is not effected is only receiving default route from the neighbor so the issue did not show.
-
@mfld use top check your system
last pid: 13878; load averages: 3.16, 3.03, 2.70 up 0+10:41:38 22:53:06 88 processes: 4 running, 84 sleeping CPU: 10.1% user, 0.0% nice, 65.3% system, 0.5% interrupt, 24.1% idle Mem: 1258M Active, 791M Inact, 550M Wired, 107M Buf, 12G Free Swap: 3881M Total, 3881M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 13563 root 1 91 0 51M 38M CPU0 0 0:09 99.41% netstat 13578 root 1 91 0 51M 38M CPU3 3 0:09 99.40% netstat 13362 root 1 91 0 51M 38M CPU1 1 0:09 99.33% netstat 8329 root 3 20 0 14M 3056K select 1 1:58 1.01% pcscd 4964 root 7 20 0 167M 64M select 2 17:40 0.46% zebra 30541 root 1 44 0 277M 183M piperd 2 46:24 0.31% php-fpm 41578 root 1 20 0 17M 7460K select 2 1:23 0.31% openvpn 82953 root 1 44 0 283M 189M piperd 2 27:16 0.31% php-fpm 19606 root 1 38 0 283M 189M piperd 2 67:03 0.30% php-fpm 9114 root 4 20 0 700M 649M select 0 11:29 0.19% bgpd 87133 root 1 20 0 17M 7412K select 3 0:21 0.08% openvpn 13095 root 1 20 0 13M 3568K CPU2 2 0:00 0.06% top 25488 root 1 20 0 17M 7412K select 2 0:20 0.05% openvpn 40838 root 1 20 0 19M 6620K select 3 3:57 0.04% ntpd 7345 ladvd 1 20 0 12M 3320K kqread 0 1:19 0.03% ladvd 57186 root 1 20 0 11M 2612K select 2 0:02 0.03% syslogd 70089 dhcpd 1 20 0 23M 11M select 0 0:05 0.02% dhcpd 95309 dhcpd 1 20 0 25M 14M select 3 0:05 0.02% dhcpd 78326 root 1 20 0 17M 7480K select 3 0:15 0.02% openvpn 47119 root 1 20 0 17M 7412K select 2 0:14 0.02% openvpn 26340 root 1 20 0 17M 7400K select 2 0:13 0.02% openvpn 27021 root 5 52 0 11M 2564K uwait 0 0:02 0.01% dpinger 27454 root 5 52 0 11M 2564K uwait 1 0:02 0.01% dpinger 24444 root 5 52 0 11M 2564K uwait 3 0:02 0.01% dpinger 28005 root 5 52 0 11M 2560K uwait 1 0:02 0.01% dpinger 22525 root 5 52 0 11M 2556K uwait 1 0:02 0.01% dpinger 23997 root 5 52 0 11M 2560K uwait 2 0:01 0.01% dpinger 23215 root 5 52 0 11M 2556K uwait 2 0:02 0.01% dpinger 21882 root 5 52 0 11M 2560K uwait 1 0:02 0.01% dpinger 25863 root 5 52 0 16M 2660K uwait 0 0:02 0.01% dpinger 22822 root 5 52 0 11M 2560K uwait 1 0:01 0.01% dpinger 25642 root 5 52 0 14M 2616K uwait 3 0:01 0.01% dpinger 25021 root 5 52 0 11M 2560K uwait 2 0:01 0.01% dpinger 24192 root 5 52 0 11M 2560K uwait 3 0:01 0.01% dpinger 25195 root 5 52 0 11M 2564K uwait 1 0:02 0.01% dpinger -
vmstat 1 5 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr md0 ad0 in sy cs us sy id 7 0 0 5.2G 11G 44086 0 1 1 25627 666 0 0 1270 101607 28731 42 17 41 7 0 0 5.2G 11G 38568 0 0 0 12 1095 0 0 187 42791 20167 87 13 0 7 0 0 5.1G 11G 42805 0 0 0 27389 1107 0 2 181 34174 18188 92 8 0 7 0 0 5.0G 12G 37077 0 0 0 64914 1068 0 0 197 12521 5629 91 9 0 7 0 0 5.2G 11G 80071 0 0 0 31381 1095 0 0 160 21740 5395 49 47 4 -
ps aux USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 88102 29.8 0.2 52392 39072 - R 09:49 0:02.17 /usr/bin/netstat --libxo json -nWr root 88401 28.0 0.2 52392 39072 - R 09:49 0:01.06 /usr/bin/netstat --libxo json -nWr root 87946 27.9 0.0 0 16 - RE 09:49 0:02.11 /usr/bin/netstat --libxo json -nWr root 88229 25.5 0.2 52392 39072 - R 09:49 0:01.40 /usr/bin/netstat --libxo json -nWr root 62236 18.4 1.1 274200 184236 - S 09:14 4:17.16 php-fpm: pool nginx (php-fpm) root 33221 18.1 1.2 281348 189380 - S 08:49 3:59.71 php-fpm: pool nginx (php-fpm) root 84966 15.9 1.2 278220 186348 - S 09:25 3:19.80 php-fpm: pool nginx (php-fpm) root 66697 15.5 1.2 277456 186684 - S 08:49 3:39.26 php-fpm: pool nginx (php-fpm) root 43319 13.4 1.2 283168 193596 - S 08:49 6:13.60 php-fpm: pool nginx (php-fpm) root 31932 11.8 1.2 289688 198684 - S 08:49 4:06.72 php-fpm: pool nginx (php-fpm) root 89231 11.8 1.4 312764 219892 - R 08:49 7:34.97 php-fpm: pool nginx (php-fpm) root 11 9.0 0.0 0 64 - RNL 08:49 79:14.65 [idle] root 12 1.0 0.0 0 496 - WL 08:49 2:08.37 [intr] frr 61216 0.1 0.4 168396 70232 - Ss 09:25 1:12.99 /usr/local/sbin/zebra -P0 -d root 0 0.0 0.0 0 576 - DLs 08:49 0:41.21 [kernel] root 1 0.0 0.0 9548 940 - ILs 08:49 0:00.03 /sbin/init -- root 2 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto] root 3 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 0] root 4 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 1] root 5 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 2] root 6 0.0 0.0 0 16 - DL 08:49 0:00.00 [crypto returns 3] root 7 0.0 0.0 0 32 - DL 08:49 0:04.57 [cam] root 8 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod1] root 9 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod2] root 10 0.0 0.0 0 16 - DL 08:49 0:00.00 [audit] root 13 0.0 0.0 0 64 - DL 08:49 0:00.01 [ng_queue] root 14 0.0 0.0 0 48 - DL 08:49 0:00.01 [geom] root 15 0.0 0.0 0 16 - DL 08:49 0:00.00 [sequencer 00] root 16 0.0 0.0 0 640 - DL 08:49 0:04.57 [usb] root 17 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod3] root 18 0.0 0.0 0 16 - DL 08:49 0:00.00 [soaiod4] root 19 0.0 0.0 0 16 - DL 08:49 0:00.00 [sctp_iterator] root 20 0.0 0.0 0 16 - DL 08:49 0:14.21 [pf purge] root 21 0.0 0.0 0 16 - DL 08:49 0:01.16 [rand_harvestq] root 22 0.0 0.0 0 48 - DL 08:49 0:00.87 [pagedaemon] root 23 0.0 0.0 0 16 - DL 08:49 0:00.00 [vmdaemon] root 24 0.0 0.0 0 144 - DL 08:49 0:00.38 [bufdaemon] root 25 0.0 0.0 0 16 - DL 08:49 0:00.01 [vnlru] root 26 0.0 0.0 0 16 - DL 08:49 0:00.28 [syncer] root 27 0.0 0.0 0 16 - DL 08:49 0:00.00 [ALQ Daemon] root 67 0.0 0.0 0 16 - DL 08:49 0:00.03 [md0] root 781 0.0 0.2 104136 32880 - Ss 08:49 0:00.08 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) root 820 0.0 0.0 11388 2948 - INs 08:49 0:00.03 /usr/local/sbin/check_reload_status root 822 0.0 0.0 11340 2740 - IN 08:49 0:00.00 check_reload_status: Monitoring daemon of check_reload_status (check_reload_status) root 990 0.0 0.0 10024 1484 - Ss 08:49 0:00.02 /sbin/devd -q -f /etc/pfSense-devd.conf root 10106 0.0 0.1 20056 10820 - Ss 08:50 0:00.13 sshd: admin@pts/0 (sshd) root 10706 0.0 0.3 137404 47276 - I 08:49 0:28.31 php-fpm: pool nginx (php-fpm) root 11827 0.0 0.0 11392 2804 - Ss 08:50 0:00.25 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf root 12249 0.0 0.1 21300 9724 - Ss 08:49 0:00.04 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient root 12266 0.0 0.1 19612 10512 - Is 08:49 0:00.00 /usr/sbin/sshd root 12428 0.0 0.0 10692 2244 - Is 08:50 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 12889 0.0 0.0 10716 2264 - I 08:50 0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) root 12915 0.0 0.0 10692 2248 - Is 08:50 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts root 13099 0.0 0.0 10716 2268 - I 08:50 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts (minicron) root 13406 0.0 0.0 10692 2248 - Is 08:50 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data root 13618 0.0 0.0 10716 2268 - I 08:50 0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data (minicron) root 17906 0.0 0.1 21300 9744 - Ss 08:49 0:00.04 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt3.conf -p /var/run/pppoe_opt3.pid -s ppp pppoeclient root 20903 0.0 0.1 21300 9744 - Ss 08:49 0:00.04 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt8.conf -p /var/run/pppoe_opt8.pid -s ppp pppoeclient dhcpd 24683 0.0 0.1 23156 12056 - Ss 09:15 0:00.74 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid lagg0 root 24967 0.0 0.0 17404 8048 - Ss 08:51 0:01.32 /usr/local/sbin/openvpn --config /var/etc/openvpn/client9/config.ovpn dhcpd 35020 0.0 0.1 25228 14424 - Ss 09:15 0:00.71 /usr/local/sbin/dhcpd -6 -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid lagg0 root 35428 0.0 0.0 11192 2612 - Is 08:49 0:00.01 /usr/local/sbin/dhcp6c -d -n -c /var/etc/dhcp6c_opt8.conf -p /var/run/dhcp6c_pppoe1.pid pppoe1 root 35763 0.0 0.0 10584 2212 - Is 09:15 0:00.00 /usr/local/sbin/dhcpleases6 -c /usr/local/bin/php-cgi -f /usr/local/sbin/prefixes.php -l /var/dhcpd/var/db/dhcpd6.leases root 37352 0.0 0.0 11592 3104 - IN 09:15 0:00.86 /bin/sh /var/db/rrd/updaterrd.sh root 42199 0.0 0.1 28108 8312 - Is 08:49 0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx) root 42286 0.0 0.1 28636 8772 - I 08:49 0:00.00 nginx: worker process (nginx) root 42571 0.0 0.1 30284 10092 - I 08:49 0:00.40 nginx: worker process (nginx) root 42871 0.0 0.0 11356 2728 - Ss 08:49 0:00.02 /usr/sbin/cron -s root 43071 0.0 0.0 17408 8048 - Ss 08:50 0:03.91 /usr/local/sbin/openvpn --config /var/etc/openvpn/client3/config.ovpn root 43733 0.0 0.0 19252 7352 - Ss 08:49 1:49.04 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid root 51389 0.0 0.0 11172 2576 - Ss 08:49 0:00.83 /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog root 52427 0.0 0.1 20056 10820 - Ss 09:36 0:00.11 sshd: admin@pts/1 (sshd) root 57658 0.0 0.0 11192 2612 - Is 08:49 0:00.05 /usr/local/sbin/dhcp6c -d -n -c /var/etc/dhcp6c_opt3.conf -p /var/run/dhcp6c_pppoe2.pid pppoe2 frr 62005 0.0 0.0 30584 6668 - Ss 09:25 0:00.03 /usr/local/sbin/staticd -d frr 63150 0.0 7.5 1267584 1212368 - Ss 09:25 0:38.60 /usr/local/sbin/bgpd -M rpki -d root 66958 0.0 0.0 11516 3056 - Is 09:36 0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid root 67272 0.0 0.0 10624 2220 - I 09:36 0:00.00 /bin/cat root 67288 0.0 0.0 16040 3988 - IC 09:36 0:00.00 /usr/local/libexec/sshg-parser root 67597 0.0 0.0 11936 2744 - IC 09:36 0:00.00 /usr/local/libexec/sshg-blocker root 67637 0.0 0.0 11516 3052 - I 09:36 0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid root 67827 0.0 0.0 11500 3044 - I 09:36 0:00.00 /bin/sh /usr/local/libexec/sshg-fw-pf root 70412 0.0 0.0 12188 3252 - Ss 08:49 0:00.08 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid -
last pid: 43434; load averages: 7.58, 7.44, 6.71 up 0+01:05:49 09:54:57 100 processes: 10 running, 90 sleeping CPU: 9.8% user, 0.4% nice, 89.6% system, 0.1% interrupt, 0.1% idle Mem: 2192M Active, 72M Inact, 1013M Wired, 568M Buf, 12G Free Swap: 3881M Total, 3881M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 809 root 1 86 0 51M 38M CPU1 1 0:14 57.85% netstat 98129 root 1 86 0 51M 38M RUN 2 0:16 57.55% netstat 1149 root 1 86 0 51M 38M CPU0 0 0:10 55.13% netstat 1377 root 1 86 0 51M 38M RUN 0 0:08 54.33% netstat 1029 root 1 86 0 51M 38M CPU3 3 0:12 54.30% netstat 1592 root 1 86 0 51M 38M RUN 1 0:08 53.72% netstat 1704 root 1 86 0 51M 38M RUN 2 0:08 53.21% netstat 8929 root 3 20 0 14M 3248K select 2 0:11 1.10% pcscd 61216 root 7 20 0 168M 72M select 0 1:35 0.64% zebra 37352 root 1 52 20 11M 3104K wait 0 0:01 0.48% sh 63150 root 4 20 0 832M 778M select 1 0:47 0.29% bgpd 84966 root 1 26 0 280M 191M piperd 0 4:02 0.15% php-fpm 66697 root 1 20 0 282M 193M piperd 1 4:20 0.15% php-fpm 62236 root 1 20 0 283M 196M piperd 2 4:58 0.14% php-fpm 33221 root 1 32 0 278M 189M piperd 1 4:41 0.14% php-fpm 31932 root 1 32 0 278M 189M piperd 0 4:49 0.14% php-fpm 43319 root 1 38 0 275M 187M piperd 2 6:56 0.14% php-fpm 89231 root 1 20 0 282M 193M piperd 3 8:17 0.13% php-fpm 43071 root 1 20 0 17M 8048K select 2 0:04 0.12% openvpn 73923 root 1 20 0 17M 8044K select 2 0:24 0.08% openvpn 43733 root 1 20 0 19M 7352K select 1 1:55 0.07% ntpd 1114 root 1 20 0 13M 3872K CPU2 2 0:00 0.06% top 87454 ladvd 1 20 0 12M 3668K kqread 2 0:24 0.06% ladvd 91582 root 1 20 0 17M 8048K select 0 0:04 0.06% openvpn 84109 root 1 20 0 17M 8048K select 3 0:02 0.04% openvpn 73851 root 1 20 0 17M 8044K select 0 0:01 0.03% openvpn 24967 root 1 20 0 17M 8048K select 2 0:01 0.03% openvpn 35020 dhcpd 1 20 0 25M 14M select 1 0:01 0.02% dhcpd 24683 dhcpd 1 20 0 23M 12M select 3 0:01 0.02% dhcpd 73977 root 5 52 0 11M 2744K uwait 1 0:00 0.01% dpinger 73179 root 5 52 0 11M 2744K uwait 3 0:00 0.01% dpinger 77833 root 5 52 0 11M 2752K uwait 3 0:00 0.01% dpinger 75360 root 5 52 0 11M 2748K uwait 0 0:00 0.01% dpinger 73643 root 5 52 0 11M 2748K uwait 3 0:00 0.01% dpinger 74945 root 5 52 0 11M 2752K uwait 0 0:00 0.01% dpinger 74142 root 5 52 0 11M 2748K uwait 1 0:00 0.01% dpinger 75772 root 5 52 0 11M 2752K uwait 3 0:00 0.01% dpinger 78751 root 5 52 0 11M 2748K uwait 2 0:00 0.01% dpinger 78385 root 5 52 0 11M 2752K uwait 2 0:00 0.01% dpinger 76483 root 5 52 0 11M 2752K uwait 0 0:00 0.01% dpinger 74324 root 5 52 0 11M 2748K uwait 1 0:00 0.01% dpinger 75975 root 5 52 0 11M 2748K uwait 3 0:00 0.01% dpinger 72716 root 5 52 0 11M 2748K uwait 2 0:00 0.01% dpinger 52427 root 1 20 0 20M 11M select 0 0:00 0.01% sshd 76955 root 5 52 0 14M 2804K uwait 1 0:00 0.01% dpinger 77541 root 5 52 0 14M 2808K uwait 2 0:00 0.01% dpinger 62005 root 1 20 0 30M 6668K select 0 0:00 0.00% staticd 17906 root 1 20 0 21M 9744K select 1 0:00 0.00% mpd5 95196 root 1 20 0 29M 5980K select 0 0:00 0.00% watchfrr 781 root 1 20 0 102M 32M kqread 3 0:00 0.00% php-fpm 70412 root 1 20 0 12M 3252K bpf 0 0:00 0.00% filterlog 10706 root 1 20 0 134M 46M lockf 0 0:28 0.00% php-fpm 51389 root 1 20 0 11M 2576K select 1 0:01 0.00% radvd 42571 root 1 20 0 30M 10M kqread 0 0:00 0.00% nginx 11827 root 1 20 0 11M 2804K select 2 0:00 0.00% syslogd 87374 root 1 20 0 13M 3684K kqread 1 0:00 0.00% ladvd 10106 root 1 20 0 20M 11M select 1 0:00 0.00% sshd 84935 root 1 20 0 11M 2672K select 0 0:00 0.00% dhcp6c 57658 root 1 20 0 11M 2612K select 2 0:00 0.00% dhcp6c 20903 root 1 20 0 21M 9744K select 2 0:00 0.00% mpd5 12249 root 1 20 0 21M 9724K select 0 0:00 0.00% mpd5 820 root 1 52 20 11M 2948K kqread 3 0:00 0.00% check_reload_status 42871 root 1 20 0 11M 2728K nanslp 0 0:00 0.00% cron 77784 root 1 20 0 11M 3008K select 2 0:00 0.00% xinetd 990 root 1 20 0 10M 1484K select 3 0:00 0.00% devd 60461 root 1 20 0 13M 3832K ttyin 3 0:00 0.00% tcsh 80785 root 1 20 0 13M 3828K pause 0 0:00 0.00% tcsh 35428 root 1 20 0 11M 2612K select 3 0:00 0.00% dhcp6c 28475 root 1 20 0 11M 2388K sbwait 3 0:00 0.00% fcgicli 991 root 1 52 0 11M 2944K wait 3 0:00 0.00% login 68662 root 1 52 0 11M 3036K wait 2 0:00 0.00% sh 66958 root 1 32 0 11M 3056K wait 3 0:00 0.00% sh -
It turns out the isssue was not related to yours. My issue was prefix lists were not migrated correctly and did not work. FRR 7.5 uses a different syntax to 7.3 and when I upgrade pfSense from 2.4.5-p1 to 2.5.0 the preflix list filters are not applied because of this. It meant the instance was accepting more routes than what can fit in its RAM and things started falling over.
Your instance is bare metal with plenty of RAM so it's a different cause.
-
Every time I have to use service frr stop command,
Webgui can only be used after FRR is stopped -
sysctl dev.cpu dev.cpu.3.temperature: 59.1C dev.cpu.3.cx_method: C1/hlt dev.cpu.3.cx_usage_counters: 32910065 dev.cpu.3.cx_usage: 100.00% last 214us dev.cpu.3.cx_lowest: C1 dev.cpu.3.cx_supported: C1/1/0 dev.cpu.3.%parent: acpi0 dev.cpu.3.%pnpinfo: _HID=none _UID=0 dev.cpu.3.%location: handle=\_PR_.P003 dev.cpu.3.%driver: cpu dev.cpu.3.%desc: ACPI CPU dev.cpu.2.temperature: 59.1C dev.cpu.2.cx_method: C1/hlt dev.cpu.2.cx_usage_counters: 37679235 dev.cpu.2.cx_usage: 100.00% last 236us dev.cpu.2.cx_lowest: C1 dev.cpu.2.cx_supported: C1/1/0 dev.cpu.2.%parent: acpi0 dev.cpu.2.%pnpinfo: _HID=none _UID=0 dev.cpu.2.%location: handle=\_PR_.P002 dev.cpu.2.%driver: cpu dev.cpu.2.%desc: ACPI CPU dev.cpu.1.temperature: 59.1C dev.cpu.1.cx_method: C1/hlt dev.cpu.1.cx_usage_counters: 32443366 dev.cpu.1.cx_usage: 100.00% last 337us dev.cpu.1.cx_lowest: C1 dev.cpu.1.cx_supported: C1/1/0 dev.cpu.1.%parent: acpi0 dev.cpu.1.%pnpinfo: _HID=none _UID=0 dev.cpu.1.%location: handle=\_PR_.P001 dev.cpu.1.%driver: cpu dev.cpu.1.%desc: ACPI CPU dev.cpu.0.temperature: 59.1C dev.cpu.0.cx_method: C1/hlt dev.cpu.0.cx_usage_counters: 35826705 dev.cpu.0.cx_usage: 100.00% last 195us dev.cpu.0.cx_lowest: C1 dev.cpu.0.cx_supported: C1/1/0 dev.cpu.0.freq_levels: 3600/19240 3200/14577 2800/10578 2400/6951 1900/4162 1400/2887 dev.cpu.0.freq: 3600 dev.cpu.0.%parent: acpi0 dev.cpu.0.%pnpinfo: _HID=none _UID=0 dev.cpu.0.%location: handle=\_PR_.P000 dev.cpu.0.%driver: cpu dev.cpu.0.%desc: ACPI CPU dev.cpu.%parent:
