Check_reload_status CPU usage

maverick_slo · Apr 9, 2013, 7:32 AM

Hey all!

I think old bug is back :)
CPU usage is in GUI at 100%

last pid: 68674;  load averages:  1.96,  1.69,  1.61                                                                                                                                   up 1+13:51:51  09:39:38
41 processes:  2 running, 39 sleeping
CPU:  0.8% user, 38.3% nice, 55.6% system,  5.3% interrupt,  0.0% idle
Mem: 53M Active, 15M Inact, 66M Wired, 1028K Cache, 31M Buf, 832M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
  259 root        1 129   20  3352K  1180K RUN     60:50 60.89% check_reload_status
81386 root        1  76    0 73520K 24932K accept   0:33  9.08% php
44054 root        1  47    0 73520K 25716K accept   0:22  1.95% php
50549 root        1  44    0  3660K  1860K RUN      0:00  0.20% top
65317 root        1  76   20  3644K  1528K wait     0:49  0.00% sh
60605 root        1  64   20  3264K  1296K select   0:47  0.00% apinger
40799 root        1   1    0  4924K  2536K select   0:31  0.00% syslogd
68136 root        1  44    0  3300K  1440K select   0:20  0.00% miniupnpd
 7141 root        1  64   20  6280K  6300K select   0:17  0.00% ntpd
24928 root        1  44    0  3264K   876K piperd   0:16  0.00% logger
98482 root        1  64   20  5452K  3984K select   0:15  0.00% openvpn
24848 root        1  44    0  6892K  3508K bpf      0:12  0.00% tcpdump
51262 dhcpd       1  44    0  8384K  5324K select   0:12  0.00% dhcpd
28806 root        1  44    0  3384K  1424K select   0:12  0.00% inetd
40735 root        1  44    0  7984K  5656K kqread   0:11  0.00% lighttpd
98824 root        1  64   20  5452K  3956K select   0:05  0.00% openvpn
15362 root        1  44    0  7880K  3868K select   0:05  0.00% mpd5
  279 root        1  44    0  3936K  2272K select   0:02  0.00% devd
46495 nobody      1  44    0  5512K  2488K select   0:02  0.00% dnsmasq
60667 root        1  64   20  4540K  1656K piperd   0:01  0.00% rrdtool
42961 root        1  44    0  3264K  1568K kqread   0:00  0.00% dhcpleases
70833 root        1  44    0  3356K  1308K nanslp   0:00  0.00% cron
41755 root        1  71    0 72496K 14376K wait     0:00  0.00% php
40811 root        1  76    0 72496K 14352K wait     0:00  0.00% php
83455 root        1  73    0  3264K  1024K nanslp   0:00  0.00% minicron
31545 root        1  44    0  8036K  3568K select   0:00  0.00% sshd
50287 root        2  44    0  3480K  1132K nanslp   0:00  0.00% sshlockout_pf
48350 root        1  76    0  4696K  2444K pause    0:00  0.00% tcsh
 9728 root        2  44    0  3480K  1052K nanslp   0:00  0.00% sshlockout_pf
79042 root        2  44    0  3480K  1112K nanslp   0:00  0.00% sshlockout_pf
84229 root        1  50    0  3264K  1024K nanslp   0:00  0.00% minicron
33768 root        1  76    0  3644K  1544K wait     0:00  0.00% sh
34584 root        1  76    0  3644K  1548K wait     0:00  0.00% sh
98024 root        1  76    0  3324K  1128K ttyin    0:00  0.00% getty
 9450 root        1  44    0  5284K  3032K select   0:00  0.00% sshd
16344 root        1  76   20  1512K   772K nanslp   0:00  0.00% sleep
84400 root        1  76    0  3264K   976K wait     0:00  0.00% minicron
83654 root        1  76    0  3264K   976K wait     0:00  0.00% minicron
84429 root        1  50    0  3264K  1024K nanslp   0:00  0.00% minicron
83056 root        1  76    0  3264K   976K wait     0:00  0.00% minicron
  261 root        1  76   20  3352K   984K kqread   0:00  0.00% check_reload_status

cmb · Apr 10, 2013, 2:28 AM

That's generally a symptom, not a cause. What's going on with the system at the time? System log.

maverick_slo · Apr 10, 2013, 6:23 AM

Hey!

There you go:

Apr 10 08:26:31 	php: : DEVD Ethernet detached event for opt1
Apr 10 08:26:30 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:29 	kernel: fxp2: link state changed to UP
Apr 10 08:26:27 	kernel: fxp2: link state changed to DOWN
Apr 10 08:26:27 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:27 	php: : HOTPLUG: Configuring interface opt1
Apr 10 08:26:27 	php: : DEVD Ethernet attached event for opt1
Apr 10 08:26:25 	php: : DEVD Ethernet detached event for opt1
Apr 10 08:26:23 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:23 	kernel: fxp2: link state changed to UP
Apr 10 08:26:21 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:21 	kernel: fxp2: link state changed to DOWN
Apr 10 08:26:21 	php: : HOTPLUG: Configuring interface opt1
Apr 10 08:26:21 	php: : DEVD Ethernet attached event for opt1
Apr 10 08:26:19 	php: : DEVD Ethernet detached event for opt1
Apr 10 08:26:17 	kernel: fxp2: link state changed to UP
Apr 10 08:26:17 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:15 	kernel: fxp2: link state changed to DOWN
Apr 10 08:26:15 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:15 	php: : HOTPLUG: Configuring interface opt1
Apr 10 08:26:15 	php: : DEVD Ethernet attached event for opt1
Apr 10 08:26:13 	php: : DEVD Ethernet detached event for opt1
Apr 10 08:26:11 	kernel: fxp2: link state changed to UP
Apr 10 08:26:11 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:09 	kernel: fxp2: link state changed to DOWN
Apr 10 08:26:09 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:09 	php: : HOTPLUG: Configuring interface opt1
Apr 10 08:26:09 	php: : DEVD Ethernet attached event for opt1
Apr 10 08:26:06 	php: : DEVD Ethernet detached event for opt1
Apr 10 08:26:04 	kernel: fxp2: link state changed to UP
Apr 10 08:26:04 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:02 	kernel: fxp2: link state changed to DOWN
Apr 10 08:26:02 	check_reload_status: Linkup starting fxp2
Apr 10 08:26:02 	php: : HOTPLUG: Configuring interface opt1
Apr 10 08:26:02 	php: : DEVD Ethernet attached event for opt1
Apr 10 08:26:00 	php: : DEVD Ethernet detached event for opt1
Apr 10 08:25:58 	kernel: fxp2: link state changed to UP
Apr 10 08:25:58 	check_reload_status: Linkup starting fxp2
Apr 10 08:25:56 	kernel: fxp2: link state changed to DOWN
Apr 10 08:25:56 	check_reload_status: Linkup starting fxp2
Apr 10 08:25:56 	php: : HOTPLUG: Configuring interface opt1
Apr 10 08:25:56 	php: : DEVD Ethernet attached event for opt1
Apr 10 08:25:55 	php: : DEVD Ethernet detached event for opt1

But strange as it is, after reboot, this dissapears…

cmb · Apr 10, 2013, 8:31 AM

Yeah with the link cycling non-stop, check_reload_status will be at high CPU usage, it has to do a number of things any time a link goes up or down and you're doing all those in a non-stop loop there. Question is why was the link flapping. Most likely causes given a reboot fixes, either the NIC is flaking out, or the device it's plugged into is flaking out. Most likely not cabling-related if a reboot fixes. The reboot in and of itself is unlikely to be the fix, it's most likely the fact that the link went down for a significant period of time during the reboot and was brought back up. If it happens again, work to eliminate one of those two causes.

maverick_slo · Apr 15, 2013, 6:50 AM

Meeeh it was update.
I upgraded to latest snap 5 days ago and now it runs just fine for 5 straight days…
NIC 100% OK, cable too. So it was snap issue afterall...

maverick_slo · Apr 20, 2013, 11:29 AM

Hi again!

I found a way how te replicate this….
On my pfsense, there are 4 identical NICs.
One of them fxp2 is causing problem but only when I reboot server to which it is plugged in.
Any idea why link is not established back?

maverick_slo · Apr 28, 2013, 5:20 PM

Huh thats odd…
LAGG interface comes up just fine.
other interface used for pppoe aswell...
Just this NIC seems to cause problems...
Maybe driver update?