High CPU load (check_reload_status) + IPv6 Gateways

sirdir

I recently upgraded from 2.0.1 to

2.1-BETA0 (amd64)
built on Fri Jul 13 19:45:32 EDT 2012
FreeBSD 8.3-RELEASE-p3

(newest version).
To be able to add two ipv6 tunnels.
But now my cpu load is incredibly high.
the processes filterdns and check_reload_status generate about 100% each.

I already re-installed (but restored the configuration). Any idea where I could investigate what's going wrong here?

I can also confirm that I see a somewhat similar issue.

In my case I have a dynamic /64 IPv6 prefix and a static /56 IPv6 assigned by my ISP (Internode). I have an ADSL modem in bridge mode with pfsense acting as the PPPoE client. As soon as I enable DHCPv6 on the WAN I see near 100% cpu on one of my cores. Both my WAN_PPPOE and WAN_DHCPv6 gateways will read as pending for about a minute or two and then show "~".

I imagine that more can be learned by examining my syslog outputs, so please let me know what would be useful to help fix this.

thesnowedone

I can confirm the same issue as Jeremy; and am also running a similar configuration with the same ISP (Internode). ADSL modem is also in bridge mode with pfsense acting as the PPPoE client.

last pid: 53130; load averages: 1.73, 1.81, 1.78 up 1+13:35:55 01:55:01
159 processes: 6 running, 124 sleeping, 29 waiting

Mem: 406M Active, 2164M Inact, 290M Wired, 50M Cache, 112M Buf, 86M Free
Swap: 8192M Total, 28K Used, 8192M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
312 root 139 20 3416K 1172K CPU3 3 37.4H 100.00% check_reload_sta
11 root 171 ki31 0K 32K CPU0 0 34.1H 69.97% idle{idle: cpu0}
11 root 171 ki31 0K 32K RUN 2 966:32 51.17% idle{idle: cpu2}
11 root 171 ki31 0K 32K CPU1 1 26.3H 47.27% idle{idle: cpu1}
11 root 171 ki31 0K 32K RUN 3 17.3H 40.87% idle{idle: cpu3}
12 root -68 - 0K 232K WAIT 0 59:19 20.75% intr{irq256: em0:
36644 root 53 0 72960K 62300K select 1 33:11 14.99% rsync
12 root -64 - 0K 232K WAIT 0 9:21 3.47% intr{irq19: uhci2
17957 root 4 0 72960K 69824K select 1 10:13 2.98% rsync
12 root -24 - 0K 232K WAIT 0 7:50 2.78% intr{swi6: task q
39145 root 46 0 58984K 21584K piperd 2 0:28 1.86% php
12 root -68 - 0K 232K WAIT 0 3:09 0.78% intr{irq257: em0:
12 root -68 - 0K 232K WAIT 0 22:53 0.20% intr{irq259: em1:
4 root -8 - 0K 8K - 1 3:54 0.20% g_down
12 root -32 - 0K 232K WAIT 0 9:16 0.00% intr{swi4: clock}
21111 nobody 44 0 5576K 2416K select 3 5:01 0.00% dnsmasq
14 root -16 - 0K 8K - 1 4:33 0.00% yarrow
27124 root 44 0 314M 119M bpf 1 4:01 0.00% snort{snort}

Version is:
2.1-BETA0 (i386)
built on Mon Jul 16 19:08:20 EDT 2012
FreeBSD 8.3-RELEASE-p3

novacoresystems

What are the specs/type of CPU you are using? Just curious. I'm running 2.1 BETA, but I'm using a high end Intel Core 2 Duo and it's a virtual machine.. I do not have this issue.

thesnowedone

My system:
Supermicro X7SPA-HF-D525 motherboard with:
Intel(R) Atom(TM) CPU D525 @ 1.80GHz
4Gb RAM

It's a couple of years old now but is still a fairly solid m/b and CPU for routing.

novacoresystems

I haven't tested pfsense 2.1 on Atom processors.. but how much load are you throwing at it? It is just you in your house or a whole company/organization?

thesnowedone

@novacoresystems:

I haven't tested pfsense 2.1 on Atom processors.. but how much load are you throwing at it? It is just you in your house or a whole company/organization?

It's a large shared house - 6 people with at least a couple of them torrenting.

novacoresystems

I have noticed torrenting crashing pfsense actually. I think it creates a bunch of states/connections. To combat that, I increased the amount of states in the advanced properties and this seems to have fixed the issue… but as far as high CPU load on your system.. it may be related to the hardware.

I think though, your problem is related to the IPv6 support which still seems a bit buggy. Maybe you guys should submit a bug? Sounds like a bug to me... Cause I'm still using IPv4 for everything

I am on the following snapshot:

2.1-BETA0 (amd64)
built on Fri Jul 20 08:01:17 EDT 2012
FreeBSD 8.3-RELEASE-p3

My hardware was an AMD Athlon II X2 250 Processor in a Biostar A770E3 motherboard with 2GB of ram.

There is no significant load on the router (just me and my fiancée at home) and I don't generally see a spike in my states beyond about 2,000 / 195,000.

@thesnowdome - would you like to file the bug and I can provide additional data?

thesnowedone

Created an bug for it - https://redmine.pfsense.org/issues/2555

Cynagen

I just read the bug ticket, and I can confirm this exact behavior with my pfSense as well, however I'm using a HE.net TunnelBroker GIF IPv6 configuration (http://doc.pfsense.org/index.php/Using_IPv6_on_2.1_with_a_Tunnel_Broker). Without using device polling, the process check_reload_status will eat up both cores and never let go, rendering the entire platform useless until there is a reboot.

My build:
2.1-BETA0 (amd64)
built on Thu Jul 26 01:49:39 EDT 2012
FreeBSD 8.3-RELEASE-p3

AMD Athlon 64 X2 Dual Core Processor 4200+ w/2GB DDR2

Issue still occuring in latest snapshot:

2.1-BETA0 (amd64) 
built on Wed Aug 15 08:43:33 EDT 2012 
FreeBSD 8.3-RELEASE-p4

Bug can be reproduced by going to Interfaces > WAN > IPv6 Configuration Type > "DHCP6"

CPU spikes immediately after committing changes.

mikesamo

same on embedded 2.1

mikesamo

to fix that issue I just have to edit my 2 default gateway and apply.

Editing my gateways (System >> Routing >> Gateways (Tab) ) does not resolve the issue. In addition, leaving the v6 WAN applied and rebooting does not clear the check_reload_status process on one of the CPU cores.

A reboot or a full clearing of my changes (including manually removing my v6 gateway) is the only way to bring the CPU load back under control.

+++
EDIT - Mod - would it be possible to edit the topic of this thread to something more descriptive (like "High CPU load (check_reload_status) + IPv6 Gateways") ?

As of this version:

2.1-BETA0 (amd64) 
built on Mon Aug 27 10:33:40 EDT 2012 
FreeBSD 8.3-RELEASE-p4

I am able to bring check_reload_status down from spiking by reversing my config changes. However, I believe that I may perhaps be approaching my v6 setup incorrectly - has anyone had any success with Internode on 2.1 BETA? I can bring a SLAAC WAN interface up without creating the same CPU spike, but am unsure as to appropriate setup (DHCPv6, RA and static LAN assignments from my static /56 from the ISP.)

Hmmm … so no one has any ideas about this?

If we have simply incorrectly configured IPv6 in our setups (and thus causing the the CPU spike) then it would be helpful if someone could point us in the right direction.

jimp

check_reload_status itself doesn't tend to be the problem in these cases. Something (such as rc.newwanip) calls pfSctl which routes a request through check_reload_status which in turn tries to take another action - the problem seems to happen more often when there are either:
(a) a lot of programs trying to make calls through check_reload_status
(b) whatever check_reload_status is calling is not behaving as expected

Usually running something like this is enough to bring it back:
killall -9 php; killall -9 lighttpd; /etc/rc.restart_webgui

Once the callers/callees of check_reload status are gone the load tends to return to normal.

Thanks Jimp and sorry for my late reply….

I have experimented with bringing a DHCPv6 WAN up and, while it still spikes, it can be brought under control by removing the interface (which I guess is expected behaviour). This is on the following version:

2.1-BETA0 (amd64) 
built on Sat Sep 15 17:04:58 EDT 2012 
FreeBSD 8.3-RELEASE-p4

I believe I probably need to go back to the drawing board and consider if I'm approaching my v6 setup in the correct way (as you indicated in (b)), although it seems that https://redmine.pfsense.org/issues/2555 is continuing to move along.