Seemingly random CPU spikes (Causes high pings and VPN + WAN to go down)
-
Hello. I'm very new to PFsense, but I've worked with other router firmware and I'm Cisco certified, but this issue is a bit beyond me.
I set up my new PFsense router using a APU1D4 T40E, with a 30GB SSD and pfsense 2.1.5.
Setup went as expected, and it works GREAT, until it doesn't…
My purpose for building this router is to maintain high speeds (245mbps) across my WAN and be able to use my entire network (save a few connections) behind my PIA VPN. I used this to set it up https://forum.pfsense.org/index.php?topic=76015.0 and configured my NAT and firewall rules to ignore a few IP's which I had made static using an alias to denote certain IPs I want to go through my default WAN gateway.
Things work GREAT! Except... once in a while, around 5 - 10 minutes, my VPN connection to PIA drops, and comes back up with a different private IP and a new WAN IP for the PIA interface and CPU spikes... I'm noticing my firewall throwing away a lot of traffic too, and then I get anywhere between 70% and 99% CPU usage and sometimes (very infrequently) I drop connections to everything for a few seconds to a minute.
I troubleshot the issue using shell and a few commands found here https://doc.pfsense.org/index.php/High_Load_Troubleshooting and these are the high CPU spikes that I'm noticing (different instances separated by space):
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 87950 root 76 20 162M 51836K piperd 0 0:01 55.96% /usr/local/bin/php -f /etc/rc.filter_configure_sync 89163 root 76 20 12160K 7136K select 0 0:00 36.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp 89163 root 76 20 12160K 7136K uwait 0 0:00 36.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp 10 root 171 ki31 0K 32K RUN 0 52:53 33.98% [idle{idle: cpu0}] 10 root 171 ki31 0K 32K RUN 1 51:34 30.96% [idle{idle: cpu1}] 80669 root 76 20 13496K 4596K select 1 0:00 13.96% /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.con PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10 root 171 ki31 0K 32K RUN 1 55:06 55.96% [idle{idle: cpu1}] 10 root 171 ki31 0K 32K RUN 0 56:20 36.96% [idle{idle: cpu0}] 257 root 124 20 6908K 1392K CPU1 1 6:46 34.96% /usr/local/sbin/check_reload_status 11 root -68 - 0K 272K RUN 0 0:44 27.98% [intr{irq256: re0}] 40507 root 76 20 162M 51836K piperd 1 0:01 17.97% /usr/local/bin/php -f /etc/rc.filter_configure_sync 11 root -68 - 0K 272K RUN 0 0:27 16.99% [intr{irq257: re1}] 40079 root 76 20 156M 43644K nanslp 0 0:01 15.97% /usr/local/bin/php -f /etc/rc.dyndns.update WAN_DHCP 40260 root 76 20 154M 41528K RUN 0 0:00 14.99% /usr/local/bin/php -f /etc/rc.openvpn WAN_DHCP 0 root -16 0 0K 144K sched 0 0:56 0.00% [kernel{swapper}] PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 257 root 76 20 6908K 1392K kqread 1 7:17 63.96% /usr/local/sbin/check_reload_status 21974 root 66 20 12160K 7136K select 0 0:00 13.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp 21974 root 76 20 12160K 7136K uwait 1 0:00 13.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp 11 root -68 - 0K 272K CPU0 0 1:17 12.99% [intr{irq256: re0}] 11 root -68 - 0K 272K WAIT 0 0:43 8.98% [intr{irq257: re1}] 3881 root 64 20 13496K 4632K select 0 0:00 2.98% /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.con 38820 root 48 0 157M 51836K accept 1 0:44 0.98% /usr/local/bin/php{php} PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 257 root 132 20 6908K 1392K RUN 1 7:32 73.97% /usr/local/sbin/check_reload_status 35974 root 130 20 156M 44300K RUN 0 0:01 63.96% /usr/local/bin/php -f /etc/rc.filter_configure_sync 11 root -68 - 0K 272K CPU0 0 1:29 25.98% [intr{irq256: re0}] 36694 root 76 20 12160K 7136K select 0 0:00 23.97% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp 36694 root 76 20 12160K 7136K uwait 0 0:00 23.97% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp 10 root 171 ki31 0K 32K RUN 1 57:25 19.97% [idle{idle: cpu1}] 10 root 171 ki31 0K 32K RUN 0 58:03 16.99% [idle{idle: cpu0}] 11 root -68 - 0K 272K WAIT 0 0:49 14.99% [intr{irq257: re1}] 2339 root 65 20 13496K 4784K select 0 0:00 4.98% /usr/local/sbin/openvpn --config /var/etc/openAny questions about my setup, I'd be happy to answer. I've heard that the community here is very helpful and very knowledgeable; I have faith that this can be addressed by experts and nullify my issue completely :)
Thank you for your time and attention to this matter
Mirrored from https://forum.pfsense.org/index.php?topic=86736.0 for increased visibility
-
The only thing I can help with is to point out that your RealTek NIC interrupts have crazy high CPU usage. I highly recommend Intel NICs. Maybe polling out help. I'm sure someone else will give you some ideas.
I wonder what /usr/local/sbin/check_reload_status does that it's consuming so much CPU.
-
Crap, I wish I knew this before I went ahead and bought the module I was working with… Either way. I'm not sure. Lately, I've been coming home to a down router that needs a reboot, so the issue is much worse... I'd like for someone with more experience to ask me for my logs so they can determine what the issue might be :(