Seemingly random CPU spikes (Causes high pings and VPN + WAN to go down)

Ayu

Hello. I'm very new to PFsense, but I've worked with other router firmware and I'm Cisco certified, but this issue is a bit beyond me.

I set up my new PFsense router using a APU1D4 T40E, with a 30GB SSD and pfsense 2.1.5.

Setup went as expected, and it works GREAT, until it doesn't…

My purpose for building this router is to maintain high speeds (245mbps) across my WAN and be able to use my entire network (save a few connections) behind my PIA VPN. I used this to set it up https://forum.pfsense.org/index.php?topic=76015.0 and configured my NAT and firewall rules to ignore a few IP's which I had made static using an alias to denote certain IPs I want to go through my default WAN gateway.

Things work GREAT! Except... once in a while, around 5 - 10 minutes, my VPN connection to PIA drops, and comes back up with a different private IP and a new WAN IP for the PIA interface and CPU spikes... I'm noticing my firewall throwing away a lot of traffic too, and then I get anywhere between 70% and 99% CPU usage and sometimes (very infrequently) I drop connections to everything for a few seconds to a minute.

I troubleshot the issue using shell and a few commands found here https://doc.pfsense.org/index.php/High_Load_Troubleshooting and these are the high CPU spikes that I'm noticing (different instances separated by space):

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
87950 root      76   20   162M 51836K piperd  0   0:01 55.96% /usr/local/bin/php -f /etc/rc.filter_configure_sync
89163 root      76   20 12160K  7136K select  0   0:00 36.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
89163 root      76   20 12160K  7136K uwait   0   0:00 36.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
   10 root     171 ki31     0K    32K RUN     0  52:53 33.98% [idle{idle: cpu0}]
   10 root     171 ki31     0K    32K RUN     1  51:34 30.96% [idle{idle: cpu1}]
80669 root      76   20 13496K  4596K select  1   0:00 13.96% /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.con

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   10 root     171 ki31     0K    32K RUN     1  55:06 55.96% [idle{idle: cpu1}]
   10 root     171 ki31     0K    32K RUN     0  56:20 36.96% [idle{idle: cpu0}]
  257 root     124   20  6908K  1392K CPU1    1   6:46 34.96% /usr/local/sbin/check_reload_status
   11 root     -68    -     0K   272K RUN     0   0:44 27.98% [intr{irq256: re0}]
40507 root      76   20   162M 51836K piperd  1   0:01 17.97% /usr/local/bin/php -f /etc/rc.filter_configure_sync
   11 root     -68    -     0K   272K RUN     0   0:27 16.99% [intr{irq257: re1}]
40079 root      76   20   156M 43644K nanslp  0   0:01 15.97% /usr/local/bin/php -f /etc/rc.dyndns.update WAN_DHCP
40260 root      76   20   154M 41528K RUN     0   0:00 14.99% /usr/local/bin/php -f /etc/rc.openvpn WAN_DHCP
    0 root     -16    0     0K   144K sched   0   0:56  0.00% [kernel{swapper}]

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
  257 root      76   20  6908K  1392K kqread  1   7:17 63.96% /usr/local/sbin/check_reload_status
21974 root      66   20 12160K  7136K select  0   0:00 13.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
21974 root      76   20 12160K  7136K uwait   1   0:00 13.96% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
   11 root     -68    -     0K   272K CPU0    0   1:17 12.99% [intr{irq256: re0}]
   11 root     -68    -     0K   272K WAIT    0   0:43  8.98% [intr{irq257: re1}]
 3881 root      64   20 13496K  4632K select  0   0:00  2.98% /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.con
38820 root      48    0   157M 51836K accept  1   0:44  0.98% /usr/local/bin/php{php}

  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
  257 root     132   20  6908K  1392K RUN     1   7:32 73.97% /usr/local/sbin/check_reload_status
35974 root     130   20   156M 44300K RUN     0   0:01 63.96% /usr/local/bin/php -f /etc/rc.filter_configure_sync
   11 root     -68    -     0K   272K CPU0    0   1:29 25.98% [intr{irq256: re0}]
36694 root      76   20 12160K  7136K select  0   0:00 23.97% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
36694 root      76   20 12160K  7136K uwait   0   0:00 23.97% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
   10 root     171 ki31     0K    32K RUN     1  57:25 19.97% [idle{idle: cpu1}]
   10 root     171 ki31     0K    32K RUN     0  58:03 16.99% [idle{idle: cpu0}]
   11 root     -68    -     0K   272K WAIT    0   0:49 14.99% [intr{irq257: re1}]
 2339 root      65   20 13496K  4784K select  0   0:00  4.98% /usr/local/sbin/openvpn --config /var/etc/open

Any questions about my setup, I'd be happy to answer. I've heard that the community here is very helpful and very knowledgeable; I have faith that this can be addressed by experts and nullify my issue completely :)

Thank you for your time and attention to this matter

Mirrored from https://forum.pfsense.org/index.php?topic=86736.0 for increased visibility

Harvy66

The only thing I can help with is to point out that your RealTek NIC interrupts have crazy high CPU usage. I highly recommend Intel NICs. Maybe polling out help. I'm sure someone else will give you some ideas.

I wonder what /usr/local/sbin/check_reload_status does that it's consuming so much CPU.

Ayu

Crap, I wish I knew this before I went ahead and bought the module I was working with… Either way. I'm not sure. Lately, I've been coming home to a down router that needs a reboot, so the issue is much worse... I'd like for someone with more experience to ask me for my logs so they can determine what the issue might be :(