PfSense High CPU Load Out Of Nowhere



  • Hello,
    I have been running my pfSense router just fine now for about a year. But in the last couple days, around when I updated to the most recent version of 2.1.3-RELEASE (i386), my CPU load jumped from its normal 23% up to 50-100%. There is no  extra traffic, nothing has changed. I tried rebooting, no change.
    Here is my debugging:

    
    # systat -vmstat 1
    
        1 users    Load  1.00  1.01  1.00                  Jun  8 08:11
    
    Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
            Tot   Share      Tot    Share    Free           in   out     in   out
    Act   70792   14960   194996    20048  372676  count            
    All   99776   18416  2328312    28432          pages            
    Proc:                                                            Interrupts
      r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        cow    3996 total
                 40        1M    8  841       156             zfod        ata0 irq14
                                                              ozfod  1998 cpu0: time
    49.8%Sys   0.0%Intr  0.4%User  0.0%Nice 49.8%Idle        %ozfod  1998 cpu1: time
    |    |    |    |    |    |    |    |    |    |    |       daefr
    =========================                                 prcfr
                                            16 dtbuf        1 totfr
    Namei     Name-cache   Dir-cache     34708 desvn          react
       Calls    hits   %    hits   %      1050 numvn          pdwak
           3       3 100                   134 frevn          pdpgs
                                                              intrn
    Disks   ad0   md0                                   55240 wire
    KB/t   0.00  0.00                                   39064 act
    tps       0     1                                   24836 inact
    MB/s   0.00  0.00                                     240 cache
    %busy     0     0                                  372436 free
                                                        28944 buf
    
    
    # top -aSH
    
    last pid: 26062;  load averages:  1.00,  1.00,  1.00                                                                                                 up 2+12:47:33  08:12:26
    118 processes: 4 running, 93 sleeping, 21 waiting
    CPU:  0.0% user,  0.0% nice, 50.0% system,  0.0% interrupt, 50.0% idle
    Mem: 37M Active, 24M Inact, 54M Wired, 240K Cache, 28M Buf, 365M Free
    Swap: 1024M Total, 1024M Free
    
      PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
       19 root     171 ki-6     0K     8K CPU0    0  60.6H 100.00% [idlepoll]
       11 root     171 ki31     0K    16K RUN     1  56.2H 85.35% [idle{idle: cpu1}]
       11 root     171 ki31     0K    16K RUN     0 204:41 13.38% [idle{idle: cpu0}]
      252 root      76   20  3352K  1196K kqread  0  12:04  0.00% /usr/local/sbin/check_reload_status
       12 root     -32    -     0K   168K WAIT    0   2:40  0.00% [intr{swi4: clock}]
    22951 root      64   20  3264K  1252K select  0   1:44  0.00% /usr/local/sbin/apinger -c /var/etc/apinger.conf
    91743 root      45    0 31060K 23480K accept  0   1:19  0.00% /usr/local/bin/php{php}
        0 root     -16    0     0K    72K sched   1   0:46  0.00% [kernel{swapper}]
    79987 root      44    0  3412K  1436K select  0   0:23  0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/
       14 root     -16    -     0K     8K -       0   0:23  0.00% [yarrow]
    55697 root      47    0 34260K 24560K accept  0   0:22  0.00% /usr/local/bin/php{php}
       12 root     -32    -     0K   168K WAIT    0   0:21  0.00% [intr{swi4: clock}]
    51103 root      76   20  3644K  1512K wait    0   0:18  0.00% /bin/sh /var/db/rrd/updaterrd.sh
    57889 root      76   20  3644K  1512K wait    0   0:17  0.00% /bin/sh /var/db/rrd/updaterrd.sh
    28829 root      44    0  8004K  5540K kqread  1   0:15  0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.c
        3 root      -8    -     0K     8K -       0   0:09  0.00% [g_up]
       21 root      20    -     0K     8K syncer  1   0:07  0.00% [syncer]
    18223 root      44    0  3264K   872K piperd  0   0:06  0.00% logger -t pf -p local0.info
    18189 root      44    0  5868K  3344K bpf     1   0:06  0.00% /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
        4 root      -8    -     0K     8K -       0   0:05  0.00% [g_down]
    74794 root      64   20  6048K  6068K select  1   0:05  0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntp
        8 root     -16    -     0K     8K pftm    1   0:03  0.00% [pfpurge]
       12 root     -64    -     0K   168K WAIT    0   0:03  0.00% [intr{irq14: ata0}]
       66 root      -8    -     0K     8K mdwait  1   0:02  0.00% [md0]
       15 root     -40    -     0K   160K -       1   0:01  0.00% [usb{usbus4}]
       36 root      -8    -     0K    16K l2arc_  1   0:01  0.00% [zfskern{l2arc_feed_threa}]
       36 root      -8    -     0K    16K arc_re  1   0:01  0.00% [zfskern{arc_reclaim_thre}]
    10033 root      56    0  3324K  1300K select  0   0:01  0.00% dhclient: bge0 [priv] (dhclient)
       23 root     -16    -     0K     8K sdflus  1   0:01  0.00% [softdepflush]
       22 root     -16    -     0K     8K vlruwt  1   0:01  0.00% [vnlru]
       20 root     -16    -     0K     8K psleep  1   0:01  0.00% [bufdaemon]
       15 root     -40    -     0K   160K -       1   0:01  0.00% [usb{usbus1}]
       15 root     -40    -     0K   160K -       1   0:01  0.00% [usb{usbus0}]
       15 root     -40    -     0K   160K -       1   0:01  0.00% [usb{usbus3}]
       15 root     -40    -     0K   160K -       1   0:01  0.00% [usb{usbus2}]
    16084 _dhcp     44    0  3324K  1400K select  1   0:01  0.00% dhclient: bge0 (dhclient)
    80937 root      44    0  3352K  1332K nanslp  1   0:00  0.00% /usr/sbin/cron -s
    89780 root      61    0  3264K  1028K nanslp  1   0:00  0.00% minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
    49939 nobody    44    0  5512K  2876K select  0   0:00  0.00% /usr/local/sbin/dnsmasq --all-servers --dns-forward-max=5000 
       16 root     -16    -     0K     8K psleep  1   0:00  0.00% [pagedaemon]
    95253 root      44    0  3324K  1204K nanslp  1   0:00  0.00% /usr/libexec/getty Pc ttyv0
    
    

    I am not sure what I should do. Should I reinstall or is it fine?

    Thanks,
    Trevor


  • Netgate Administrator

    Do you have device polling enabled in System: Advanced: Networking:? Disable it if you do.

    https://forum.pfsense.org/index.php?topic=30817.0

    Steve



  • Hello,

    We had a problem with high cpu as well. We were running snort and openvpn. The cpu went up every time the network traffic went up. for example if we tried to download a large file, the cpu would jump to 80%. We fixed it by enabling device polling. from System: Advanced: Networking. Enabling device polling is actually a good idea if your hardware is being used close to its capacity. see http://www.cyberciti.biz/faq/freebsd-device-polling-network-polling-tutorial/ and https://blog.pfsense.org/?p=115. Enabling polling disables cpu interrupts, which take a lot of cpu resources. After we enabled device polling the cpu went down to 40%. We also disabled ipv6 under System: Advanced: Networking. see https://forum.pfsense.org/index.php?topic=77493.msg422407#msg422407. If you are running pfsense in a virtualized environment you can get a performance boot by using virtio drivers. https://doc.pfsense.org/index.php/VirtIO_Driver_Support.

    Nadir Latif


  • Netgate Administrator

    You may like to read this thread:
    https://forum.pfsense.org/index.php?topic=83861.0;all
    The symptoms you dedcribe sound very similar.

    Steve



  • On high speed networks we can get a performance boost by moving the tcp/ip processing from the operating system to the network card hardware. This is called TCP Offloading. see http://en.wikipedia.org/wiki/TCP_offload_engine. FreeBSD has good support for TCP offloading. In Pfsense it can be enabled by unchecking the "Disable hardware checksum offload" option. In pfsense this option is unchecked by default.


  • Netgate Administrator

    The hardware offloading features available in the System: Advanced: Networking: section of the webgui do not include a complete TOE as referenced in that Wiki page. They only offload smaller functions: TSO, LRO and checksum.

    It looks like there is at least some support for TOE in FreeBSD but you would need to enable in manually in pfSense. Importantly I have no idea how it would interact with pf. As referenced in the wiki article once you've handed off the entire TCP stack to hardware much of the OS internal networking features are by-passed. It could be potentially completely redundant in pfSense.

    Steve