Hi CPU load PfSense 2.3.4



  • Hi all.

    Thought I try and ask in here and see if I can resolve an issue with high CPU usage at high loads. (Around 400-800Mbit)
    I run the PfSense on a Hyper-V server 2012 R2, Gen 1 for the VM, with a fairly good patch level in the Windows OS. We have:

    1. Broadcom cards (NetXtreme BCM5720 Gigabit Ethernet PCIe) in the backend that all traffic goes through via Hyper-V virtual switches.
    2. Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz on each host (3 in total).

    I have turned off Hardware Checksum Offloading as some threads suggested this (all 3 of them) so far but it dont change much.

    I havnt had much load on the hosts before but since we moved from FortiGate appliance box to PfSense on the backup systems our load in the PfSense increased radically and what I can see is that the interrupts seems to go nuts. If I push it to hard I start to get timeouts too.

    I followed a guide to take out some statistics from the firewall and I´ll post it below. Hopefully someone can make some sense out of it. I´ll admit I am a bit of a laymen when it comes to linux and troubleshooting so any suggestions to what might be amiss would be nice.

    2.3.4-RELEASE-p1 (amd64) 
    built on Fri Jul 14 14:52:43 CDT 2017 
    FreeBSD 10.3-RELEASE-p19 
    *** Welcome to pfSense 2.3.4-RELEASE-p1 (amd64 full-install) on AZ-PfSVault ***
    
     WAN (wan)       -> hn0        -> v4: **************
     LAN (lan)       -> hn1        -> v4: **************
     LAN2 (opt1)     -> hn2        -> v4: **************
    
     0) Logout (SSH only)                  9) pfTop
     1) Assign Interfaces                 10) Filter Logs
     2) Set interface(s) IP address       11) Restart webConfigurator
     3) Reset webConfigurator password    12) PHP shell + pfSense tools
     4) Reset to factory defaults         13) Update from console
     5) Reboot system                     14) Disable Secure Shell (sshd)
     6) Halt system                       15) Restore recent configuration
     7) Ping host                         16) Restart PHP-FPM
     8) Shell
    
    Enter an option: 8
    
    [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: top -aSH
    
    last pid: 43261;  load averages:  2.59,  2.29,  1.78    up 0+16:45:28  05:54:20
    131 processes: 7 running, 100 sleeping, 24 waiting
    CPU:  3.8% user,  0.0% nice,  5.4% system, 48.9% interrupt, 41.8% idle
    Mem: 15M Active, 87M Inact, 244M Wired, 158M Buf, 1614M Free
    Swap: 4096M Total, 4096M Free
    
      PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
       12 root     -60    -     0K   416K CPU3    3 627:31  85.06% [intr{swi4: hv_e
       12 root     -60    -     0K   416K CPU1    1 623:19  78.76% [intr{swi4: hv_e
       11 root     155 ki31     0K    64K CPU0    0 758:39  51.17% [idle{idle: cpu0
       11 root     155 ki31     0K    64K RUN     1 577:32  38.77% [idle{idle: cpu1
       11 root     155 ki31     0K    64K RUN     3 574:45  38.57% [idle{idle: cpu3
       11 root     155 ki31     0K    64K RUN     2 580:14  34.28% [idle{idle: cpu2
       12 root     -60    -     0K   416K WAIT    2 119:48  33.79% [intr{swi4: hv_e
       12 root     -60    -     0K   416K WAIT    0 125:52  18.55% [intr{swi4: cloc
    25324 root      44    0   268M 32684K accept  0   0:01   8.69% php-fpm: pool ng
    58711 root      20    0 21856K  3128K select  0   0:06   0.49% top -aSH
    44001 root      20    0 21856K  3140K CPU2    2   0:01   0.10% top -aSH
       15 root     -16    -     0K    16K -       1   2:09   0.00% [rand_harvestq]
       12 root     -72    -     0K   416K WAIT    1   1:27   0.00% [intr{swi1: neti
        0 root     -16    -     0K   192K swapin  0   0:54   0.00% [kernel{swapper}
      338 root      52    0 12280K  2072K select  0   0:31   0.00% /usr/sbin/hv_kvp
       12 root     -72    -     0K   416K WAIT    0   0:30   0.00% [intr{swi1: neti
    [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: systat -vmstat 1
    
        2 users    Load  2.39  2.30  1.83                  Sep 12 05:55
    
    Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
            Tot   Share      Tot    Share    Free           in   out     in   out
    Act  113508   11748  1393084    13316 1648456  count
    All  118740   16924  1458844    66164          pages
    Proc:                                                            Interrupts
      r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        ioflt  4723 total
      2          62      3076 3823  16k      1374 3926    845 cow         atkbd0 1
                                                         2699 zfod        fdc0 6
    15.1%Sys  47.2%Intr  2.5%User  0.0%Nice 35.3%Idle         ozfod       ata1 15
    |    |    |    |    |    |    |    |    |    |           %ozfod  1110 cpu0:timer
    ========+++++++++++++++++++++++>                          daefr   745 cpu3:timer
                                            59 dtbuf     1972 prcfr   860 cpu2:timer
    Namei     Name-cache   Dir-cache    112490 desvn     3825 totfr   686 cpu1:timer
       Calls    hits   %    hits   %      1071 numvn          react   564 cpu0:hyper
        1979    1972 100                   128 frevn          pdwak    88 cpu1:hyper
                                                            7 pdpgs   670 cpu2:hyper
    Disks   md0   da0   cd0 pass0 pass1                       intrn       cpu3:hyper
    KB/t   0.00  0.00  0.00  0.00  0.00                251096 wire
    tps       0     0     0     0     0                 18068 act
    MB/s   0.00  0.00  0.00  0.00  0.00                 89792 inact
    %busy     0     0     0     0     0                       cache
    [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: netstat -m
    4512/5103/9615 mbufs in use (current/cache/total)
    348/2698/3046/125468 mbuf clusters in use (current/cache/total/max)
    348/2688 mbuf+clusters out of packet secondary zone in use (current/cache)
    0/102/102/62733 4k (page size) jumbo clusters in use (current/cache/total/max)
    0/0/0/18587 9k jumbo clusters in use (current/cache/total/max)
    0/0/0/10455 16k jumbo clusters in use (current/cache/total/max)
    1824K/7079K/8903K bytes allocated to network (current/cache/total)
    0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
    0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
    0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
    0/0/0 requests for jumbo clusters denied (4k/9k/16k)
    0 requests for sfbufs denied
    0 requests for sfbufs delayed
    0 requests for I/O initiated by sendfile
    [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: systat -iostat 1
    
                        /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
         Load Average   |||||||||
    
              /0%  /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
    cpu  user|
         nice|
       system|
    interrupt|XXXXXXXXXX
         idle|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    
              /0%  /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
    md0   MB/s
          tps|
    da0   MB/s
          tps|
    cd0   MB/s
          tps|
    pass0 MB/s
          tps|
    pass1 MB/s
          tps|
    
    [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: