Hi CPU load PfSense 2.3.4
-
Hi all.
Thought I try and ask in here and see if I can resolve an issue with high CPU usage at high loads. (Around 400-800Mbit)
I run the PfSense on a Hyper-V server 2012 R2, Gen 1 for the VM, with a fairly good patch level in the Windows OS. We have:1. Broadcom cards (NetXtreme BCM5720 Gigabit Ethernet PCIe) in the backend that all traffic goes through via Hyper-V virtual switches.
2. Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz on each host (3 in total).I have turned off Hardware Checksum Offloading as some threads suggested this (all 3 of them) so far but it dont change much.
I havnt had much load on the hosts before but since we moved from FortiGate appliance box to PfSense on the backup systems our load in the PfSense increased radically and what I can see is that the interrupts seems to go nuts. If I push it to hard I start to get timeouts too.
I followed a guide to take out some statistics from the firewall and I´ll post it below. Hopefully someone can make some sense out of it. I´ll admit I am a bit of a laymen when it comes to linux and troubleshooting so any suggestions to what might be amiss would be nice.
2.3.4-RELEASE-p1 (amd64) built on Fri Jul 14 14:52:43 CDT 2017 FreeBSD 10.3-RELEASE-p19 *** Welcome to pfSense 2.3.4-RELEASE-p1 (amd64 full-install) on AZ-PfSVault *** WAN (wan) -> hn0 -> v4: ************** LAN (lan) -> hn1 -> v4: ************** LAN2 (opt1) -> hn2 -> v4: ************** 0) Logout (SSH only) 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart webConfigurator 3) Reset webConfigurator password 12) PHP shell + pfSense tools 4) Reset to factory defaults 13) Update from console 5) Reboot system 14) Disable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell Enter an option: 8 [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: top -aSH last pid: 43261; load averages: 2.59, 2.29, 1.78 up 0+16:45:28 05:54:20 131 processes: 7 running, 100 sleeping, 24 waiting CPU: 3.8% user, 0.0% nice, 5.4% system, 48.9% interrupt, 41.8% idle Mem: 15M Active, 87M Inact, 244M Wired, 158M Buf, 1614M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -60 - 0K 416K CPU3 3 627:31 85.06% [intr{swi4: hv_e 12 root -60 - 0K 416K CPU1 1 623:19 78.76% [intr{swi4: hv_e 11 root 155 ki31 0K 64K CPU0 0 758:39 51.17% [idle{idle: cpu0 11 root 155 ki31 0K 64K RUN 1 577:32 38.77% [idle{idle: cpu1 11 root 155 ki31 0K 64K RUN 3 574:45 38.57% [idle{idle: cpu3 11 root 155 ki31 0K 64K RUN 2 580:14 34.28% [idle{idle: cpu2 12 root -60 - 0K 416K WAIT 2 119:48 33.79% [intr{swi4: hv_e 12 root -60 - 0K 416K WAIT 0 125:52 18.55% [intr{swi4: cloc 25324 root 44 0 268M 32684K accept 0 0:01 8.69% php-fpm: pool ng 58711 root 20 0 21856K 3128K select 0 0:06 0.49% top -aSH 44001 root 20 0 21856K 3140K CPU2 2 0:01 0.10% top -aSH 15 root -16 - 0K 16K - 1 2:09 0.00% [rand_harvestq] 12 root -72 - 0K 416K WAIT 1 1:27 0.00% [intr{swi1: neti 0 root -16 - 0K 192K swapin 0 0:54 0.00% [kernel{swapper} 338 root 52 0 12280K 2072K select 0 0:31 0.00% /usr/sbin/hv_kvp 12 root -72 - 0K 416K WAIT 0 0:30 0.00% [intr{swi1: neti [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: systat -vmstat 1 2 users Load 2.39 2.30 1.83 Sep 12 05:55 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 113508 11748 1393084 13316 1648456 count All 118740 16924 1458844 66164 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt ioflt 4723 total 2 62 3076 3823 16k 1374 3926 845 cow atkbd0 1 2699 zfod fdc0 6 15.1%Sys 47.2%Intr 2.5%User 0.0%Nice 35.3%Idle ozfod ata1 15 | | | | | | | | | | %ozfod 1110 cpu0:timer ========+++++++++++++++++++++++> daefr 745 cpu3:timer 59 dtbuf 1972 prcfr 860 cpu2:timer Namei Name-cache Dir-cache 112490 desvn 3825 totfr 686 cpu1:timer Calls hits % hits % 1071 numvn react 564 cpu0:hyper 1979 1972 100 128 frevn pdwak 88 cpu1:hyper 7 pdpgs 670 cpu2:hyper Disks md0 da0 cd0 pass0 pass1 intrn cpu3:hyper KB/t 0.00 0.00 0.00 0.00 0.00 251096 wire tps 0 0 0 0 0 18068 act MB/s 0.00 0.00 0.00 0.00 0.00 89792 inact %busy 0 0 0 0 0 cache [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: netstat -m 4512/5103/9615 mbufs in use (current/cache/total) 348/2698/3046/125468 mbuf clusters in use (current/cache/total/max) 348/2688 mbuf+clusters out of packet secondary zone in use (current/cache) 0/102/102/62733 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/18587 9k jumbo clusters in use (current/cache/total/max) 0/0/0/10455 16k jumbo clusters in use (current/cache/total/max) 1824K/7079K/8903K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: systat -iostat 1 /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 Load Average ||||||||| /0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100 cpu user| nice| system| interrupt|XXXXXXXXXX idle|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX /0% /10 /20 /30 /40 /50 /60 /70 /80 /90 /100 md0 MB/s tps| da0 MB/s tps| cd0 MB/s tps| pass0 MB/s tps| pass1 MB/s tps| [2.3.4-RELEASE][admin@AZ-PfSVault.localdomain]/root: