How to troubleshoot High load average values on a new 2.1 NanoBSD VGA install?
-
I noticed higher power consumption (19 Watts) instead of the "usual" 15 Watts. After viewing the pfSense Status: Dashboard, I noticed that the load average values are not the usual 0.00 0.01 0.00 but "0.80, 0.80, 0.77" instead. This was at approximately 6 hours of uptime.
The "pfSense High load troubleshooting documentation" does help me a little.
systat -vmstat 1 output:
2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156032 18300 1020328 23724 1615800 count 14 All 189244 23004 1074838k 43000 pages 14 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 157k total 63 312k 4 407 155k 117 zfod 155k ehci0 16 ozfod 24 ehci1 23 0.0%Sys 19.5%Intr 0.0%User 0.0%Nice 80.5%Idle %ozfod 399 cpu0: time | | | | | | | | | | | daefr igb1:que 0 ++++++++++ prcfr igb1:que 1 33 dtbuf totfr 4 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 564 frevn pdpgs 399 cpu1: time intrn 399 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 399 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111644 act tps 0 3 0 0 0 0 39192 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613820 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156232 18300 1020328 23724 1615600 count All 189444 23004 1074838k 43000 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 158k total 63 313k 77 698 156k 119 52 50 zfod 156k ehci0 16 ozfod 25 ehci1 23 0.0%Sys 17.7%Intr 0.0%User 0.0%Nice 82.3%Idle %ozfod 401 cpu0: time | | | | | | | | | | | daefr igb1:que 0 +++++++++ prcfr igb1:que 1 35 dtbuf 1 totfr 64 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 3 3 100 564 frevn pdpgs 401 cpu1: time intrn 401 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 401 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111852 act tps 0 1 0 0 0 0 39184 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613620 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156232 18300 1020328 23724 1615600 count All 189444 23004 1074838k 43000 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 158k total 63 313k 77 698 156k 119 52 50 zfod 156k ehci0 16 ozfod 25 ehci1 23 0.0%Sys 17.7%Intr 0.0%User 0.0%Nice 82.3%Idle %ozfod 401 cpu0: time | | | | | | | | | | | daefr igb1:que 0 +++++++++ prcfr igb1:que 1 35 dtbuf 1 totfr 64 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 3 3 100 564 frevn pdpgs 401 cpu1: time intrn 401 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 401 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111852 act tps 0 1 0 0 0 0 39184 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613620 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156232 18300 1020328 23724 1615600 count All 189444 23004 1074838k 43000 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 158k total 63 313k 77 698 156k 119 52 50 zfod 156k ehci0 16 ozfod 25 ehci1 23 0.0%Sys 17.7%Intr 0.0%User 0.0%Nice 82.3%Idle %ozfod 401 cpu0: time | | | | | | | | | | | daefr igb1:que 0 +++++++++ prcfr igb1:que 1 35 dtbuf 1 totfr 64 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 3 3 100 564 frevn pdpgs 401 cpu1: time intrn 401 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 401 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111852 act tps 0 1 0 0 0 0 39184 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613620 free 89872 buf 2 users Load 0.78 0.81 0.77 Feb 3 23:13 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 156232 18300 1020328 23724 1615600 count All 189444 23004 1074838k 43000 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 158k total 63 313k 77 698 156k 119 52 50 zfod 156k ehci0 16 ozfod 25 ehci1 23 0.0%Sys 17.7%Intr 0.0%User 0.0%Nice 82.3%Idle %ozfod 401 cpu0: time | | | | | | | | | | | daefr igb1:que 0 +++++++++ prcfr igb1:que 1 35 dtbuf 1 totfr 64 igb1:que 2 Namei Name-cache Dir-cache 109357 desvn react igb1:que 3 Calls hits % hits % 1606 numvn pdwak igb1:link 3 3 100 564 frevn pdpgs 401 cpu1: time intrn 401 cpu2: time Disks md0 md1 da0 da1 pass0 pass1 118676 wire 401 cpu3: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 111852 act tps 0 1 0 0 0 0 39184 inact MB/s 0.00 0.00 0.00 0.00 0.00 0.00 1980 cache %busy 0 0 0 0 0 0 1613620 free 89872 buf -
When you say 'usual' you mean this system has been running for some time previously? What has changed?
Your load seems to be all interrupts and that seems to be all ehci0.
What does 'vmstat -i' show?
What USB devices are you using?Steve
-
The interrupt storm is a known FreeBSD issue: PR 156596.
The high interrupt rate on EHCI USB starts when disconnecting the monitor.
