High interrupt CPU usage in v2.7.1
-
Hi all,
I've upgraded pfSense (it's a VM) from v2.7.0 to v2.7.1 last night, which went okay with no issues. It rebooted okay and everything came back online. However, I found something different.
With v2.7.0, the CPU usage shows under 5% most of the time in the web interface. Using
top -aSHcommand, I could see high percentage figures in idle section for CPU.When upgraded to v2.7.1, things were completely changed. the CPU usage in the web interface was over 40% all the time and it reached 100% frequently. The result from
top -aSHcommand was different as well. The high percentage figures were shown in interrupt section, not idle section.I've reverted the VM back to v.2.7.0 just in case. Is there anyone having the same experience? If so, is this expected?
Hope you can help. Thank you.
Eoin
-
@eoin what hypervisor?
-
@SteveITS It's XCP-ng v8.2.1.
-
I have the same thing. high interruptions on 2.7.1 and xcp-ng 8.2.1
-
installed from scratch. nothing is configured except the lan interface.
-
I have the same thing. high interruptions on 2.7.1 and XenServer 7.6
-
J jimp moved this topic from Problems Installing or Upgrading pfSense Software on
-
J jimp moved this topic from General pfSense Questions on
-
After updating to 2.7.2 nothing has changed
-
S sknigen referenced this topic on
-
S stephenw10 referenced this topic on
-
I can confirm having this issue on XCP-NG 8.2.1 running CE 2.7.2
-
Can confirm, run many instances of pfSense, and am seeing the same thing with CE 2.7.2 on xcp-ng, but not on ESXi or Proxmox.
-
I can confirm the issue is not limited to just VMs. I have tested a duo core Intel 3.0Ghz with 16 GB Ram and 160GB hard drive. As a test base config of 2.7.0 CPU was between 3-7 % moved same config to 2.7.1 and the CPU was between 30 - 60 % under the same load and configuration. moving to 2.7.2 changes nothing.
-
Same issue. I discovered that when any client is uploading over the WAN interface CPU usage actually drops drastically. From ~60% at idle to ~3% during light uploads.
Not really sure what that implies other than it possibly relating to interface interrupts.Here is the output of
vmstat -iinterrupt total rate
irq1: atkbd0 896 1
irq15: ata1 751 1
irq23: uhci0 16 0
cpu0:xen 318278 288
cpu1:xen 299866 271
cpu2:xen 291278 263
cpu3:xen 319322 289
cpu4:xen 306686 277
cpu5:xen 358225 324
cpu6:xen 343418 311
cpu7:xen 332859 301
irq2096: cpu0:r 268 0
irq2097: cpu0:itlb 11705 11
irq2098: cpu0:b 144030 130
irq2102: cpu1:r 275 0
irq2103: cpu1:itlb 11923 11
irq2104: cpu1:b 151704 137
irq2108: cpu2:r 327 0
irq2109: cpu2:itlb 12107 11
irq2110: cpu2:b 160624 145
irq2114: cpu3:r 296 0
irq2115: cpu3:itlb 11953 11
irq2116: cpu3:b 151243 137
irq2120: cpu4:r 257 0
irq2121: cpu4:itlb 12224 11
irq2122: cpu4:b 150400 136
irq2126: cpu5:r 245 0
irq2127: cpu5:itlb 11784 11
irq2128: cpu5:b 132161 120
irq2132: cpu6:r 259 0
irq2133: cpu6:itlb 11828 11
irq2134: cpu6:b 139764 126
irq2138: cpu7:r 129 0
irq2139: cpu7:itlb 12124 11
irq2140: cpu7:b 139024 126
irq2144: xen_et0:c0 23144 21
irq2145: xen_et0:c1 19830 18
irq2146: xen_et0:c2 17688 16
irq2147: xen_et0:c3 19772 18
irq2148: xen_et0:c4 17587 16
irq2149: xen_et0:c5 19831 18
irq2150: xen_et0:c6 18880 17
irq2151: xen_et0:c7 18379 17
irq2152: xenstore0 1699 2
irq2161: xbd0 5878 5
irq2162: xn0 134371 122
irq2163: xn0 149761 135
irq2164: xn0 112412 102
irq2165: xn0 128987 117
irq2166: xn1 157139 142
irq2167: xn1 183768 166
irq2168: xn1 193236 175
irq2169: xn1 216377 196
Total 5276988 4773pfSense UI shows the same interface interrupt rates of 300-400/s+.
-
S Squish referenced this topic on
-
Exact same issue, several XenServer / Citrix Hypervisor VMs, upgraded from 2.7.0 to 2.7.2 and interrupt CPU usage went crazy.
-
could this be related to libpfctl?
"...a new libpfctl package will build for 2.7.1 CE and 23.09 Plus."and I noticed that on 2.7.0 and xcp 8.2 suricata never seemed to crash, unlike Hyper-V.
-
This post is deleted! -
No Suricata or Snort here, but that does seem suspect. Interrupts are high even on fresh install. I didn't do any thorough testing but I did find it very peculiar that interrupts decreased with some WAN Tx throughput. If others observe the same it may help someone further narrow things down.
Booting with old kernel breaks WAN but fixes CPU issue, fwiw.
My current host is a HPE Proliant DL325 G10 Epyc 7351P running XCP-NG 8.2. Not sure about the NIC but I suspect they are 10Gbe ports, everything else here is 1Gbps. I made sure the VM had plenty of resources, and tried reinstalling guest additions and some other obvious things.
I'll be transitioning to a Ryzen host in the next couple weeks or so and will be sure to report anything interesting.
-
Same issue occurring with me. Fresh install of XCP-NG 8.2, single VM with pfsense, TCP offload disabled both in the XCP-ng host as well as pfsense interface.
Getting 60% interrupts at all times, goes down when loaded.
Interestingly, this seems to result in upload speeds of 300-400Mbit while downloads are 920Mbit on a 1Gb fiber WAN.
I really love the XCP-ng infrastructure but this is likely going to force me to switch to proxmox for these router + unifi controller builds. -
I can also confirm this is happening on XCP-ng 8.2 here is my
vmstat -iinterrupt total rate irq1: atkbd0 2 0 irq23: uhci0 16 0 cpu0:xen 143780 161 cpu1:xen 142710 160 irq2096: cpu0:r 298 0 irq2097: cpu0:itlb 6922 8 irq2098: cpu0:b 35713 40 irq2102: cpu1:r 115 0 irq2103: cpu1:itlb 7700 9 irq2104: cpu1:b 39100 44 irq2108: xen_et0:c0 86311 97 irq2109: xen_et0:c1 85965 96 irq2110: xenstore0 3020 3 irq2113: xbd0 11017 12 irq2114: xn0 819 1 irq2115: xn0 957 1 irq2116: xn0 1699 2 irq2117: xn0 4956 6 irq2118: xn1 1114 1 irq2119: xn1 130 0 irq2120: xn1 90 0 irq2121: xn1 418 0 irq2124: xn2 1200 1 irq2125: xn2 273 0 Total 574325 643 -
System referenced this topic on
-
Same issue with 2.7.1 on XCP-ng 8.2.1 Clean install for testing purposes, everything is default with the exception of DHCP service is disabled.
[2.7.2-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i interrupt total rate irq1: atkbd0 128 0 irq15: ata1 1375 0 irq23: uhci0 16 0 cpu0:xen 176561 47 cpu1:xen 152210 41 cpu2:xen 180816 48 cpu3:xen 141422 38 irq2096: cpu0:r 274 0 irq2097: cpu0:itlb 6901 2 irq2098: cpu0:b 71712 19 irq2102: cpu1:r 249 0 irq2103: cpu1:itlb 7447 2 irq2104: cpu1:b 77524 21 irq2108: cpu2:r 230 0 irq2109: cpu2:itlb 7074 2 irq2110: cpu2:b 76266 20 irq2114: cpu3:r 101 0 irq2115: cpu3:itlb 7389 2 irq2116: cpu3:b 74840 20 irq2120: xen_et0:c0 61626 16 irq2121: xen_et0:c1 44576 12 irq2122: xen_et0:c2 44672 12 irq2123: xen_et0:c3 44854 12 irq2124: xenstore0 1437 0 irq2129: xbd0 30860 8 irq2138: xn2 5051 1 irq2139: xn2 11971 3 irq2140: xn2 47811 13 irq2141: xn2 22958 6 irq2142: xn3 1783 0 irq2143: xn3 1866 0 irq2144: xn3 8840 2 irq2145: xn3 1782 0 Total 1312622 351 -
@onu sorry, 2.7.2 for pfSense
-
just to be thorough, I installed Proxmox on the same system, then pfSense as a VM.
way different results.
[2.7.2-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i interrupt total rate irq1: atkbd0 2 0 irq11: uhci0+ 20 0 irq15: ata1 449 0 cpu0:timer 19445 21 cpu1:timer 16727 18 cpu2:timer 14113 15 cpu3:timer 14768 16 irq27: virtio_pci1 12081 13 irq29: virtio_pci2 6783 7 irq30: virtio_pci2 1 0 irq32: virtio_pci3 194987 212 irq33: virtio_pci3 278 0 Total 279654 304 [2.7.2-RELEASE][admin@pfSense.home.arpa]/root: