High interrupt CPU usage in v2.7.1
-
Exact same issue, several XenServer / Citrix Hypervisor VMs, upgraded from 2.7.0 to 2.7.2 and interrupt CPU usage went crazy.
-
could this be related to libpfctl?
"...a new libpfctl package will build for 2.7.1 CE and 23.09 Plus."and I noticed that on 2.7.0 and xcp 8.2 suricata never seemed to crash, unlike Hyper-V.
-
This post is deleted! -
No Suricata or Snort here, but that does seem suspect. Interrupts are high even on fresh install. I didn't do any thorough testing but I did find it very peculiar that interrupts decreased with some WAN Tx throughput. If others observe the same it may help someone further narrow things down.
Booting with old kernel breaks WAN but fixes CPU issue, fwiw.
My current host is a HPE Proliant DL325 G10 Epyc 7351P running XCP-NG 8.2. Not sure about the NIC but I suspect they are 10Gbe ports, everything else here is 1Gbps. I made sure the VM had plenty of resources, and tried reinstalling guest additions and some other obvious things.
I'll be transitioning to a Ryzen host in the next couple weeks or so and will be sure to report anything interesting.
-
Same issue occurring with me. Fresh install of XCP-NG 8.2, single VM with pfsense, TCP offload disabled both in the XCP-ng host as well as pfsense interface.
Getting 60% interrupts at all times, goes down when loaded.
Interestingly, this seems to result in upload speeds of 300-400Mbit while downloads are 920Mbit on a 1Gb fiber WAN.
I really love the XCP-ng infrastructure but this is likely going to force me to switch to proxmox for these router + unifi controller builds. -
I can also confirm this is happening on XCP-ng 8.2 here is my
vmstat -iinterrupt total rate irq1: atkbd0 2 0 irq23: uhci0 16 0 cpu0:xen 143780 161 cpu1:xen 142710 160 irq2096: cpu0:r 298 0 irq2097: cpu0:itlb 6922 8 irq2098: cpu0:b 35713 40 irq2102: cpu1:r 115 0 irq2103: cpu1:itlb 7700 9 irq2104: cpu1:b 39100 44 irq2108: xen_et0:c0 86311 97 irq2109: xen_et0:c1 85965 96 irq2110: xenstore0 3020 3 irq2113: xbd0 11017 12 irq2114: xn0 819 1 irq2115: xn0 957 1 irq2116: xn0 1699 2 irq2117: xn0 4956 6 irq2118: xn1 1114 1 irq2119: xn1 130 0 irq2120: xn1 90 0 irq2121: xn1 418 0 irq2124: xn2 1200 1 irq2125: xn2 273 0 Total 574325 643 -
System referenced this topic on
-
Same issue with 2.7.1 on XCP-ng 8.2.1 Clean install for testing purposes, everything is default with the exception of DHCP service is disabled.
[2.7.2-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i interrupt total rate irq1: atkbd0 128 0 irq15: ata1 1375 0 irq23: uhci0 16 0 cpu0:xen 176561 47 cpu1:xen 152210 41 cpu2:xen 180816 48 cpu3:xen 141422 38 irq2096: cpu0:r 274 0 irq2097: cpu0:itlb 6901 2 irq2098: cpu0:b 71712 19 irq2102: cpu1:r 249 0 irq2103: cpu1:itlb 7447 2 irq2104: cpu1:b 77524 21 irq2108: cpu2:r 230 0 irq2109: cpu2:itlb 7074 2 irq2110: cpu2:b 76266 20 irq2114: cpu3:r 101 0 irq2115: cpu3:itlb 7389 2 irq2116: cpu3:b 74840 20 irq2120: xen_et0:c0 61626 16 irq2121: xen_et0:c1 44576 12 irq2122: xen_et0:c2 44672 12 irq2123: xen_et0:c3 44854 12 irq2124: xenstore0 1437 0 irq2129: xbd0 30860 8 irq2138: xn2 5051 1 irq2139: xn2 11971 3 irq2140: xn2 47811 13 irq2141: xn2 22958 6 irq2142: xn3 1783 0 irq2143: xn3 1866 0 irq2144: xn3 8840 2 irq2145: xn3 1782 0 Total 1312622 351 -
@onu sorry, 2.7.2 for pfSense
-
just to be thorough, I installed Proxmox on the same system, then pfSense as a VM.
way different results.
[2.7.2-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i interrupt total rate irq1: atkbd0 2 0 irq11: uhci0+ 20 0 irq15: ata1 449 0 cpu0:timer 19445 21 cpu1:timer 16727 18 cpu2:timer 14113 15 cpu3:timer 14768 16 irq27: virtio_pci1 12081 13 irq29: virtio_pci2 6783 7 irq30: virtio_pci2 1 0 irq32: virtio_pci3 194987 212 irq33: virtio_pci3 278 0 Total 279654 304 [2.7.2-RELEASE][admin@pfSense.home.arpa]/root: -
I have the same issue. Running pfSense CE 2.7.2 in a VM on a XCP-ng host (8.2.1). Interrupt CPU usage is between 60-80%.
-
Some more data:
Topon the vm:last pid: 23296; load averages: 0.08, 0.14, 0.14 up 0+18:46:31 10:20:03 286 threads: 2 running, 251 sleeping, 33 waiting CPU: 0.0% user, 0.0% nice, 0.0% system, 84.0% interrupt, 16.0% idle Mem: 85M Active, 142M Inact, 299M Wired, 56K Buf, 403M Free ARC: 171M Total, 46M MFU, 118M MRU, 4096B Anon, 841K Header, 5776K Other 143M Compressed, 278M Uncompressed, 1.94:1 Ratio Swap: 1024M Total, 1024M FreeTopOn the hypervisor:top - 10:16:34 up 9 days, 12:45, 1 user, load average: 651.11, 650.81, 649.76 Threads: 1763 total, 1 running, 1696 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.4 us, 0.7 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.3 st KiB Mem : 1329412 total, 39032 free, 892192 used, 398188 buff/cache KiB Swap: 1048572 total, 951548 free, 97024 used. 277052 avail Memvmstat -iinterrupt total rate irq1: atkbd0 1094 0 irq15: ata1 44975 1 irq23: uhci0 20 0 cpu0:xen 23137567 342 irq2102: xen_et0:c0 7093144 105 irq2103: xenstore0 8039 0 irq2105: xbd0 599431 9 irq2106: xn0 925434 14 irq2107: xn0 501940 7 irq2108: xn0 437785 6 irq2109: xn0 4301938 64 irq2110: xn1 2861373 42 irq2111: xn1 1348751 20 irq2112: xn1 1813507 27 irq2113: xn1 3541821 52 irq2114: xn2 44963 1 irq2115: xn2 15997 0 irq2116: xn2 50798 1 irq2117: xn2 133775 2 irq2121: xn3 6 0 Total 46862358 693 -
I am seeing the same thing, though it looks like it might be an accounting bug and not a real issue - see: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277231
Regardless, it does not seem to be related to the network cards or the version of Xen.
I see the same thing with only a physical Intel X520-DA2 passed through to the VM.
I also tested on XCP-ng 8.3-beta2 with Xen 4.13 as well as with Xen 4.17; all show the same issue. -
@Raghnall Thank you for finding the bug report! I did search on the FreeBSD forums but couldn't find anything.
-
Same thing here, running pfSense 2.7.2 on XCP-ng 8.3.0
It appears that this is just a bug on pfSense reporting itself because the VM stats show everything peacefully working.
-
In top it shows high interrupt
-
S stephenw10 referenced this topic on