High interrupt CPU usage in v2.7.1

[sknigen]

I have the same thing. high interruptions on 2.7.1 and XenServer 7.6

[sknigen]

After updating to 2.7.2 nothing has changed

[mdearman]

I can confirm having this issue on XCP-NG 8.2.1 running CE 2.7.2

[tsrats]

Can confirm, run many instances of pfSense, and am seeing the same thing with CE 2.7.2 on xcp-ng, but not on ESXi or Proxmox.

[cocojeff3]

I can confirm the issue is not limited to just VMs. I have tested a duo core Intel 3.0Ghz with 16 GB Ram and 160GB hard drive. As a test base config of 2.7.0 CPU was between 3-7 % moved same config to 2.7.1 and the CPU was between 30 - 60 % under the same load and configuration. moving to 2.7.2 changes nothing.

[Squish]

Same issue. I discovered that when any client is uploading over the WAN interface CPU usage actually drops drastically. From ~60% at idle to ~3% during light uploads.
Not really sure what that implies other than it possibly relating to interface interrupts.

Here is the output of vmstat -i

interrupt total rate
irq1: atkbd0 896 1
irq15: ata1 751 1
irq23: uhci0 16 0
cpu0:xen 318278 288
cpu1:xen 299866 271
cpu2:xen 291278 263
cpu3:xen 319322 289
cpu4:xen 306686 277
cpu5:xen 358225 324
cpu6:xen 343418 311
cpu7:xen 332859 301
irq2096: cpu0:r 268 0
irq2097: cpu0:itlb 11705 11
irq2098: cpu0:b 144030 130
irq2102: cpu1:r 275 0
irq2103: cpu1:itlb 11923 11
irq2104: cpu1:b 151704 137
irq2108: cpu2:r 327 0
irq2109: cpu2:itlb 12107 11
irq2110: cpu2:b 160624 145
irq2114: cpu3:r 296 0
irq2115: cpu3:itlb 11953 11
irq2116: cpu3:b 151243 137
irq2120: cpu4:r 257 0
irq2121: cpu4:itlb 12224 11
irq2122: cpu4:b 150400 136
irq2126: cpu5:r 245 0
irq2127: cpu5:itlb 11784 11
irq2128: cpu5:b 132161 120
irq2132: cpu6:r 259 0
irq2133: cpu6:itlb 11828 11
irq2134: cpu6:b 139764 126
irq2138: cpu7:r 129 0
irq2139: cpu7:itlb 12124 11
irq2140: cpu7:b 139024 126
irq2144: xen_et0:c0 23144 21
irq2145: xen_et0:c1 19830 18
irq2146: xen_et0:c2 17688 16
irq2147: xen_et0:c3 19772 18
irq2148: xen_et0:c4 17587 16
irq2149: xen_et0:c5 19831 18
irq2150: xen_et0:c6 18880 17
irq2151: xen_et0:c7 18379 17
irq2152: xenstore0 1699 2
irq2161: xbd0 5878 5
irq2162: xn0 134371 122
irq2163: xn0 149761 135
irq2164: xn0 112412 102
irq2165: xn0 128987 117
irq2166: xn1 157139 142
irq2167: xn1 183768 166
irq2168: xn1 193236 175
irq2169: xn1 216377 196
Total 5276988 4773

pfSense UI shows the same interface interrupt rates of 300-400/s+.

[jmperezbeth]

Exact same issue, several XenServer / Citrix Hypervisor VMs, upgraded from 2.7.0 to 2.7.2 and interrupt CPU usage went crazy.

[dmds]

could this be related to libpfctl?
"...a new libpfctl package will build for 2.7.1 CE and 23.09 Plus."

and I noticed that on 2.7.0 and xcp 8.2 suricata never seemed to crash, unlike Hyper-V.

[Squish]

This post is deleted!

[Squish]

No Suricata or Snort here, but that does seem suspect. Interrupts are high even on fresh install. I didn't do any thorough testing but I did find it very peculiar that interrupts decreased with some WAN Tx throughput. If others observe the same it may help someone further narrow things down.

Booting with old kernel breaks WAN but fixes CPU issue, fwiw.

My current host is a HPE Proliant DL325 G10 Epyc 7351P running XCP-NG 8.2. Not sure about the NIC but I suspect they are 10Gbe ports, everything else here is 1Gbps. I made sure the VM had plenty of resources, and tried reinstalling guest additions and some other obvious things.

I'll be transitioning to a Ryzen host in the next couple weeks or so and will be sure to report anything interesting.

[efny]

Same issue occurring with me. Fresh install of XCP-NG 8.2, single VM with pfsense, TCP offload disabled both in the XCP-ng host as well as pfsense interface.
Getting 60% interrupts at all times, goes down when loaded.
Interestingly, this seems to result in upload speeds of 300-400Mbit while downloads are 920Mbit on a 1Gb fiber WAN.
I really love the XCP-ng infrastructure but this is likely going to force me to switch to proxmox for these router + unifi controller builds.

[chigh09]

I can also confirm this is happening on XCP-ng 8.2 here is my vmstat -i

interrupt                          total       rate
irq1: atkbd0                           2          0
irq23: uhci0                          16          0
cpu0:xen                          143780        161
cpu1:xen                          142710        160
irq2096: cpu0:r                      298          0
irq2097: cpu0:itlb                  6922          8
irq2098: cpu0:b                    35713         40
irq2102: cpu1:r                      115          0
irq2103: cpu1:itlb                  7700          9
irq2104: cpu1:b                    39100         44
irq2108: xen_et0:c0                86311         97
irq2109: xen_et0:c1                85965         96
irq2110: xenstore0                  3020          3
irq2113: xbd0                      11017         12
irq2114: xn0                         819          1
irq2115: xn0                         957          1
irq2116: xn0                        1699          2
irq2117: xn0                        4956          6
irq2118: xn1                        1114          1
irq2119: xn1                         130          0
irq2120: xn1                          90          0
irq2121: xn1                         418          0
irq2124: xn2                        1200          1
irq2125: xn2                         273          0
Total                             574325        643

[onu]

Same issue with 2.7.1 on XCP-ng 8.2.1 Clean install for testing purposes, everything is default with the exception of DHCP service is disabled.

[2.7.2-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i
interrupt                          total       rate
irq1: atkbd0                         128          0
irq15: ata1                         1375          0
irq23: uhci0                          16          0
cpu0:xen                          176561         47
cpu1:xen                          152210         41
cpu2:xen                          180816         48
cpu3:xen                          141422         38
irq2096: cpu0:r                      274          0
irq2097: cpu0:itlb                  6901          2
irq2098: cpu0:b                    71712         19
irq2102: cpu1:r                      249          0
irq2103: cpu1:itlb                  7447          2
irq2104: cpu1:b                    77524         21
irq2108: cpu2:r                      230          0
irq2109: cpu2:itlb                  7074          2
irq2110: cpu2:b                    76266         20
irq2114: cpu3:r                      101          0
irq2115: cpu3:itlb                  7389          2
irq2116: cpu3:b                    74840         20
irq2120: xen_et0:c0                61626         16
irq2121: xen_et0:c1                44576         12
irq2122: xen_et0:c2                44672         12
irq2123: xen_et0:c3                44854         12
irq2124: xenstore0                  1437          0
irq2129: xbd0                      30860          8
irq2138: xn2                        5051          1
irq2139: xn2                       11971          3
irq2140: xn2                       47811         13
irq2141: xn2                       22958          6
irq2142: xn3                        1783          0
irq2143: xn3                        1866          0
irq2144: xn3                        8840          2
irq2145: xn3                        1782          0
Total                            1312622        351

[onu]

@onu sorry, 2.7.2 for pfSense

[onu]

just to be thorough, I installed Proxmox on the same system, then pfSense as a VM.

way different results.

[2.7.2-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i
interrupt                          total       rate
irq1: atkbd0                           2          0
irq11: uhci0+                         20          0
irq15: ata1                          449          0
cpu0:timer                         19445         21
cpu1:timer                         16727         18
cpu2:timer                         14113         15
cpu3:timer                         14768         16
irq27: virtio_pci1                 12081         13
irq29: virtio_pci2                  6783          7
irq30: virtio_pci2                     1          0
irq32: virtio_pci3                194987        212
irq33: virtio_pci3                   278          0
Total                             279654        304
[2.7.2-RELEASE][admin@pfSense.home.arpa]/root: 

[tmb]

I have the same issue. Running pfSense CE 2.7.2 in a VM on a XCP-ng host (8.2.1). Interrupt CPU usage is between 60-80%.

[tmb]

Some more data:

Top on the vm:

last pid: 23296;  load averages:  0.08,  0.14,  0.14                                                   up 0+18:46:31  10:20:03
286 threads:   2 running, 251 sleeping, 33 waiting
CPU:  0.0% user,  0.0% nice,  0.0% system, 84.0% interrupt, 16.0% idle
Mem: 85M Active, 142M Inact, 299M Wired, 56K Buf, 403M Free
ARC: 171M Total, 46M MFU, 118M MRU, 4096B Anon, 841K Header, 5776K Other
     143M Compressed, 278M Uncompressed, 1.94:1 Ratio
Swap: 1024M Total, 1024M Free

Top On the hypervisor:

top - 10:16:34 up 9 days, 12:45,  1 user,  load average: 651.11, 650.81, 649.76
Threads: 1763 total,   1 running, 1696 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.7 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.3 st
KiB Mem :  1329412 total,    39032 free,   892192 used,   398188 buff/cache
KiB Swap:  1048572 total,   951548 free,    97024 used.   277052 avail Mem 

vmstat -i

interrupt                          total       rate
irq1: atkbd0                        1094          0
irq15: ata1                        44975          1
irq23: uhci0                          20          0
cpu0:xen                        23137567        342
irq2102: xen_et0:c0              7093144        105
irq2103: xenstore0                  8039          0
irq2105: xbd0                     599431          9
irq2106: xn0                      925434         14
irq2107: xn0                      501940          7
irq2108: xn0                      437785          6
irq2109: xn0                     4301938         64
irq2110: xn1                     2861373         42
irq2111: xn1                     1348751         20
irq2112: xn1                     1813507         27
irq2113: xn1                     3541821         52
irq2114: xn2                       44963          1
irq2115: xn2                       15997          0
irq2116: xn2                       50798          1
irq2117: xn2                      133775          2
irq2121: xn3                           6          0
Total                           46862358        693

[Raghnall]

I am seeing the same thing, though it looks like it might be an accounting bug and not a real issue - see: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277231

Regardless, it does not seem to be related to the network cards or the version of Xen.
I see the same thing with only a physical Intel X520-DA2 passed through to the VM.
I also tested on XCP-ng 8.3-beta2 with Xen 4.13 as well as with Xen 4.17; all show the same issue.

[tmb]

@Raghnall Thank you for finding the bug report! I did search on the FreeBSD forums but couldn't find anything.

[pfsense2023]

Same thing here, running pfSense 2.7.2 on XCP-ng 8.3.0

It appears that this is just a bug on pfSense reporting itself because the VM stats show everything peacefully working.