High interrupts | KVM | Pass-thru NICs | 366FLR
-
Howdy,
I am getting high interrupt on hardware that should not be working this hard. I seem to be getting about 5K interrupts per second on a iperf3 test. Hardware Checksum Offloading and Hardware TCP Segmentation Offloading helped greatly lower the interrupt rate. Help?
- pfsense: 2.4.5-RELEASE-p1 (amd64)
- Unraid: 6.8.3 VM (KVM)
- Direct passthru of NICs
- pfatt.sh used for WAN
Hardware Checksum Offloading
☐
Hardware TCP Segmentation Offloading
☐
Hardware Large Receive Offloading
ifconfig igb2
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 48:df:37:d3:78:2f hwaddr 48:df:37:d3:78:2f inet6 fe80::4adf:37ff:fed3:782f%igb2 prefixlen 64 scopeid 0x3 inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 inet 10.0.0.2 netmask 0xffffffff broadcast 10.0.0.2 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: activesystat -vmstat 1
2 users Load 0.25 0.08 0.03 Jun 25 14:58 Mem usage: 16%Phy 1%Kmem Mem: KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 133036 12952 525452 16452 1675792 count All 138372 18156 563744 54604 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt ioflt 6236 total 66 22k 4 83k 5773 17 1 cow atkbd0 1 1 zfod pcib1 22 6.6%Sys 6.6%Intr 1.6%User 0.0%Nice 85.2%Idle ozfod xhci0 23 | | | | | | | | | | %ozfod 232 cpu0:timer ===++++ daefr 231 cpu1:timer 40 dtbuf prcfr 444 igb0:que 0 Namei Name-cache Dir-cache 112107 desvn totfr 55 igb0:que 1 Calls hits % hits % 1329 numvn react igb0:link 3 3 100 226 frevn pdwak 1 igb1:que 0 33 pdpgs 1 igb1:que 1 Disks md0 ada0 pass0 intrn igb1:link KB/t 0.00 0.00 0.00 152020 wire 5228 igb2:que 0 tps 0 0 0 81392 act 44 igb2:que 1 MB/s 0.00 0.00 0.00 75008 inact igb2:link %busy 0 0 0 laund ahci0 265 1675792 free 18492 bufvmmstat -i
interrupt total rate irq1: atkbd0 2 0 irq22: pcib1 1 0 irq23: xhci0 28 0 cpu0:timer 1347179 23 cpu1:timer 616159 11 irq256: igb0:que 0 8934693 153 irq257: igb0:que 1 4135406 71 irq258: igb0:link 2 0 irq259: igb1:que 0 59511 1 irq260: igb1:que 1 58464 1 irq261: igb1:link 2 0 irq262: igb2:que 0 11522908 197 irq263: igb2:que 1 4699425 80 irq264: igb2:link 3 0 irq265: ahci0 78091 1 Total 31451874 538top -aCHIPS
last pid: 4511; load averages: 0.39, 0.28, 0.19 up 0+16:15:01 14:52:49 162 processes: 4 running, 133 sleeping, 25 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 28.5% interrupt, 71.5% idle CPU 1: 1.8% user, 0.0% nice, 24.2% system, 0.0% interrupt, 73.9% idle Mem: 21M Active, 147M Inact, 324M Wired, 24M Buf, 1445M Free Swap: 512M Total, 512M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 11 root 155 ki31 0K 32K RUN 1 964:28 75.68% [idle{idle: cpu1}] 11 root 155 ki31 0K 32K CPU0 0 964:19 73.45% [idle{idle: cpu0}] 12 root -92 - 0K 400K WAIT 0 1:13 25.91% [intr{irq262: igb2:que 0}] 36924 root 37 0 9736K 5428K CPU0 0 0:19 23.88% /usr/local/bin/iperf3 -s 79946 www 20 0 18804K 13576K kqread 0 0:18 0.15% /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -x /tmp/haproxy.socket -sf 76560 7 root -16 - 0K 16K pftm 0 0:48 0.13% [pf purge] 70088 root 20 0 7820K 4140K CPU1 1 0:00 0.06% top -aCHIPS/boot/loader.config.local
# # Flow control # hw.igb.fc_setting=0 # # Set number of queues to number of cores divided by number of ports. 0 lets FreeBSD decide # hw.igb.num_queues=0 # # Increase packet descriptors (set as 1024,2048, or 4096) ONLY! # Allows a larger number of packets to be processed. # Use "netstat -ihw 1" in the shell and make sure the idrops are zero # If the NIC has constant disconnects, lower this value # if not zero then lower this value. hw.igb.rxd="2048" # For i340/i350 use 2048 hw.igb.txd="2048" # For i340/i350 use 2048 net.link.ifqmaxlen="4096" # value here equal sum of above values. For i340/i350 use 4096 # # Increase interuppt rate hw.igb.max_interrupt_rate="64000" # # Unlimited packet processing # Use this only if you are sure that the NICs have dedicated IRQs # View the IRQ assignments by executing this in the shell "vmstat -i" # A value of "-1" means unlimited packet processing # hw.igb.rx_process_limit="-1" hw.igb.tx_process_limit="-1" # #net.pf.source_nodes_hashsize="65536" # #net.pf.states_hashsize="2097152" # #net.inet.tcp.syncache.hashsize="2048" # #net.inet.tcp.syncache.bucketlimit="16" # #net.inet.tcp.syncache.cachelimit="32768" #pmcstat -TS instructions -w1
%SAMP IMAGE FUNCTION CALLERS 7.6 kernel bpf_filter bpf_mtap 5.2 kernel ether_nh_input netisr_dispatch_src 4.0 kernel pf_test pf_check_out:2.4 pf_check_in:1.7 4.0 kernel bzero pf_test:1.4 bpf_filter:0.8 pf_test_state_tcp:0.6 m_demote:0.6 m_pkthdr_init:0.6 3.0 kernel tcp_output tcp_do_segment 3.0 kernel acpi_timer_get_timec nanouptime:2.2 binuptime:0.7 2.8 kernel igb_rxeof igb_msix_que 2.8 kernel lock_delay __rw_wlock_hard 2.8 kernel __rw_rlock_int bpf_mtap 2.5 kernel copyout uiomove_faultflag 2.2 kernel ithread_loop fork_exit 2.2 libc.so.7 bsearch 0x622b 1.9 kernel pf_test_state_tcp pf_test 1.9 kernel tcp_do_segment tcp_input 1.8 kernel uma_zalloc_arg pf_test 1.7 kernel pf_normalize_ip pf_testiperf3 to router
iperf3 -c router -t 60 Connecting to host router, port 5201 [ 4] local 10.0.0.98 port 49264 connected to 10.0.0.1 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 114 MBytes 954 Mbits/sec 0 215 KBytes [ 4] 1.00-2.00 sec 113 MBytes 945 Mbits/sec 0 215 KBytes [ 4] 2.00-3.00 sec 112 MBytes 936 Mbits/sec 0 215 KBytes [ 4] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 215 KBytes [ 4] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 0 215 KBytes [ 4] 5.00-6.00 sec 112 MBytes 943 Mbits/sec 0 215 KBytes [ 4] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 0 215 KBytes [ 4] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 215 KBytes [ 4] 8.00-9.00 sec 112 MBytes 942 Mbits/sec 0 215 KBytes [ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 0 215 KBytes [ 4] 10.00-11.00 sec 93.8 MBytes 787 Mbits/sec 0 252 KBytes [ 4] 11.00-12.00 sec 112 MBytes 940 Mbits/sec 0 252 KBytes [ 4] 12.00-13.00 sec 112 MBytes 940 Mbits/sec 0 249 KBytes [ 4] 13.00-14.00 sec 112 MBytes 941 Mbits/sec 0 249 KBytes [ 4] 14.00-15.00 sec 112 MBytes 943 Mbits/sec 0 252 KBytes [ 4] 15.00-16.00 sec 112 MBytes 943 Mbits/sec 0 246 KBytes [ 4] 16.00-17.00 sec 112 MBytes 939 Mbits/sec 0 249 KBytes [ 4] 17.00-18.00 sec 113 MBytes 944 Mbits/sec 0 249 KBytes [ 4] 18.00-19.00 sec 112 MBytes 940 Mbits/sec 0 249 KBytes [ 4] 19.00-20.00 sec 112 MBytes 941 Mbits/sec 0 249 KBytes [ 4] 20.00-21.00 sec 95.0 MBytes 797 Mbits/sec 0 218 KBytes [ 4] 21.00-22.00 sec 112 MBytes 943 Mbits/sec 0 218 KBytes [ 4] 22.00-23.00 sec 113 MBytes 944 Mbits/sec 0 218 KBytes [ 4] 23.00-24.00 sec 111 MBytes 935 Mbits/sec 0 218 KBytes [ 4] 24.00-25.00 sec 112 MBytes 943 Mbits/sec 0 218 KBytes [ 4] 25.00-26.00 sec 112 MBytes 940 Mbits/sec 0 218 KBytes [ 4] 26.00-27.00 sec 112 MBytes 941 Mbits/sec 0 218 KBytes [ 4] 27.00-28.00 sec 112 MBytes 942 Mbits/sec 0 218 KBytes [ 4] 28.00-29.00 sec 113 MBytes 945 Mbits/sec 0 218 KBytes [ 4] 29.00-30.00 sec 112 MBytes 939 Mbits/sec 0 218 KBytes [ 4] 30.00-31.00 sec 93.3 MBytes 783 Mbits/sec 0 252 KBytes [ 4] 31.00-32.00 sec 112 MBytes 943 Mbits/sec 0 252 KBytes [ 4] 32.00-33.00 sec 112 MBytes 942 Mbits/sec 0 255 KBytes [ 4] 33.00-34.00 sec 112 MBytes 939 Mbits/sec 0 252 KBytes [ 4] 34.00-35.00 sec 113 MBytes 946 Mbits/sec 0 252 KBytes [ 4] 35.00-36.00 sec 112 MBytes 938 Mbits/sec 0 249 KBytes [ 4] 36.00-37.00 sec 113 MBytes 944 Mbits/sec 0 249 KBytes [ 4] 37.00-38.00 sec 112 MBytes 942 Mbits/sec 0 252 KBytes [ 4] 38.00-39.00 sec 112 MBytes 941 Mbits/sec 0 249 KBytes [ 4] 39.00-40.00 sec 112 MBytes 939 Mbits/sec 0 252 KBytes [ 4] 40.00-41.00 sec 95.2 MBytes 799 Mbits/sec 0 221 KBytes [ 4] 41.00-42.00 sec 112 MBytes 941 Mbits/sec 0 223 KBytes [ 4] 42.00-43.00 sec 112 MBytes 936 Mbits/sec 0 221 KBytes [ 4] 43.00-44.00 sec 112 MBytes 941 Mbits/sec 0 221 KBytes [ 4] 44.00-45.00 sec 113 MBytes 945 Mbits/sec 0 221 KBytes [ 4] 45.00-46.00 sec 112 MBytes 941 Mbits/sec 0 221 KBytes [ 4] 46.00-47.00 sec 112 MBytes 942 Mbits/sec 0 221 KBytes [ 4] 47.00-48.00 sec 112 MBytes 940 Mbits/sec 0 221 KBytes [ 4] 48.00-49.00 sec 112 MBytes 943 Mbits/sec 0 221 KBytes [ 4] 49.00-50.00 sec 112 MBytes 938 Mbits/sec 0 221 KBytes [ 4] 50.00-51.00 sec 96.9 MBytes 813 Mbits/sec 0 5.66 KBytes [ 4] 51.00-52.00 sec 111 MBytes 927 Mbits/sec 0 218 KBytes [ 4] 52.00-53.00 sec 113 MBytes 946 Mbits/sec 0 218 KBytes [ 4] 53.00-54.00 sec 112 MBytes 941 Mbits/sec 0 221 KBytes [ 4] 54.00-55.00 sec 112 MBytes 938 Mbits/sec 0 218 KBytes [ 4] 55.00-56.00 sec 113 MBytes 944 Mbits/sec 0 221 KBytes [ 4] 56.00-57.00 sec 112 MBytes 938 Mbits/sec 0 218 KBytes [ 4] 57.00-58.00 sec 112 MBytes 938 Mbits/sec 0 221 KBytes [ 4] 58.00-59.00 sec 112 MBytes 941 Mbits/sec 0 218 KBytes [ 4] 59.00-60.00 sec 112 MBytes 943 Mbits/sec 0 218 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 6.49 GBytes 929 Mbits/sec 0 sender [ 4] 0.00-60.00 sec 6.49 GBytes 929 Mbits/sec receiver -
Forgot to say, but all data above if when I was doing the iperf3 test to raise the interrupt rate.
-
I hand also an idea to migrate from physical hardware to virtualization the pfsense box .
Tried esxi an unraid , KVM with pass-trough , the results where horrible . Even on vmxnet3/virtio the results are poor .
The machine is capable of delivering 20Gb/s between 2 VM Linux hosts with vmxnet3 , but with pfsense in between max 700-800Mb :(With pfsense with passtrough I maxed out an i5-4570 with an i350-t4 adapter and did not reach gigabit speed(only pfsense was running on the hardware).On bare metal it works perfect
Now I'm running pfsense on E3-1220L v3 (13W)CPU bare metal and it still beats the crap of i5-4570 in virtualization
.Pfsense is simple not designed to run properly in virtualized env.