High interrupts | KVM | Pass-thru NICs | 366FLR



  • Howdy,

    I am getting high interrupt on hardware that should not be working this hard. I seem to be getting about 5K interrupts per second on a iperf3 test. Hardware Checksum Offloading and Hardware TCP Segmentation Offloading helped greatly lower the interrupt rate. Help?

    • pfsense: 2.4.5-RELEASE-p1 (amd64)
    • Unraid: 6.8.3 VM (KVM)
    • Direct passthru of NICs
    • pfatt.sh used for WAN

    Hardware Checksum Offloading

    Hardware TCP Segmentation Offloading

    Hardware Large Receive Offloading

    ☑

    ifconfig igb2

    igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=6503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    	ether 48:df:37:d3:78:2f
    	hwaddr 48:df:37:d3:78:2f
    	inet6 fe80::4adf:37ff:fed3:782f%igb2 prefixlen 64 scopeid 0x3
    	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    	inet 10.0.0.2 netmask 0xffffffff broadcast 10.0.0.2
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    

    systat -vmstat 1

        2 users    Load  0.25  0.08  0.03                  Jun 25 14:58
       Mem usage:  16%Phy  1%Kmem
    Mem: KB    REAL            VIRTUAL                      VN PAGER   SWAP PAGER
            Tot   Share      Tot    Share    Free           in   out     in   out
    Act  133036   12952   525452    16452 1675792  count
    All  138372   18156   563744    54604          pages
    Proc:                                                            Interrupts
      r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        ioflt  6236 total
                 66       22k    4  83k 5773   17    1        cow         atkbd0 1
                                                            1 zfod        pcib1 22
     6.6%Sys   6.6%Intr  1.6%User  0.0%Nice 85.2%Idle         ozfod       xhci0 23
    |    |    |    |    |    |    |    |    |    |           %ozfod   232 cpu0:timer
    ===++++                                                   daefr   231 cpu1:timer
                                            40 dtbuf          prcfr   444 igb0:que 0
    Namei     Name-cache   Dir-cache    112107 desvn          totfr    55 igb0:que 1
       Calls    hits   %    hits   %      1329 numvn          react       igb0:link
           3       3 100                   226 frevn          pdwak     1 igb1:que 0
                                                           33 pdpgs     1 igb1:que 1
    Disks   md0  ada0 pass0                                   intrn       igb1:link
    KB/t   0.00  0.00  0.00                            152020 wire   5228 igb2:que 0
    tps       0     0     0                             81392 act      44 igb2:que 1
    MB/s   0.00  0.00  0.00                             75008 inact       igb2:link
    %busy     0     0     0                                   laund       ahci0 265
                                                      1675792 free
                                                        18492 buf
    

    vmmstat -i

    interrupt                          total       rate
    irq1: atkbd0                           2          0
    irq22: pcib1                           1          0
    irq23: xhci0                          28          0
    cpu0:timer                       1347179         23
    cpu1:timer                        616159         11
    irq256: igb0:que 0               8934693        153
    irq257: igb0:que 1               4135406         71
    irq258: igb0:link                      2          0
    irq259: igb1:que 0                 59511          1
    irq260: igb1:que 1                 58464          1
    irq261: igb1:link                      2          0
    irq262: igb2:que 0              11522908        197
    irq263: igb2:que 1               4699425         80
    irq264: igb2:link                      3          0
    irq265: ahci0                      78091          1
    Total                           31451874        538
    

    top -aCHIPS

    last pid:  4511;  load averages:  0.39,  0.28,  0.19                                                                                                                                   up 0+16:15:01  14:52:49
    162 processes: 4 running, 133 sleeping, 25 waiting
    CPU 0:  0.0% user,  0.0% nice,  0.0% system, 28.5% interrupt, 71.5% idle
    CPU 1:  1.8% user,  0.0% nice, 24.2% system,  0.0% interrupt, 73.9% idle
    Mem: 21M Active, 147M Inact, 324M Wired, 24M Buf, 1445M Free
    Swap: 512M Total, 512M Free
    
      PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
       11 root       155 ki31     0K    32K RUN     1 964:28  75.68% [idle{idle: cpu1}]
       11 root       155 ki31     0K    32K CPU0    0 964:19  73.45% [idle{idle: cpu0}]
       12 root       -92    -     0K   400K WAIT    0   1:13  25.91% [intr{irq262: igb2:que 0}]
    36924 root        37    0  9736K  5428K CPU0    0   0:19  23.88% /usr/local/bin/iperf3 -s
    79946 www         20    0 18804K 13576K kqread  0   0:18   0.15% /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -x /tmp/haproxy.socket -sf 76560
        7 root       -16    -     0K    16K pftm    0   0:48   0.13% [pf purge]
    70088 root        20    0  7820K  4140K CPU1    1   0:00   0.06% top -aCHIPS
    

    /boot/loader.config.local

    #
    # Flow control
    #
    hw.igb.fc_setting=0
    #
    # Set number of queues to number of cores divided by number of ports. 0 lets FreeBSD decide
    #
    hw.igb.num_queues=0
    #
    # Increase packet descriptors (set as 1024,2048, or 4096) ONLY!
    # Allows a larger number of packets to be processed.
    # Use "netstat -ihw 1" in the shell and make sure the idrops are zero
    # If the NIC has constant disconnects, lower this value
    # if not zero then lower this value.
    hw.igb.rxd="2048" # For i340/i350 use 2048
    hw.igb.txd="2048" # For i340/i350 use 2048
    net.link.ifqmaxlen="4096" # value here equal sum of above values. For i340/i350 use 4096
    #
    # Increase interuppt rate
    hw.igb.max_interrupt_rate="64000"
    #
    # Unlimited packet processing
    # Use this only if you are sure that the NICs have dedicated IRQs
    # View the IRQ assignments by executing this in the shell "vmstat -i"
    # A value of "-1" means unlimited packet processing
    #
    hw.igb.rx_process_limit="-1"
    hw.igb.tx_process_limit="-1"
    #
    #net.pf.source_nodes_hashsize="65536"
    #
    #net.pf.states_hashsize="2097152"
    #
    #net.inet.tcp.syncache.hashsize="2048"
    #
    #net.inet.tcp.syncache.bucketlimit="16"
    #
    #net.inet.tcp.syncache.cachelimit="32768"
    #
    

    pmcstat -TS instructions -w1

    %SAMP IMAGE      FUNCTION             CALLERS
      7.6 kernel     bpf_filter           bpf_mtap
      5.2 kernel     ether_nh_input       netisr_dispatch_src
      4.0 kernel     pf_test              pf_check_out:2.4 pf_check_in:1.7
      4.0 kernel     bzero                pf_test:1.4 bpf_filter:0.8 pf_test_state_tcp:0.6 m_demote:0.6 m_pkthdr_init:0.6
      3.0 kernel     tcp_output           tcp_do_segment
      3.0 kernel     acpi_timer_get_timec nanouptime:2.2 binuptime:0.7
      2.8 kernel     igb_rxeof            igb_msix_que
      2.8 kernel     lock_delay           __rw_wlock_hard
      2.8 kernel     __rw_rlock_int       bpf_mtap
      2.5 kernel     copyout              uiomove_faultflag
      2.2 kernel     ithread_loop         fork_exit
      2.2 libc.so.7  bsearch              0x622b
      1.9 kernel     pf_test_state_tcp    pf_test
      1.9 kernel     tcp_do_segment       tcp_input
      1.8 kernel     uma_zalloc_arg       pf_test
      1.7 kernel     pf_normalize_ip      pf_test
    

    iperf3 to router

    iperf3 -c router -t 60
    Connecting to host router, port 5201
    [  4] local 10.0.0.98 port 49264 connected to 10.0.0.1 port 5201
    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
    [  4]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0    215 KBytes
    [  4]   1.00-2.00   sec   113 MBytes   945 Mbits/sec    0    215 KBytes
    [  4]   2.00-3.00   sec   112 MBytes   936 Mbits/sec    0    215 KBytes
    [  4]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    0    215 KBytes
    [  4]   4.00-5.00   sec   112 MBytes   943 Mbits/sec    0    215 KBytes
    [  4]   5.00-6.00   sec   112 MBytes   943 Mbits/sec    0    215 KBytes
    [  4]   6.00-7.00   sec   112 MBytes   941 Mbits/sec    0    215 KBytes
    [  4]   7.00-8.00   sec   112 MBytes   942 Mbits/sec    0    215 KBytes
    [  4]   8.00-9.00   sec   112 MBytes   942 Mbits/sec    0    215 KBytes
    [  4]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    215 KBytes
    [  4]  10.00-11.00  sec  93.8 MBytes   787 Mbits/sec    0    252 KBytes
    [  4]  11.00-12.00  sec   112 MBytes   940 Mbits/sec    0    252 KBytes
    [  4]  12.00-13.00  sec   112 MBytes   940 Mbits/sec    0    249 KBytes
    [  4]  13.00-14.00  sec   112 MBytes   941 Mbits/sec    0    249 KBytes
    [  4]  14.00-15.00  sec   112 MBytes   943 Mbits/sec    0    252 KBytes
    [  4]  15.00-16.00  sec   112 MBytes   943 Mbits/sec    0    246 KBytes
    [  4]  16.00-17.00  sec   112 MBytes   939 Mbits/sec    0    249 KBytes
    [  4]  17.00-18.00  sec   113 MBytes   944 Mbits/sec    0    249 KBytes
    [  4]  18.00-19.00  sec   112 MBytes   940 Mbits/sec    0    249 KBytes
    [  4]  19.00-20.00  sec   112 MBytes   941 Mbits/sec    0    249 KBytes
    [  4]  20.00-21.00  sec  95.0 MBytes   797 Mbits/sec    0    218 KBytes
    [  4]  21.00-22.00  sec   112 MBytes   943 Mbits/sec    0    218 KBytes
    [  4]  22.00-23.00  sec   113 MBytes   944 Mbits/sec    0    218 KBytes
    [  4]  23.00-24.00  sec   111 MBytes   935 Mbits/sec    0    218 KBytes
    [  4]  24.00-25.00  sec   112 MBytes   943 Mbits/sec    0    218 KBytes
    [  4]  25.00-26.00  sec   112 MBytes   940 Mbits/sec    0    218 KBytes
    [  4]  26.00-27.00  sec   112 MBytes   941 Mbits/sec    0    218 KBytes
    [  4]  27.00-28.00  sec   112 MBytes   942 Mbits/sec    0    218 KBytes
    [  4]  28.00-29.00  sec   113 MBytes   945 Mbits/sec    0    218 KBytes
    [  4]  29.00-30.00  sec   112 MBytes   939 Mbits/sec    0    218 KBytes
    [  4]  30.00-31.00  sec  93.3 MBytes   783 Mbits/sec    0    252 KBytes
    [  4]  31.00-32.00  sec   112 MBytes   943 Mbits/sec    0    252 KBytes
    [  4]  32.00-33.00  sec   112 MBytes   942 Mbits/sec    0    255 KBytes
    [  4]  33.00-34.00  sec   112 MBytes   939 Mbits/sec    0    252 KBytes
    [  4]  34.00-35.00  sec   113 MBytes   946 Mbits/sec    0    252 KBytes
    [  4]  35.00-36.00  sec   112 MBytes   938 Mbits/sec    0    249 KBytes
    [  4]  36.00-37.00  sec   113 MBytes   944 Mbits/sec    0    249 KBytes
    [  4]  37.00-38.00  sec   112 MBytes   942 Mbits/sec    0    252 KBytes
    [  4]  38.00-39.00  sec   112 MBytes   941 Mbits/sec    0    249 KBytes
    [  4]  39.00-40.00  sec   112 MBytes   939 Mbits/sec    0    252 KBytes
    [  4]  40.00-41.00  sec  95.2 MBytes   799 Mbits/sec    0    221 KBytes
    [  4]  41.00-42.00  sec   112 MBytes   941 Mbits/sec    0    223 KBytes
    [  4]  42.00-43.00  sec   112 MBytes   936 Mbits/sec    0    221 KBytes
    [  4]  43.00-44.00  sec   112 MBytes   941 Mbits/sec    0    221 KBytes
    [  4]  44.00-45.00  sec   113 MBytes   945 Mbits/sec    0    221 KBytes
    [  4]  45.00-46.00  sec   112 MBytes   941 Mbits/sec    0    221 KBytes
    [  4]  46.00-47.00  sec   112 MBytes   942 Mbits/sec    0    221 KBytes
    [  4]  47.00-48.00  sec   112 MBytes   940 Mbits/sec    0    221 KBytes
    [  4]  48.00-49.00  sec   112 MBytes   943 Mbits/sec    0    221 KBytes
    [  4]  49.00-50.00  sec   112 MBytes   938 Mbits/sec    0    221 KBytes
    [  4]  50.00-51.00  sec  96.9 MBytes   813 Mbits/sec    0   5.66 KBytes
    [  4]  51.00-52.00  sec   111 MBytes   927 Mbits/sec    0    218 KBytes
    [  4]  52.00-53.00  sec   113 MBytes   946 Mbits/sec    0    218 KBytes
    [  4]  53.00-54.00  sec   112 MBytes   941 Mbits/sec    0    221 KBytes
    [  4]  54.00-55.00  sec   112 MBytes   938 Mbits/sec    0    218 KBytes
    [  4]  55.00-56.00  sec   113 MBytes   944 Mbits/sec    0    221 KBytes
    [  4]  56.00-57.00  sec   112 MBytes   938 Mbits/sec    0    218 KBytes
    [  4]  57.00-58.00  sec   112 MBytes   938 Mbits/sec    0    221 KBytes
    [  4]  58.00-59.00  sec   112 MBytes   941 Mbits/sec    0    218 KBytes
    [  4]  59.00-60.00  sec   112 MBytes   943 Mbits/sec    0    218 KBytes
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth       Retr
    [  4]   0.00-60.00  sec  6.49 GBytes   929 Mbits/sec    0             sender
    [  4]   0.00-60.00  sec  6.49 GBytes   929 Mbits/sec                  receiver
    


  • Forgot to say, but all data above if when I was doing the iperf3 test to raise the interrupt rate.



  • I hand also an idea to migrate from physical hardware to virtualization the pfsense box .

    Tried esxi an unraid , KVM with pass-trough , the results where horrible . Even on vmxnet3/virtio the results are poor .
    The machine is capable of delivering 20Gb/s between 2 VM Linux hosts with vmxnet3 , but with pfsense in between max 700-800Mb :(

    With pfsense with passtrough I maxed out an i5-4570 with an i350-t4 adapter and did not reach gigabit speed(only pfsense was running on the hardware).On bare metal it works perfect

    Now I'm running pfsense on E3-1220L v3 (13W)CPU bare metal and it still beats the crap of i5-4570 in virtualization 😂 .

    Pfsense is simple not designed to run properly in virtualized env.


Log in to reply