Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    High interrupts | KVM | Pass-thru NICs | 366FLR

    Virtualization
    2
    3
    214
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      JasonSw last edited by

      Howdy,

      I am getting high interrupt on hardware that should not be working this hard. I seem to be getting about 5K interrupts per second on a iperf3 test. Hardware Checksum Offloading and Hardware TCP Segmentation Offloading helped greatly lower the interrupt rate. Help?

      • pfsense: 2.4.5-RELEASE-p1 (amd64)
      • Unraid: 6.8.3 VM (KVM)
      • Direct passthru of NICs
      • pfatt.sh used for WAN

      Hardware Checksum Offloading

      โ˜

      Hardware TCP Segmentation Offloading

      โ˜

      Hardware Large Receive Offloading

      โ˜‘

      ifconfig igb2

      igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
      	options=6503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
      	ether 48:df:37:d3:78:2f
      	hwaddr 48:df:37:d3:78:2f
      	inet6 fe80::4adf:37ff:fed3:782f%igb2 prefixlen 64 scopeid 0x3
      	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
      	inet 10.0.0.2 netmask 0xffffffff broadcast 10.0.0.2
      	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
      	media: Ethernet autoselect (1000baseT <full-duplex>)
      	status: active
      

      systat -vmstat 1

          2 users    Load  0.25  0.08  0.03                  Jun 25 14:58
         Mem usage:  16%Phy  1%Kmem
      Mem: KB    REAL            VIRTUAL                      VN PAGER   SWAP PAGER
              Tot   Share      Tot    Share    Free           in   out     in   out
      Act  133036   12952   525452    16452 1675792  count
      All  138372   18156   563744    54604          pages
      Proc:                                                            Interrupts
        r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        ioflt  6236 total
                   66       22k    4  83k 5773   17    1        cow         atkbd0 1
                                                              1 zfod        pcib1 22
       6.6%Sys   6.6%Intr  1.6%User  0.0%Nice 85.2%Idle         ozfod       xhci0 23
      |    |    |    |    |    |    |    |    |    |           %ozfod   232 cpu0:timer
      ===++++                                                   daefr   231 cpu1:timer
                                              40 dtbuf          prcfr   444 igb0:que 0
      Namei     Name-cache   Dir-cache    112107 desvn          totfr    55 igb0:que 1
         Calls    hits   %    hits   %      1329 numvn          react       igb0:link
             3       3 100                   226 frevn          pdwak     1 igb1:que 0
                                                             33 pdpgs     1 igb1:que 1
      Disks   md0  ada0 pass0                                   intrn       igb1:link
      KB/t   0.00  0.00  0.00                            152020 wire   5228 igb2:que 0
      tps       0     0     0                             81392 act      44 igb2:que 1
      MB/s   0.00  0.00  0.00                             75008 inact       igb2:link
      %busy     0     0     0                                   laund       ahci0 265
                                                        1675792 free
                                                          18492 buf
      

      vmmstat -i

      interrupt                          total       rate
      irq1: atkbd0                           2          0
      irq22: pcib1                           1          0
      irq23: xhci0                          28          0
      cpu0:timer                       1347179         23
      cpu1:timer                        616159         11
      irq256: igb0:que 0               8934693        153
      irq257: igb0:que 1               4135406         71
      irq258: igb0:link                      2          0
      irq259: igb1:que 0                 59511          1
      irq260: igb1:que 1                 58464          1
      irq261: igb1:link                      2          0
      irq262: igb2:que 0              11522908        197
      irq263: igb2:que 1               4699425         80
      irq264: igb2:link                      3          0
      irq265: ahci0                      78091          1
      Total                           31451874        538
      

      top -aCHIPS

      last pid:  4511;  load averages:  0.39,  0.28,  0.19                                                                                                                                   up 0+16:15:01  14:52:49
      162 processes: 4 running, 133 sleeping, 25 waiting
      CPU 0:  0.0% user,  0.0% nice,  0.0% system, 28.5% interrupt, 71.5% idle
      CPU 1:  1.8% user,  0.0% nice, 24.2% system,  0.0% interrupt, 73.9% idle
      Mem: 21M Active, 147M Inact, 324M Wired, 24M Buf, 1445M Free
      Swap: 512M Total, 512M Free
      
        PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
         11 root       155 ki31     0K    32K RUN     1 964:28  75.68% [idle{idle: cpu1}]
         11 root       155 ki31     0K    32K CPU0    0 964:19  73.45% [idle{idle: cpu0}]
         12 root       -92    -     0K   400K WAIT    0   1:13  25.91% [intr{irq262: igb2:que 0}]
      36924 root        37    0  9736K  5428K CPU0    0   0:19  23.88% /usr/local/bin/iperf3 -s
      79946 www         20    0 18804K 13576K kqread  0   0:18   0.15% /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -x /tmp/haproxy.socket -sf 76560
          7 root       -16    -     0K    16K pftm    0   0:48   0.13% [pf purge]
      70088 root        20    0  7820K  4140K CPU1    1   0:00   0.06% top -aCHIPS
      

      /boot/loader.config.local

      #
      # Flow control
      #
      hw.igb.fc_setting=0
      #
      # Set number of queues to number of cores divided by number of ports. 0 lets FreeBSD decide
      #
      hw.igb.num_queues=0
      #
      # Increase packet descriptors (set as 1024,2048, or 4096) ONLY!
      # Allows a larger number of packets to be processed.
      # Use "netstat -ihw 1" in the shell and make sure the idrops are zero
      # If the NIC has constant disconnects, lower this value
      # if not zero then lower this value.
      hw.igb.rxd="2048" # For i340/i350 use 2048
      hw.igb.txd="2048" # For i340/i350 use 2048
      net.link.ifqmaxlen="4096" # value here equal sum of above values. For i340/i350 use 4096
      #
      # Increase interuppt rate
      hw.igb.max_interrupt_rate="64000"
      #
      # Unlimited packet processing
      # Use this only if you are sure that the NICs have dedicated IRQs
      # View the IRQ assignments by executing this in the shell "vmstat -i"
      # A value of "-1" means unlimited packet processing
      #
      hw.igb.rx_process_limit="-1"
      hw.igb.tx_process_limit="-1"
      #
      #net.pf.source_nodes_hashsize="65536"
      #
      #net.pf.states_hashsize="2097152"
      #
      #net.inet.tcp.syncache.hashsize="2048"
      #
      #net.inet.tcp.syncache.bucketlimit="16"
      #
      #net.inet.tcp.syncache.cachelimit="32768"
      #
      

      pmcstat -TS instructions -w1

      %SAMP IMAGE      FUNCTION             CALLERS
        7.6 kernel     bpf_filter           bpf_mtap
        5.2 kernel     ether_nh_input       netisr_dispatch_src
        4.0 kernel     pf_test              pf_check_out:2.4 pf_check_in:1.7
        4.0 kernel     bzero                pf_test:1.4 bpf_filter:0.8 pf_test_state_tcp:0.6 m_demote:0.6 m_pkthdr_init:0.6
        3.0 kernel     tcp_output           tcp_do_segment
        3.0 kernel     acpi_timer_get_timec nanouptime:2.2 binuptime:0.7
        2.8 kernel     igb_rxeof            igb_msix_que
        2.8 kernel     lock_delay           __rw_wlock_hard
        2.8 kernel     __rw_rlock_int       bpf_mtap
        2.5 kernel     copyout              uiomove_faultflag
        2.2 kernel     ithread_loop         fork_exit
        2.2 libc.so.7  bsearch              0x622b
        1.9 kernel     pf_test_state_tcp    pf_test
        1.9 kernel     tcp_do_segment       tcp_input
        1.8 kernel     uma_zalloc_arg       pf_test
        1.7 kernel     pf_normalize_ip      pf_test
      

      iperf3 to router

      iperf3 -c router -t 60
      Connecting to host router, port 5201
      [  4] local 10.0.0.98 port 49264 connected to 10.0.0.1 port 5201
      [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
      [  4]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0    215 KBytes
      [  4]   1.00-2.00   sec   113 MBytes   945 Mbits/sec    0    215 KBytes
      [  4]   2.00-3.00   sec   112 MBytes   936 Mbits/sec    0    215 KBytes
      [  4]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    0    215 KBytes
      [  4]   4.00-5.00   sec   112 MBytes   943 Mbits/sec    0    215 KBytes
      [  4]   5.00-6.00   sec   112 MBytes   943 Mbits/sec    0    215 KBytes
      [  4]   6.00-7.00   sec   112 MBytes   941 Mbits/sec    0    215 KBytes
      [  4]   7.00-8.00   sec   112 MBytes   942 Mbits/sec    0    215 KBytes
      [  4]   8.00-9.00   sec   112 MBytes   942 Mbits/sec    0    215 KBytes
      [  4]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    215 KBytes
      [  4]  10.00-11.00  sec  93.8 MBytes   787 Mbits/sec    0    252 KBytes
      [  4]  11.00-12.00  sec   112 MBytes   940 Mbits/sec    0    252 KBytes
      [  4]  12.00-13.00  sec   112 MBytes   940 Mbits/sec    0    249 KBytes
      [  4]  13.00-14.00  sec   112 MBytes   941 Mbits/sec    0    249 KBytes
      [  4]  14.00-15.00  sec   112 MBytes   943 Mbits/sec    0    252 KBytes
      [  4]  15.00-16.00  sec   112 MBytes   943 Mbits/sec    0    246 KBytes
      [  4]  16.00-17.00  sec   112 MBytes   939 Mbits/sec    0    249 KBytes
      [  4]  17.00-18.00  sec   113 MBytes   944 Mbits/sec    0    249 KBytes
      [  4]  18.00-19.00  sec   112 MBytes   940 Mbits/sec    0    249 KBytes
      [  4]  19.00-20.00  sec   112 MBytes   941 Mbits/sec    0    249 KBytes
      [  4]  20.00-21.00  sec  95.0 MBytes   797 Mbits/sec    0    218 KBytes
      [  4]  21.00-22.00  sec   112 MBytes   943 Mbits/sec    0    218 KBytes
      [  4]  22.00-23.00  sec   113 MBytes   944 Mbits/sec    0    218 KBytes
      [  4]  23.00-24.00  sec   111 MBytes   935 Mbits/sec    0    218 KBytes
      [  4]  24.00-25.00  sec   112 MBytes   943 Mbits/sec    0    218 KBytes
      [  4]  25.00-26.00  sec   112 MBytes   940 Mbits/sec    0    218 KBytes
      [  4]  26.00-27.00  sec   112 MBytes   941 Mbits/sec    0    218 KBytes
      [  4]  27.00-28.00  sec   112 MBytes   942 Mbits/sec    0    218 KBytes
      [  4]  28.00-29.00  sec   113 MBytes   945 Mbits/sec    0    218 KBytes
      [  4]  29.00-30.00  sec   112 MBytes   939 Mbits/sec    0    218 KBytes
      [  4]  30.00-31.00  sec  93.3 MBytes   783 Mbits/sec    0    252 KBytes
      [  4]  31.00-32.00  sec   112 MBytes   943 Mbits/sec    0    252 KBytes
      [  4]  32.00-33.00  sec   112 MBytes   942 Mbits/sec    0    255 KBytes
      [  4]  33.00-34.00  sec   112 MBytes   939 Mbits/sec    0    252 KBytes
      [  4]  34.00-35.00  sec   113 MBytes   946 Mbits/sec    0    252 KBytes
      [  4]  35.00-36.00  sec   112 MBytes   938 Mbits/sec    0    249 KBytes
      [  4]  36.00-37.00  sec   113 MBytes   944 Mbits/sec    0    249 KBytes
      [  4]  37.00-38.00  sec   112 MBytes   942 Mbits/sec    0    252 KBytes
      [  4]  38.00-39.00  sec   112 MBytes   941 Mbits/sec    0    249 KBytes
      [  4]  39.00-40.00  sec   112 MBytes   939 Mbits/sec    0    252 KBytes
      [  4]  40.00-41.00  sec  95.2 MBytes   799 Mbits/sec    0    221 KBytes
      [  4]  41.00-42.00  sec   112 MBytes   941 Mbits/sec    0    223 KBytes
      [  4]  42.00-43.00  sec   112 MBytes   936 Mbits/sec    0    221 KBytes
      [  4]  43.00-44.00  sec   112 MBytes   941 Mbits/sec    0    221 KBytes
      [  4]  44.00-45.00  sec   113 MBytes   945 Mbits/sec    0    221 KBytes
      [  4]  45.00-46.00  sec   112 MBytes   941 Mbits/sec    0    221 KBytes
      [  4]  46.00-47.00  sec   112 MBytes   942 Mbits/sec    0    221 KBytes
      [  4]  47.00-48.00  sec   112 MBytes   940 Mbits/sec    0    221 KBytes
      [  4]  48.00-49.00  sec   112 MBytes   943 Mbits/sec    0    221 KBytes
      [  4]  49.00-50.00  sec   112 MBytes   938 Mbits/sec    0    221 KBytes
      [  4]  50.00-51.00  sec  96.9 MBytes   813 Mbits/sec    0   5.66 KBytes
      [  4]  51.00-52.00  sec   111 MBytes   927 Mbits/sec    0    218 KBytes
      [  4]  52.00-53.00  sec   113 MBytes   946 Mbits/sec    0    218 KBytes
      [  4]  53.00-54.00  sec   112 MBytes   941 Mbits/sec    0    221 KBytes
      [  4]  54.00-55.00  sec   112 MBytes   938 Mbits/sec    0    218 KBytes
      [  4]  55.00-56.00  sec   113 MBytes   944 Mbits/sec    0    221 KBytes
      [  4]  56.00-57.00  sec   112 MBytes   938 Mbits/sec    0    218 KBytes
      [  4]  57.00-58.00  sec   112 MBytes   938 Mbits/sec    0    221 KBytes
      [  4]  58.00-59.00  sec   112 MBytes   941 Mbits/sec    0    218 KBytes
      [  4]  59.00-60.00  sec   112 MBytes   943 Mbits/sec    0    218 KBytes
      - - - - - - - - - - - - - - - - - - - - - - - - -
      [ ID] Interval           Transfer     Bandwidth       Retr
      [  4]   0.00-60.00  sec  6.49 GBytes   929 Mbits/sec    0             sender
      [  4]   0.00-60.00  sec  6.49 GBytes   929 Mbits/sec                  receiver
      
      1 Reply Last reply Reply Quote 0
      • J
        JasonSw last edited by

        Forgot to say, but all data above if when I was doing the iperf3 test to raise the interrupt rate.

        1 Reply Last reply Reply Quote 0
        • C
          CIURI86 last edited by

          I hand also an idea to migrate from physical hardware to virtualization the pfsense box .

          Tried esxi an unraid , KVM with pass-trough , the results where horrible . Even on vmxnet3/virtio the results are poor .
          The machine is capable of delivering 20Gb/s between 2 VM Linux hosts with vmxnet3 , but with pfsense in between max 700-800Mb :(

          With pfsense with passtrough I maxed out an i5-4570 with an i350-t4 adapter and did not reach gigabit speed(only pfsense was running on the hardware).On bare metal it works perfect

          Now I'm running pfsense on E3-1220L v3 (13W)CPU bare metal and it still beats the crap of i5-4570 in virtualization ๐Ÿ˜‚ .

          Pfsense is simple not designed to run properly in virtualized env.

          1 Reply Last reply Reply Quote 0
          • First post
            Last post