High Latency and Packet Loss



  • Hello,

    We're having some trouble with packetloss and high latency on pfsense.

    We are doing basically 2 tests from our LAN network:
    1- Ping pfsense LAN interface.
    2- Ping a published NAT address (the request go throught the LAN interface, go to firewall rule, makes NAT from our IP Address, go to firewall again and make NAT from the published IP address to the internal IP address from the published server).

    When the firewall has few states:

    • Test 1 goes ok with low latency (- 1ms) and no packet loss. Test number 2 returns low latency (- 1ms) but with high packet loss (29%).

    When the firewall has a lot of sessions (400.000 +):

    • Test 1 shows medium latency (+ 10 ms) and avarage packet loss (up to 8 %). Test number 2 goes really bad. A lot of packet loss (35+%) and high latency (+ 90 ms). Also, sometimes, pinging localhost (127.0.0.1) from the firewall shows also medium latency (~ 30 ms) and packet loss (~ 5%). We can also see packet drops in TCP sessions as well.

    We can see increasing drops in net.inet.ip.intr_queue_drops. But if we increase net.inet.ip.intr_queue_maxlen things get worst… More packet drops and/or more latency as well...

    Using Tcpdump we are able to see the ICMP request arriving at the OS, but there's no ICMP reply leaving.

    Does anybody knows how to solve this? Thanks.

    Pinging localhost

    ping 127.0.0.1
    PING 127.0.0.1 (127.0.0.1): 56 data bytes
    64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=30.987 ms
    64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=31.615 ms
    64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=24.459 ms
    64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=29.403 ms
    64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=31.975 ms
    64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=30.762 ms
    64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=30.252 ms
    

    This is our configuartion:

    uname -a

    8.1-RELEASE-p6 FreeBSD 8.1-RELEASE-p6 #1: Mon Dec 12 18:43:24 EST 2011     root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8  amd64
    

    sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'

    hw.machine: amd64
    hw.model: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
    hw.ncpu: 8
    hw.machine_arch: amd64
    

    pciconf -lv

    bce0@pci0:1:0:0:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
        class      = network
        subclass   = ethernet
    bce1@pci0:1:0:1:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
        class      = network
        subclass   = ethernet
    bce2@pci0:2:0:0:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
        class      = network
        subclass   = ethernet
    bce3@pci0:2:0:1:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
        class      = network
        subclass   = ethernet
    igb0@pci0:7:0:0:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
        class      = network
        subclass   = ethernet
    igb1@pci0:7:0:1:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
        class      = network
        subclass   = ethernet
    igb2@pci0:8:0:0:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
        class      = network
        subclass   = ethernet
    igb3@pci0:8:0:1:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
        class      = network
        subclass   = ethernet
    

    vmstat -i

    interrupt                          total       rate
    irq1: atkbd0                           6          0
    irq3: uart1                            6          0
    irq16: mpt0                    246863035          8
    irq19: ehci0                    37957316          1
    irq21: uhci2 ehci1                  1313          0
    irq23: atapci0                        68          0
    cpu0: timer                  58571902608       1999
    irq256: bce0                120034379327       4098
    irq257: bce1                172847533185       5901
    irq258: bce2                198187420291       6767
    irq259: bce3                142394626104       4862
    irq260: igb0:que 0          153250618590       5232
    irq261: igb0:que 1                166550          0
    irq262: igb0:que 2                177835          0
    irq263: igb0:que 3              49628180          1
    irq264: igb0:link                    431          0
    irq269: igb1:link                      2          0
    cpu1: timer                  58571893615       1999
    cpu3: timer                  58571893616       1999
    cpu2: timer                  58571893616       1999
    cpu7: timer                  58571893616       1999
    cpu5: timer                  58571893616       1999
    cpu6: timer                  58571893616       1999
    cpu4: timer                  58571893616       1999
    Total                      1255624530158      42873
    

    cat /boot/loader.conf

    autoboot_delay="3"
    vm.kmem_size="435544320"
    vm.kmem_size_max="535544320"
    kern.ipc.nmbclusters="65535"
    
    

    Interface configuration:

      REDE_WIRELESS (lan)       -> igb1       -> 192.168.254.5
      INTERNET_OI (bce0)        -> bce0       -> 201.x.x.x
      INTERNET_EBT (bce1)       -> bce1       -> 201.x.x.x
      CORE_AV06 (bce2)          -> bce2       -> 10.247.1.2
      CORE_AV07 (bce3)          -> bce3       -> 10.247.1.6
      FAILOVER_SYNC (igb0)      -> igb0       -> 192.168.254.1
    

    ifconfig:

    bce0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:18
            inet 201.x.x.x netmask 0xfffffffc broadcast 201.x.x.x
            inet6 fe80::1a03:73ff:fef6:4118%bce0 prefixlen 64 scopeid 0x1
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    bce1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=c00b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1a
            inet 201.x.x.x netmask 0xfffffffc broadcast 201.x.x.x
            inet6 fe80::1a03:73ff:fef6:411a%bce1 prefixlen 64 scopeid 0x2
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    bce2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=c00b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1c
            inet 10.247.1.2 netmask 0xfffffffc broadcast 10.247.1.3
            inet6 fe80::1a03:73ff:fef6:411c%bce2 prefixlen 64 scopeid 0x3
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    bce3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1e
            inet 10.247.1.6 netmask 0xfffffffc broadcast 10.247.1.7
            inet6 fe80::1a03:73ff:fef6:411e%bce3 prefixlen 64 scopeid 0x4
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    igb0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum>ether 00:1b:21:2c:7f:70
            inet6 fe80::21b:21ff:fe2c:7f70%igb0 prefixlen 64 scopeid 0x5
            inet 192.168.254.1 netmask 0xfffffffc broadcast 192.168.254.3
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum>ether 00:1b:21:2c:7f:71
            inet 192.168.254.5 netmask 0xfffffffc broadcast 192.168.254.7
            inet6 fe80::21b:21ff:fe2c:7f71%igb1 prefixlen 64 scopeid 0x6
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect
            status: no carrier
    igb2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
            options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:1b:21:2c:7f:74
            media: Ethernet autoselect
            status: no carrier
    igb3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
            options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:1b:21:2c:7f:75
            media: Ethernet autoselect
            status: no carrier</rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast>
    

    sysctl -a | grep buffer:

    vfs.hifreebuffers: 5886
    vfs.lofreebuffers: 2943
    vfs.numfreebuffers: 52855
    vfs.hidirtybuffers: 13241
    vfs.lodirtybuffers: 6620
    vfs.numdirtybuffers: 32
    vfs.altbufferflushes: 0
    vfs.dirtybufferflushes: 0
    dev.bce.0.com_no_buffers: 0
    dev.bce.1.com_no_buffers: 0
    dev.bce.2.com_no_buffers: 0
    dev.bce.3.com_no_buffers: 0
    

    sysctl -a net.inet.ip.intr_queue_maxlen:

    net.inet.ip.intr_queue_maxlen: 2000
    

    sysctl -a net.inet.ip.intr_queue_drops:

    net.inet.ip.intr_queue_drops: 1030600422
    

    sysctl -a net.inet.ip.dummynet.io_pkt_drop:

    net.inet.ip.dummynet.io_pkt_drop: 28969914416
    

    top:

    last pid: 16582;  load averages:  0.21,  0.28,  0.25  up 338+23:21:10 09:51:07
    70 processes:  1 running, 69 sleeping
    CPU:  0.3% user,  0.0% nice,  6.2% system, 14.0% interrupt, 79.5% idle
    Mem: 185M Active, 872M Inact, 1133M Wired, 132K Cache, 826M Buf, 5705M Free
    Swap:
    

    Info on pfsense first page:

    
    State table size 449347/786000
    MBUF Usage	53846/65536
    
    

    systat -ifstat 1:

                       /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
         Load Average   |
    
          Interface           Traffic               Peak                Total
                lo0  in      0.000 KB/s          0.000 KB/s           23.719 MB
                     out     0.000 KB/s          0.000 KB/s           23.719 MB
    
               igb0  in      0.000 KB/s          0.000 KB/s            7.991 GB
                     out     0.000 KB/s          0.000 KB/s          132.728 TB
    
               bce3  in      5.291 MB/s          6.517 MB/s           51.852 TB
                     out     0.000 KB/s          0.082 KB/s          455.033 MB
    
               bce2  in      5.263 MB/s          6.136 MB/s           55.472 TB
                     out    53.853 MB/s         56.798 MB/s          400.682 TB
    
               bce1  in     26.486 MB/s         28.272 MB/s          227.789 TB
                     out     7.070 MB/s          8.967 MB/s           87.946 TB
    
               bce0  in     31.162 MB/s         34.104 MB/s          188.689 TB
                     out     2.390 MB/s          2.834 MB/s           16.662 TB
    
    

    EDIT: TSO is not enabled. Added more info



  • I don't have much experience in this, but those BCE interrupts seem really high compared to what I'm seeing with my Intel NICs

    irq256: bce0                120034379327       4098
    irq257: bce1                172847533185       5901
    irq258: bce2                198187420291       6767
    irq259: bce3                142394626104       4862
    

    This is what I am seeing on my box with an i350-T2 Intel NIC. I ran this while my connection was idle, under 1mb/s and when it was up around 150mb/s. Same results both times.

    $ vmstat -i
    interrupt                          total       rate
    irq16: ehci0                     7723801          1
    irq19: atapci0                   3550465          0
    irq23: ehci1                     7717985          1
    cpu0: timer                  10320023265       2000
    irq256: igb0:que 0             219473396         42
    irq257: igb0:que 1             213597353         41
    irq258: igb0:que 2             205839895         39
    irq259: igb0:que 3             221348363         42
    irq260: igb0:link                      2          0
    irq261: igb1:que 0             221394524         42
    irq262: igb1:que 1             208434216         40
    irq263: igb1:que 2             207530147         40
    irq264: igb1:que 3             222207709         43
    irq265: igb1:link                      7          0
    cpu1: timer                  10320003271       2000
    cpu2: timer                  10320003275       2000
    cpu3: timer                  10320003275       2000
    Total                        43018850949       8337
    


  • @Harvy66:

    I don't have much experience in this, but those BCE interrupts seem really high compared to what I'm seeing with my Intel NICs

    irq256: bce0                120034379327       4098
    irq257: bce1                172847533185       5901
    irq258: bce2                198187420291       6767
    irq259: bce3                142394626104       4862
    

    This is what I am seeing on my box with an i350-T2 Intel NIC. I ran this while my connection was idle, under 1mb/s and when it was up around 150mb/s. Same results both times.

    $ vmstat -i
    interrupt                          total       rate
    irq16: ehci0                     7723801          1
    irq19: atapci0                   3550465          0
    irq23: ehci1                     7717985          1
    cpu0: timer                  10320023265       2000
    irq256: igb0:que 0             219473396         42
    irq257: igb0:que 1             213597353         41
    irq258: igb0:que 2             205839895         39
    irq259: igb0:que 3             221348363         42
    irq260: igb0:link                      2          0
    irq261: igb1:que 0             221394524         42
    irq262: igb1:que 1             208434216         40
    irq263: igb1:que 2             207530147         40
    irq264: igb1:que 3             222207709         43
    irq265: igb1:link                      7          0
    cpu1: timer                  10320003271       2000
    cpu2: timer                  10320003275       2000
    cpu3: timer                  10320003275       2000
    Total                        43018850949       8337
    

    Harvy. Thank you very much for your reply!

    You are talking about 150 Mbps (bits per second) ou 150 MBps (bytes per second)??

    Do you know where can I have more information about appropriate values for interrupt rate and how to tune it?

    Thank you again.



  • Bits Per Second.

    Mind you, I have a quite high end NIC, but a 50x difference seems quite crazy. I myself have little knowledge in these kinds of things. The only thing that comes to mind is you may want to look into polling. I know polling is many times used for situations where you can't get the interrupts down.

    I hope someone with more knowledge responds, but I hope I gave you some ideas to look into.

    Good Luck!

    P.S. I think I may try an iperf when I get home, and let you know how it went for interrupt rates.



  • I just ran vmstat -i while I was running iperf against PFSense, and the interrupt rate was unflinching. A flat 40/core, for a total of 120/sec.