<![CDATA[High Latency and Packet Loss]]>Hello,

We're having some trouble with packetloss and high latency on pfsense.

We are doing basically 2 tests from our LAN network:
1- Ping pfsense LAN interface.
2- Ping a published NAT address (the request go throught the LAN interface, go to firewall rule, makes NAT from our IP Address, go to firewall again and make NAT from the published IP address to the internal IP address from the published server).

When the firewall has few states:

  • Test 1 goes ok with low latency (- 1ms) and no packet loss. Test number 2 returns low latency (- 1ms) but with high packet loss (29%).

When the firewall has a lot of sessions (400.000 +):

  • Test 1 shows medium latency (+ 10 ms) and avarage packet loss (up to 8 %). Test number 2 goes really bad. A lot of packet loss (35+%) and high latency (+ 90 ms). Also, sometimes, pinging localhost (127.0.0.1) from the firewall shows also medium latency (~ 30 ms) and packet loss (~ 5%). We can also see packet drops in TCP sessions as well.

We can see increasing drops in net.inet.ip.intr_queue_drops. But if we increase net.inet.ip.intr_queue_maxlen things get worst… More packet drops and/or more latency as well...

Using Tcpdump we are able to see the ICMP request arriving at the OS, but there's no ICMP reply leaving.

Does anybody knows how to solve this? Thanks.

Pinging localhost

ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=30.987 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=31.615 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=24.459 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=29.403 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=31.975 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=30.762 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=30.252 ms

This is our configuartion:

uname -a

8.1-RELEASE-p6 FreeBSD 8.1-RELEASE-p6 #1: Mon Dec 12 18:43:24 EST 2011     root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8  amd64

sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'

hw.machine: amd64
hw.model: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
hw.ncpu: 8
hw.machine_arch: amd64

pciconf -lv

bce0@pci0:1:0:0:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
    class      = network
    subclass   = ethernet
bce1@pci0:1:0:1:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
    class      = network
    subclass   = ethernet
bce2@pci0:2:0:0:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
    class      = network
    subclass   = ethernet
bce3@pci0:2:0:1:        class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00
    class      = network
    subclass   = ethernet
igb0@pci0:7:0:0:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    class      = network
    subclass   = ethernet
igb1@pci0:7:0:1:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    class      = network
    subclass   = ethernet
igb2@pci0:8:0:0:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    class      = network
    subclass   = ethernet
igb3@pci0:8:0:1:        class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    class      = network
    subclass   = ethernet

vmstat -i

interrupt                          total       rate
irq1: atkbd0                           6          0
irq3: uart1                            6          0
irq16: mpt0                    246863035          8
irq19: ehci0                    37957316          1
irq21: uhci2 ehci1                  1313          0
irq23: atapci0                        68          0
cpu0: timer                  58571902608       1999
irq256: bce0                120034379327       4098
irq257: bce1                172847533185       5901
irq258: bce2                198187420291       6767
irq259: bce3                142394626104       4862
irq260: igb0:que 0          153250618590       5232
irq261: igb0:que 1                166550          0
irq262: igb0:que 2                177835          0
irq263: igb0:que 3              49628180          1
irq264: igb0:link                    431          0
irq269: igb1:link                      2          0
cpu1: timer                  58571893615       1999
cpu3: timer                  58571893616       1999
cpu2: timer                  58571893616       1999
cpu7: timer                  58571893616       1999
cpu5: timer                  58571893616       1999
cpu6: timer                  58571893616       1999
cpu4: timer                  58571893616       1999
Total                      1255624530158      42873

cat /boot/loader.conf

autoboot_delay="3"
vm.kmem_size="435544320"
vm.kmem_size_max="535544320"
kern.ipc.nmbclusters="65535"

Interface configuration:

  REDE_WIRELESS (lan)       -> igb1       -> 192.168.254.5
  INTERNET_OI (bce0)        -> bce0       -> 201.x.x.x
  INTERNET_EBT (bce1)       -> bce1       -> 201.x.x.x
  CORE_AV06 (bce2)          -> bce2       -> 10.247.1.2
  CORE_AV07 (bce3)          -> bce3       -> 10.247.1.6
  FAILOVER_SYNC (igb0)      -> igb0       -> 192.168.254.1

ifconfig:

bce0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:18
        inet 201.x.x.x netmask 0xfffffffc broadcast 201.x.x.x
        inet6 fe80::1a03:73ff:fef6:4118%bce0 prefixlen 64 scopeid 0x1
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c00b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1a
        inet 201.x.x.x netmask 0xfffffffc broadcast 201.x.x.x
        inet6 fe80::1a03:73ff:fef6:411a%bce1 prefixlen 64 scopeid 0x2
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c00b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1c
        inet 10.247.1.2 netmask 0xfffffffc broadcast 10.247.1.3
        inet6 fe80::1a03:73ff:fef6:411c%bce2 prefixlen 64 scopeid 0x3
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bce3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1e
        inet 10.247.1.6 netmask 0xfffffffc broadcast 10.247.1.7
        inet6 fe80::1a03:73ff:fef6:411e%bce3 prefixlen 64 scopeid 0x4
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum>ether 00:1b:21:2c:7f:70
        inet6 fe80::21b:21ff:fe2c:7f70%igb0 prefixlen 64 scopeid 0x5
        inet 192.168.254.1 netmask 0xfffffffc broadcast 192.168.254.3
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum>ether 00:1b:21:2c:7f:71
        inet 192.168.254.5 netmask 0xfffffffc broadcast 192.168.254.7
        inet6 fe80::21b:21ff:fe2c:7f71%igb1 prefixlen 64 scopeid 0x6
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect
        status: no carrier
igb2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:1b:21:2c:7f:74
        media: Ethernet autoselect
        status: no carrier
igb3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
        options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:1b:21:2c:7f:75
        media: Ethernet autoselect
        status: no carrier</rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast>

sysctl -a | grep buffer:

vfs.hifreebuffers: 5886
vfs.lofreebuffers: 2943
vfs.numfreebuffers: 52855
vfs.hidirtybuffers: 13241
vfs.lodirtybuffers: 6620
vfs.numdirtybuffers: 32
vfs.altbufferflushes: 0
vfs.dirtybufferflushes: 0
dev.bce.0.com_no_buffers: 0
dev.bce.1.com_no_buffers: 0
dev.bce.2.com_no_buffers: 0
dev.bce.3.com_no_buffers: 0

sysctl -a net.inet.ip.intr_queue_maxlen:

net.inet.ip.intr_queue_maxlen: 2000

sysctl -a net.inet.ip.intr_queue_drops:

net.inet.ip.intr_queue_drops: 1030600422

sysctl -a net.inet.ip.dummynet.io_pkt_drop:

net.inet.ip.dummynet.io_pkt_drop: 28969914416

top:

last pid: 16582;  load averages:  0.21,  0.28,  0.25  up 338+23:21:10 09:51:07
70 processes:  1 running, 69 sleeping
CPU:  0.3% user,  0.0% nice,  6.2% system, 14.0% interrupt, 79.5% idle
Mem: 185M Active, 872M Inact, 1133M Wired, 132K Cache, 826M Buf, 5705M Free
Swap:

Info on pfsense first page:


State table size 449347/786000
MBUF Usage	53846/65536

systat -ifstat 1:

                   /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |

      Interface           Traffic               Peak                Total
            lo0  in      0.000 KB/s          0.000 KB/s           23.719 MB
                 out     0.000 KB/s          0.000 KB/s           23.719 MB

           igb0  in      0.000 KB/s          0.000 KB/s            7.991 GB
                 out     0.000 KB/s          0.000 KB/s          132.728 TB

           bce3  in      5.291 MB/s          6.517 MB/s           51.852 TB
                 out     0.000 KB/s          0.082 KB/s          455.033 MB

           bce2  in      5.263 MB/s          6.136 MB/s           55.472 TB
                 out    53.853 MB/s         56.798 MB/s          400.682 TB

           bce1  in     26.486 MB/s         28.272 MB/s          227.789 TB
                 out     7.070 MB/s          8.967 MB/s           87.946 TB

           bce0  in     31.162 MB/s         34.104 MB/s          188.689 TB
                 out     2.390 MB/s          2.834 MB/s           16.662 TB

EDIT: TSO is not enabled. Added more info

]]>https://forum.netgate.com/topic/75073/high-latency-and-packet-lossRSS for NodeTue, 21 Jul 2026 01:22:32 GMTFri, 24 Oct 2014 12:18:05 GMT60<![CDATA[Reply to High Latency and Packet Loss on Wed, 29 Oct 2014 00:22:35 GMT]]>I just ran vmstat -i while I was running iperf against PFSense, and the interrupt rate was unflinching. A flat 40/core, for a total of 120/sec.

]]>https://forum.netgate.com/post/492282https://forum.netgate.com/post/492282Wed, 29 Oct 2014 00:22:35 GMT<![CDATA[Reply to High Latency and Packet Loss on Tue, 28 Oct 2014 15:05:04 GMT]]>Bits Per Second.

Mind you, I have a quite high end NIC, but a 50x difference seems quite crazy. I myself have little knowledge in these kinds of things. The only thing that comes to mind is you may want to look into polling. I know polling is many times used for situations where you can't get the interrupts down.

I hope someone with more knowledge responds, but I hope I gave you some ideas to look into.

Good Luck!

P.S. I think I may try an iperf when I get home, and let you know how it went for interrupt rates.

]]>https://forum.netgate.com/post/492125https://forum.netgate.com/post/492125Tue, 28 Oct 2014 15:05:04 GMT<![CDATA[Reply to High Latency and Packet Loss on Tue, 28 Oct 2014 13:41:51 GMT]]>@Harvy66:

I don't have much experience in this, but those BCE interrupts seem really high compared to what I'm seeing with my Intel NICs

irq256: bce0                120034379327       4098
irq257: bce1                172847533185       5901
irq258: bce2                198187420291       6767
irq259: bce3                142394626104       4862

This is what I am seeing on my box with an i350-T2 Intel NIC. I ran this while my connection was idle, under 1mb/s and when it was up around 150mb/s. Same results both times.

$ vmstat -i
interrupt                          total       rate
irq16: ehci0                     7723801          1
irq19: atapci0                   3550465          0
irq23: ehci1                     7717985          1
cpu0: timer                  10320023265       2000
irq256: igb0:que 0             219473396         42
irq257: igb0:que 1             213597353         41
irq258: igb0:que 2             205839895         39
irq259: igb0:que 3             221348363         42
irq260: igb0:link                      2          0
irq261: igb1:que 0             221394524         42
irq262: igb1:que 1             208434216         40
irq263: igb1:que 2             207530147         40
irq264: igb1:que 3             222207709         43
irq265: igb1:link                      7          0
cpu1: timer                  10320003271       2000
cpu2: timer                  10320003275       2000
cpu3: timer                  10320003275       2000
Total                        43018850949       8337

Harvy. Thank you very much for your reply!

You are talking about 150 Mbps (bits per second) ou 150 MBps (bytes per second)??

Do you know where can I have more information about appropriate values for interrupt rate and how to tune it?

Thank you again.

]]>https://forum.netgate.com/post/492107https://forum.netgate.com/post/492107Tue, 28 Oct 2014 13:41:51 GMT<![CDATA[Reply to High Latency and Packet Loss on Sat, 25 Oct 2014 16:41:54 GMT]]>I don't have much experience in this, but those BCE interrupts seem really high compared to what I'm seeing with my Intel NICs

irq256: bce0                120034379327       4098
irq257: bce1                172847533185       5901
irq258: bce2                198187420291       6767
irq259: bce3                142394626104       4862

This is what I am seeing on my box with an i350-T2 Intel NIC. I ran this while my connection was idle, under 1mb/s and when it was up around 150mb/s. Same results both times.

$ vmstat -i
interrupt                          total       rate
irq16: ehci0                     7723801          1
irq19: atapci0                   3550465          0
irq23: ehci1                     7717985          1
cpu0: timer                  10320023265       2000
irq256: igb0:que 0             219473396         42
irq257: igb0:que 1             213597353         41
irq258: igb0:que 2             205839895         39
irq259: igb0:que 3             221348363         42
irq260: igb0:link                      2          0
irq261: igb1:que 0             221394524         42
irq262: igb1:que 1             208434216         40
irq263: igb1:que 2             207530147         40
irq264: igb1:que 3             222207709         43
irq265: igb1:link                      7          0
cpu1: timer                  10320003271       2000
cpu2: timer                  10320003275       2000
cpu3: timer                  10320003275       2000
Total                        43018850949       8337
]]>https://forum.netgate.com/post/491617https://forum.netgate.com/post/491617Sat, 25 Oct 2014 16:41:54 GMT