High Latency and Packet Loss
-
Hello,
We're having some trouble with packetloss and high latency on pfsense.
We are doing basically 2 tests from our LAN network:
1- Ping pfsense LAN interface.
2- Ping a published NAT address (the request go throught the LAN interface, go to firewall rule, makes NAT from our IP Address, go to firewall again and make NAT from the published IP address to the internal IP address from the published server).When the firewall has few states:
- Test 1 goes ok with low latency (- 1ms) and no packet loss. Test number 2 returns low latency (- 1ms) but with high packet loss (29%).
When the firewall has a lot of sessions (400.000 +):
- Test 1 shows medium latency (+ 10 ms) and avarage packet loss (up to 8 %). Test number 2 goes really bad. A lot of packet loss (35+%) and high latency (+ 90 ms). Also, sometimes, pinging localhost (127.0.0.1) from the firewall shows also medium latency (~ 30 ms) and packet loss (~ 5%). We can also see packet drops in TCP sessions as well.
We can see increasing drops in net.inet.ip.intr_queue_drops. But if we increase net.inet.ip.intr_queue_maxlen things get worst… More packet drops and/or more latency as well...
Using Tcpdump we are able to see the ICMP request arriving at the OS, but there's no ICMP reply leaving.
Does anybody knows how to solve this? Thanks.
Pinging localhost
ping 127.0.0.1 PING 127.0.0.1 (127.0.0.1): 56 data bytes 64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=30.987 ms 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=31.615 ms 64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=24.459 ms 64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=29.403 ms 64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=31.975 ms 64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=30.762 ms 64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=30.252 msThis is our configuartion:
uname -a
8.1-RELEASE-p6 FreeBSD 8.1-RELEASE-p6 #1: Mon Dec 12 18:43:24 EST 2011 root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'
hw.machine: amd64 hw.model: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz hw.ncpu: 8 hw.machine_arch: amd64pciconf -lv
bce0@pci0:1:0:0: class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00 class = network subclass = ethernet bce1@pci0:1:0:1: class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00 class = network subclass = ethernet bce2@pci0:2:0:0: class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00 class = network subclass = ethernet bce3@pci0:2:0:1: class=0x020000 card=0x02361028 chip=0x163914e4 rev=0x20 hdr=0x00 class = network subclass = ethernet igb0@pci0:7:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 class = network subclass = ethernet igb1@pci0:7:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 class = network subclass = ethernet igb2@pci0:8:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 class = network subclass = ethernet igb3@pci0:8:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00 class = network subclass = ethernetvmstat -i
interrupt total rate irq1: atkbd0 6 0 irq3: uart1 6 0 irq16: mpt0 246863035 8 irq19: ehci0 37957316 1 irq21: uhci2 ehci1 1313 0 irq23: atapci0 68 0 cpu0: timer 58571902608 1999 irq256: bce0 120034379327 4098 irq257: bce1 172847533185 5901 irq258: bce2 198187420291 6767 irq259: bce3 142394626104 4862 irq260: igb0:que 0 153250618590 5232 irq261: igb0:que 1 166550 0 irq262: igb0:que 2 177835 0 irq263: igb0:que 3 49628180 1 irq264: igb0:link 431 0 irq269: igb1:link 2 0 cpu1: timer 58571893615 1999 cpu3: timer 58571893616 1999 cpu2: timer 58571893616 1999 cpu7: timer 58571893616 1999 cpu5: timer 58571893616 1999 cpu6: timer 58571893616 1999 cpu4: timer 58571893616 1999 Total 1255624530158 42873cat /boot/loader.conf
autoboot_delay="3" vm.kmem_size="435544320" vm.kmem_size_max="535544320" kern.ipc.nmbclusters="65535"Interface configuration:
REDE_WIRELESS (lan) -> igb1 -> 192.168.254.5 INTERNET_OI (bce0) -> bce0 -> 201.x.x.x INTERNET_EBT (bce1) -> bce1 -> 201.x.x.x CORE_AV06 (bce2) -> bce2 -> 10.247.1.2 CORE_AV07 (bce3) -> bce3 -> 10.247.1.6 FAILOVER_SYNC (igb0) -> igb0 -> 192.168.254.1ifconfig:
bce0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:18 inet 201.x.x.x netmask 0xfffffffc broadcast 201.x.x.x inet6 fe80::1a03:73ff:fef6:4118%bce0 prefixlen 64 scopeid 0x1 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active bce1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=c00b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1a inet 201.x.x.x netmask 0xfffffffc broadcast 201.x.x.x inet6 fe80::1a03:73ff:fef6:411a%bce1 prefixlen 64 scopeid 0x2 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active bce2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=c00b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1c inet 10.247.1.2 netmask 0xfffffffc broadcast 10.247.1.3 inet6 fe80::1a03:73ff:fef6:411c%bce2 prefixlen 64 scopeid 0x3 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active bce3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=c00bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate>ether 18:03:73:f6:41:1e inet 10.247.1.6 netmask 0xfffffffc broadcast 10.247.1.7 inet6 fe80::1a03:73ff:fef6:411e%bce3 prefixlen 64 scopeid 0x4 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active igb0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum>ether 00:1b:21:2c:7f:70 inet6 fe80::21b:21ff:fe2c:7f70%igb0 prefixlen 64 scopeid 0x5 inet 192.168.254.1 netmask 0xfffffffc broadcast 192.168.254.3 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>) status: active igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum>ether 00:1b:21:2c:7f:71 inet 192.168.254.5 netmask 0xfffffffc broadcast 192.168.254.7 inet6 fe80::21b:21ff:fe2c:7f71%igb1 prefixlen 64 scopeid 0x6 nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect status: no carrier igb2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:1b:21:2c:7f:74 media: Ethernet autoselect status: no carrier igb3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=1bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4>ether 00:1b:21:2c:7f:75 media: Ethernet autoselect status: no carrier</rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4></broadcast,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast>sysctl -a | grep buffer:
vfs.hifreebuffers: 5886 vfs.lofreebuffers: 2943 vfs.numfreebuffers: 52855 vfs.hidirtybuffers: 13241 vfs.lodirtybuffers: 6620 vfs.numdirtybuffers: 32 vfs.altbufferflushes: 0 vfs.dirtybufferflushes: 0 dev.bce.0.com_no_buffers: 0 dev.bce.1.com_no_buffers: 0 dev.bce.2.com_no_buffers: 0 dev.bce.3.com_no_buffers: 0sysctl -a net.inet.ip.intr_queue_maxlen:
net.inet.ip.intr_queue_maxlen: 2000sysctl -a net.inet.ip.intr_queue_drops:
net.inet.ip.intr_queue_drops: 1030600422sysctl -a net.inet.ip.dummynet.io_pkt_drop:
net.inet.ip.dummynet.io_pkt_drop: 28969914416top:
last pid: 16582; load averages: 0.21, 0.28, 0.25 up 338+23:21:10 09:51:07 70 processes: 1 running, 69 sleeping CPU: 0.3% user, 0.0% nice, 6.2% system, 14.0% interrupt, 79.5% idle Mem: 185M Active, 872M Inact, 1133M Wired, 132K Cache, 826M Buf, 5705M Free Swap:Info on pfsense first page:
State table size 449347/786000 MBUF Usage 53846/65536systat -ifstat 1:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 Load Average | Interface Traffic Peak Total lo0 in 0.000 KB/s 0.000 KB/s 23.719 MB out 0.000 KB/s 0.000 KB/s 23.719 MB igb0 in 0.000 KB/s 0.000 KB/s 7.991 GB out 0.000 KB/s 0.000 KB/s 132.728 TB bce3 in 5.291 MB/s 6.517 MB/s 51.852 TB out 0.000 KB/s 0.082 KB/s 455.033 MB bce2 in 5.263 MB/s 6.136 MB/s 55.472 TB out 53.853 MB/s 56.798 MB/s 400.682 TB bce1 in 26.486 MB/s 28.272 MB/s 227.789 TB out 7.070 MB/s 8.967 MB/s 87.946 TB bce0 in 31.162 MB/s 34.104 MB/s 188.689 TB out 2.390 MB/s 2.834 MB/s 16.662 TBEDIT: TSO is not enabled. Added more info
-
I don't have much experience in this, but those BCE interrupts seem really high compared to what I'm seeing with my Intel NICs
irq256: bce0 120034379327 4098 irq257: bce1 172847533185 5901 irq258: bce2 198187420291 6767 irq259: bce3 142394626104 4862This is what I am seeing on my box with an i350-T2 Intel NIC. I ran this while my connection was idle, under 1mb/s and when it was up around 150mb/s. Same results both times.
$ vmstat -i interrupt total rate irq16: ehci0 7723801 1 irq19: atapci0 3550465 0 irq23: ehci1 7717985 1 cpu0: timer 10320023265 2000 irq256: igb0:que 0 219473396 42 irq257: igb0:que 1 213597353 41 irq258: igb0:que 2 205839895 39 irq259: igb0:que 3 221348363 42 irq260: igb0:link 2 0 irq261: igb1:que 0 221394524 42 irq262: igb1:que 1 208434216 40 irq263: igb1:que 2 207530147 40 irq264: igb1:que 3 222207709 43 irq265: igb1:link 7 0 cpu1: timer 10320003271 2000 cpu2: timer 10320003275 2000 cpu3: timer 10320003275 2000 Total 43018850949 8337 -
I don't have much experience in this, but those BCE interrupts seem really high compared to what I'm seeing with my Intel NICs
irq256: bce0 120034379327 4098 irq257: bce1 172847533185 5901 irq258: bce2 198187420291 6767 irq259: bce3 142394626104 4862This is what I am seeing on my box with an i350-T2 Intel NIC. I ran this while my connection was idle, under 1mb/s and when it was up around 150mb/s. Same results both times.
$ vmstat -i interrupt total rate irq16: ehci0 7723801 1 irq19: atapci0 3550465 0 irq23: ehci1 7717985 1 cpu0: timer 10320023265 2000 irq256: igb0:que 0 219473396 42 irq257: igb0:que 1 213597353 41 irq258: igb0:que 2 205839895 39 irq259: igb0:que 3 221348363 42 irq260: igb0:link 2 0 irq261: igb1:que 0 221394524 42 irq262: igb1:que 1 208434216 40 irq263: igb1:que 2 207530147 40 irq264: igb1:que 3 222207709 43 irq265: igb1:link 7 0 cpu1: timer 10320003271 2000 cpu2: timer 10320003275 2000 cpu3: timer 10320003275 2000 Total 43018850949 8337Harvy. Thank you very much for your reply!
You are talking about 150 Mbps (bits per second) ou 150 MBps (bytes per second)??
Do you know where can I have more information about appropriate values for interrupt rate and how to tune it?
Thank you again.
-
Bits Per Second.
Mind you, I have a quite high end NIC, but a 50x difference seems quite crazy. I myself have little knowledge in these kinds of things. The only thing that comes to mind is you may want to look into polling. I know polling is many times used for situations where you can't get the interrupts down.
I hope someone with more knowledge responds, but I hope I gave you some ideas to look into.
Good Luck!
P.S. I think I may try an iperf when I get home, and let you know how it went for interrupt rates.
-
I just ran vmstat -i while I was running iperf against PFSense, and the interrupt rate was unflinching. A flat 40/core, for a total of 120/sec.