EM0 High Interrupt, Enabling polling brings down the interface. 2.0.2-RELEASE



  • Hey all,

    I am using 2.0.2-RELEASE (i386), and I have a Intel Pro Gigabit card, using the EM driver. When I turn on Polling, from the router, I can send about 7 packets to another host with ping before the connection drops. I think that I am getting a watchdog timeout error. With the polling disabled, it works fine, but generates high interrupts on a gigabit network. Please advise.

    [2.0.2-RELEASE][admin@router.*****.com]/root(1): dmesg

    Copyright © 1992-2010 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 8.1-RELEASE-p13 #0: Fri Dec  7 16:51:57 EST 2012
        root@snapshots-8_1-i386.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 i386
    Timecounter "i8254" frequency 1193182 Hz quality 0
    CPU: Intel(R) Celeron(R) CPU 3.06GHz (3066.79-MHz 686-class CPU)
      Origin = "GenuineIntel"  Id = 0xf49  Family = f  Model = 4  Stepping = 9
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x651d <sse3,dtes64,mon,ds_cpl,tm2,cnxt-id,cx16,xtpr>AMD Features=0x20000000 <lm>TSC: P-state invariant
    real memory  = 536870912 (512 MB)
    avail memory = 468828160 (447 MB)
    ACPI APIC Table: <award  awrdacpi="">ioapic0 <version 1.1="">irqs 0-23 on motherboard
    wpi: You need to read the LICENSE file in /usr/share/doc/legal/intel_wpi/.
    wpi: If you agree with the license, set legal.intel_wpi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (wpi_fw, 0xc098a6c0, 0) error 1
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xc078b700, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc078b7a0, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc078b840, 0) error 1
    wlan: mac acl policy registered
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <award awrdacpi="">on motherboard
    acpi0: [ITHREAD]
    acpi0: Power Button (fixed)
    acpi0: reservation of 0, a0000 (3) failed
    acpi0: reservation of 100000, 1def0000 (3) failed
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
    cpu0: <acpi cpu="">on acpi0
    acpi_button0: <power button="">on acpi0
    acpi_button1: <sleep button="">on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff,0x480-0x48f,0x1000-0x10df,0x10e0-0x10ff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    agp0: <sis 661="" host="" to="" agp="" bridge="">on hostb0
    pcib1: <pci-pci bridge="">at device 1.0 on pci0
    pci1: <pci bus="">on pcib1
    vgapci0: <vga-compatible display="">port 0x9000-0x907f mem 0xd8000000-0xdfffffff,0xe1000000-0xe101ffff at device 0.0 on pci1
    isab0: <pci-isa bridge="">at device 2.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <sis 964="" udma133="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x4000-0x400f at device 2.5 on pci0
    ata0: <ata 0="" channel="">on atapci0
    ata0: [ITHREAD]
    ata1: <ata 1="" channel="">on atapci0
    ata1: [ITHREAD]
    pci0: <multimedia, audio="">at device 2.7 (no driver attached)
    ohci0: <sis 5571="" usb="" controller="">mem 0xe1142000-0xe1142fff irq 20 at device 3.0 on pci0
    ohci0: [ITHREAD]
    usbus0: <sis 5571="" usb="" controller="">on ohci0
    ohci1: <sis 5571="" usb="" controller="">mem 0xe1143000-0xe1143fff irq 21 at device 3.1 on pci0
    ohci1: [ITHREAD]
    usbus1: <sis 5571="" usb="" controller="">on ohci1
    ohci2: <sis 5571="" usb="" controller="">mem 0xe1144000-0xe1144fff irq 22 at device 3.2 on pci0
    ohci2: [ITHREAD]
    usbus2: <sis 5571="" usb="" controller="">on ohci2
    ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xe1140000-0xe1140fff irq 23 at device 3.3 on pci0
    ehci0: [ITHREAD]
    usbus3: EHCI version 1.0
    usbus3: <ehci (generic)="" usb="" 2.0="" controller="">on ehci0
    sis0: <sis 10="" 900="" 100basetx="">port 0xa800-0xa8ff mem 0xe1141000-0xe1141fff irq 19 at device 4.0 on pci0
    miibus0: <mii bus="">on sis0
    rlphy0: <rtl8201l 10="" 100="" media="" interface="">PHY 9 on miibus0
    rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    sis0: [ITHREAD]
    atapci1: <sis 964="" udma133="" controller="">port 0xac00-0xac07,0xb000-0xb003,0xb400-0xb407,0xb800-0xb803,0xbc00-0xbc0f irq 17 at device 5.0 on pci0
    atapci1: [ITHREAD]
    ata2: <ata 0="" channel="">on atapci1
    ata2: [ITHREAD]
    ata3: <ata 1="" channel="">on atapci1
    ata3: [ITHREAD]
    em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0xc000-0xc03f mem 0xe1100000-0xe111ffff,0xe1120000-0xe113ffff irq 18 at device 10.0 on pci0
    em0: [FILTER]
    acpi_tz0: <thermal zone="">on acpi0
    atrtc0: <at realtime="" clock="">port 0x70-0x73 irq 8 on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart0: [FILTER]
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
    uart1: [FILTER]
    ppc0: <parallel port="">port 0x378-0x37f,0x778-0x77b irq 7 drq 3 on acpi0
    ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
    ppc0: FIFO with 16/16/16 bytes threshold
    ppc0: [ITHREAD]
    ppbus0: <parallel port="" bus="">on ppc0
    plip0: <plip network="" interface="">on ppbus0
    plip0: [ITHREAD]
    lpt0: <printer>on ppbus0
    lpt0: [ITHREAD]
    lpt0: Interrupt-driven port
    ppi0: <parallel i="" o="">on ppbus0
    atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    atkbd0: [ITHREAD]
    psm0: <ps 2="" mouse="">irq 12 on atkbdc0
    psm0: [GIANT-LOCKED]
    psm0: [ITHREAD]
    psm0: model IntelliMouse Explorer, device ID 4
    pmtimer0 on isa0
    orm0: <isa option="" roms="">at iomem 0xc0000-0xc7fff,0xd0000-0xd17ff,0xef000-0xeffff pnpid ORM0000 on isa0
    sc0: <system console="">at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
    Timecounter "TSC" frequency 3066792481 Hz quality 800
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 12Mbps Full Speed USB v1.0
    usbus2: 12Mbps Full Speed USB v1.0
    usbus3: 480Mbps High Speed USB v2.0
    ugen0.1: <sis>at usbus0
    uhub0: <sis 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
    ugen1.1: <sis>at usbus1
    uhub1: <sis 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
    ugen2.1: <sis>at usbus2
    uhub2: <sis 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
    ugen3.1: <sis>at usbus3
    uhub3: <sis 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus3
    acd0: DMA limited to UDMA33, device found non-ATA66 cable
    acd0: DVDR <lite-on dvdrw="" shw-160p6s="" prs2="">at ata1-master UDMA33
    ad4: DMA limited to UDMA33, device found non-ATA66 cable
    ad4: 78533MB <hitachi hds728080pla380="" pf2oa60a="">at ata2-master UDMA33
    Root mount waiting for: usbus3 usbus2 usbus1 usbus0
    uhub2: 2 ports with 2 removable, self powered
    uhub0: 3 ports with 3 removable, self powered
    uhub1: 3 ports with 3 removable, self powered
    Root mount waiting for: usbus3
    Root mount waiting for: usbus3
    Root mount waiting for: usbus3
    uhub3: 8 ports with 8 removable, self powered
    Trying to mount root from ufs:/dev/ad4s1a
    pflog0: promiscuous mode enabled
    sis0: link state changed to UP
    em0: link state changed to UP
    em0: Watchdog timeout – resetting
    em0: link state changed to DOWN
    em0: link state changed to UP
    em0: Watchdog timeout -- resetting
    em0: link state changed to DOWN
    em0: link state changed to UP
    sis0: watchdog timeout
    arp: 10.1.10.11 moved from 00:19:d1:e1:7a:1f to 00:40:ca:72:5b:ee on em0
    arp: 10.1.10.11 moved from 00:40:ca:72:5b:ee to 00:19:d1:e1:7a:1f on em0

    [2.0.2-RELEASE][admin@router.*****.com]/root(2): ifconfig -a

    sis0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=80008 <vlan_mtu,linkstate>ether 00:16:ec:5b:2d:d7
    inet 173.14.10.137 netmask 0xfffffffc broadcast 173.14.10.139
    inet6 fe80::216:ecff:fe5b:2dd7%sis0 prefixlen 64 scopeid 0x1
    nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:07:e9:0f:f3:50
    inet 10.1.10.1 netmask 0xffffff00 broadcast 10.1.10.255
    inet6 fe80::207:e9ff:fe0f:f350%em0 prefixlen 64 scopeid 0x2
    nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
    pfsync0: flags=0<> metric 0 mtu 1460
    syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
    pflog0: flags=100 <promisc>metric 0 mtu 33200
    enc0: flags=0<> metric 0 mtu 1536
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
    options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
    nd6 options=43 <performnud,accept_rtadv></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></vlan_mtu,linkstate></up,broadcast,running,simplex,multicast></hitachi></lite-on></sis></sis></sis></sis></sis></sis></sis></sis></cpu></generic></system></isa></ps></at></keyboard></parallel></printer></plip></parallel></parallel></at></thermal></intel(r)></ata></ata></sis></rtl8201l></mii></sis></ehci></ehci></sis></sis></sis></sis></sis></sis></multimedia,></ata></ata></sis></isa></pci-isa></vga-compatible></pci></pci-pci></sis></acpi></acpi></sleep></power></acpi></award></software></version></award ></lm></sse3,dtes64,mon,ds_cpl,tm2,cnxt-id,cx16,xtpr></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>



  • A snapshot build of pfSense 2.1 might work better for you. Snapshot builds of pfSense 2.1 have more up to date device drivers than build of pfSense 2.0.x.



  • I just did the update to the latest snapshot 2.1 there was. Problem is still occuring. On a machine that I have pinging the router…

    I am only testing the polling using the command ifconfig em0 polling and ifconfig em0 -polling

    64 bytes from 10.1.10.1: icmp_req=399 ttl=64 time=0.204 ms
    64 bytes from 10.1.10.1: icmp_req=400 ttl=64 time=0.207 ms
    64 bytes from 10.1.10.1: icmp_req=401 ttl=64 time=0.257 ms
    64 bytes from 10.1.10.1: icmp_req=402 ttl=64 time=0.105 ms
    64 bytes from 10.1.10.1: icmp_req=403 ttl=64 time=0.167 ms
    64 bytes from 10.1.10.1: icmp_req=404 ttl=64 time=0.221 ms

    Enable Polling

    64 bytes from 10.1.10.1: icmp_req=405 ttl=64 time=11085 ms
    64 bytes from 10.1.10.1: icmp_req=406 ttl=64 time=10085 ms
    64 bytes from 10.1.10.1: icmp_req=407 ttl=64 time=9085 ms
    64 bytes from 10.1.10.1: icmp_req=408 ttl=64 time=8085 ms
    64 bytes from 10.1.10.1: icmp_req=409 ttl=64 time=7086 ms
    64 bytes from 10.1.10.1: icmp_req=410 ttl=64 time=6086 ms
    64 bytes from 10.1.10.1: icmp_req=411 ttl=64 time=5086 ms
    64 bytes from 10.1.10.1: icmp_req=412 ttl=64 time=4086 ms
    64 bytes from 10.1.10.1: icmp_req=413 ttl=64 time=3086 ms
    64 bytes from 10.1.10.1: icmp_req=414 ttl=64 time=2087 ms
    64 bytes from 10.1.10.1: icmp_req=415 ttl=64 time=1087 ms
    64 bytes from 10.1.10.1: icmp_req=416 ttl=64 time=87.2 ms

    Disable Polling

    64 bytes from 10.1.10.1: icmp_req=417 ttl=64 time=0.243 ms
    64 bytes from 10.1.10.1: icmp_req=418 ttl=64 time=0.146 ms
    64 bytes from 10.1.10.1: icmp_req=419 ttl=64 time=0.164 ms
    64 bytes from 10.1.10.1: icmp_req=420 ttl=64 time=0.167 ms

    Re-enable polling

    64 bytes from 10.1.10.1: icmp_req=421 ttl=64 time=9365 ms
    64 bytes from 10.1.10.1: icmp_req=422 ttl=64 time=8365 ms
    64 bytes from 10.1.10.1: icmp_req=423 ttl=64 time=7366 ms
    64 bytes from 10.1.10.1: icmp_req=424 ttl=64 time=6366 ms
    64 bytes from 10.1.10.1: icmp_req=425 ttl=64 time=5366 ms
    64 bytes from 10.1.10.1: icmp_req=426 ttl=64 time=4366 ms
    64 bytes from 10.1.10.1: icmp_req=427 ttl=64 time=3367 ms
    64 bytes from 10.1.10.1: icmp_req=428 ttl=64 time=2367 ms
    64 bytes from 10.1.10.1: icmp_req=429 ttl=64 time=1367 ms
    64 bytes from 10.1.10.1: icmp_req=430 ttl=64 time=368 ms

    Disable Polling.

    64 bytes from 10.1.10.1: icmp_req=431 ttl=64 time=0.133 ms
    64 bytes from 10.1.10.1: icmp_req=432 ttl=64 time=0.240 ms
    64 bytes from 10.1.10.1: icmp_req=433 ttl=64 time=0.154 ms
    64 bytes from 10.1.10.1: icmp_req=434 ttl=64 time=0.239 ms
    64 bytes from 10.1.10.1: icmp_req=435 ttl=64 time=0.145 ms
    64 bytes from 10.1.10.1: icmp_req=436 ttl=64 time=0.305 ms
    64 bytes from 10.1.10.1: icmp_req=437 ttl=64 time=0.111 ms



  • @cncking2000:

    Problem is still occuring.

    What problem? Latency when polling is enabled? High interrupt load?

    You haven't presented any evidence of high interrupt load. Please post output of pfSense shell command```
    vmstat -i



  • All that is seen below is without polling. Enabling polling drops the connections, and will not work.

    [2.1-BETA1][admin@router.*****.com]/root(31): vmstat -i
    interrupt                          total       rate
    irq1: atkbd0                         374          0
    irq15: ata1                           68          0
    irq17: atapci1                    100178         30
    irq18: em0                        774085        237
    irq19: sis0                       439173        134
    cpu0: timer                      6521961       1999
    Total                            7835839       2402

    Problem is incredibly high latency with polling enabled.

    [2.1-BETA1][admin@router.*****.com]/root(32): top -SH

    last pid: 87256;  load averages:  1.43,  0.77,  0.41    up 0+01:01:42  15:43:49
    128 processes: 4 running, 104 sleeping, 20 waiting
    CPU:  0.0% user,  0.0% nice,  100% system,  0.0% interrupt,  0.0% idle
    Mem: 76M Active, 257M Inact, 92M Wired, 12M Cache, 56M Buf, 13M Free
    Swap: 1024M Total, 1024M Free

    PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       0 root     -68    0     0K    64K -        2:03 62.26% kernel{em0 taskq}

    IPerf

    pca@PCA-Linux ~ $ iperf -c 10.1.10.1 -d -i 2 -t 50
    –----------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 85.3 KByte (default)


    Client connecting to 10.1.10.1, TCP port 5001
    TCP window size:  169 KByte (default)

    [  5] local 10.1.10.121 port 35687 connected with 10.1.10.1 port 5001
    [  4] local 10.1.10.121 port 5001 connected with 10.1.10.1 port 50877
    [ ID] Interval       Transfer     Bandwidth
    [  5]  0.0- 2.0 sec  78.9 MBytes   331 Mbits/sec
    [  4]  0.0- 2.0 sec  47.3 MBytes   198 Mbits/sec
    [  5]  2.0- 4.0 sec  87.6 MBytes   368 Mbits/sec
    [  4]  2.0- 4.0 sec  30.4 MBytes   128 Mbits/sec
    [  5]  4.0- 6.0 sec  82.0 MBytes   344 Mbits/sec
    [  4]  4.0- 6.0 sec  39.9 MBytes   168 Mbits/sec
    [  5]  6.0- 8.0 sec  81.5 MBytes   342 Mbits/sec
    [  4]  6.0- 8.0 sec  40.6 MBytes   170 Mbits/sec
    [  5]  8.0-10.0 sec   101 MBytes   423 Mbits/sec
    [  4]  8.0-10.0 sec  23.9 MBytes   100 Mbits/sec
    [  5] 10.0-12.0 sec  79.6 MBytes   334 Mbits/sec
    [  4] 10.0-12.0 sec  45.4 MBytes   190 Mbits/sec
    [  5] 12.0-14.0 sec  90.0 MBytes   377 Mbits/sec
    [  4] 12.0-14.0 sec  35.3 MBytes   148 Mbits/sec
    [  5] 14.0-16.0 sec  68.0 MBytes   285 Mbits/sec
    [  4] 14.0-16.0 sec  56.9 MBytes   239 Mbits/sec
    [  5] 16.0-18.0 sec  77.8 MBytes   326 Mbits/sec
    [  4] 16.0-18.0 sec  46.8 MBytes   196 Mbits/sec
    [  5] 18.0-20.0 sec  90.2 MBytes   379 Mbits/sec
    [  4] 18.0-20.0 sec  35.5 MBytes   149 Mbits/sec
    [  5] 20.0-22.0 sec  91.1 MBytes   382 Mbits/sec
    [  4] 20.0-22.0 sec  35.1 MBytes   147 Mbits/sec
    [  5] 22.0-24.0 sec  92.5 MBytes   388 Mbits/sec
    [  4] 22.0-24.0 sec  33.9 MBytes   142 Mbits/sec
    [  5] 24.0-26.0 sec  89.9 MBytes   377 Mbits/sec
    [  4] 24.0-26.0 sec  36.6 MBytes   154 Mbits/sec
    [  5] 26.0-28.0 sec  91.9 MBytes   385 Mbits/sec
    [  4] 26.0-28.0 sec  34.4 MBytes   144 Mbits/sec
    [  5] 28.0-30.0 sec  92.6 MBytes   388 Mbits/sec
    [  4] 28.0-30.0 sec  33.8 MBytes   142 Mbits/sec
    [  5] 30.0-32.0 sec  89.6 MBytes   376 Mbits/sec
    [  4] 30.0-32.0 sec  36.2 MBytes   152 Mbits/sec
    [  5] 32.0-34.0 sec  87.9 MBytes   369 Mbits/sec
    [  4] 32.0-34.0 sec  38.7 MBytes   162 Mbits/sec
    [  5] 34.0-36.0 sec  82.1 MBytes   344 Mbits/sec
    [  4] 34.0-36.0 sec  44.5 MBytes   187 Mbits/sec
    [  5] 36.0-38.0 sec  88.9 MBytes   373 Mbits/sec
    [  4] 36.0-38.0 sec  37.3 MBytes   156 Mbits/sec
    [  5] 38.0-40.0 sec  83.0 MBytes   348 Mbits/sec
    [  4] 38.0-40.0 sec  42.9 MBytes   180 Mbits/sec
    [  5] 40.0-42.0 sec  85.4 MBytes   358 Mbits/sec
    [  4] 40.0-42.0 sec  40.5 MBytes   170 Mbits/sec
    [  5] 42.0-44.0 sec  83.0 MBytes   348 Mbits/sec
    [  4] 42.0-44.0 sec  41.5 MBytes   174 Mbits/sec
    [  5] 44.0-46.0 sec  89.6 MBytes   376 Mbits/sec
    [  4] 44.0-46.0 sec  32.9 MBytes   138 Mbits/sec
    [  5] 46.0-48.0 sec  86.5 MBytes   363 Mbits/sec
    [  4] 46.0-48.0 sec  39.9 MBytes   167 Mbits/sec
    [  5] 48.0-50.0 sec  78.9 MBytes   331 Mbits/sec
    [  5]  0.0-50.0 sec  2.10 GBytes   361 Mbits/sec
    [  4] 48.0-50.0 sec  47.9 MBytes   201 Mbits/sec
    [  4]  0.0-50.0 sec   978 MBytes   164 Mbits/sec



  • @cncking2000:

    Problem is incredibly high latency with polling enabled.

    Don't use polling then! (I will comment later on polling.)

    @cncking2000:

    [2.1-BETA1][admin@router.*****.com]/root(32): top -SH

    last pid: 87256;  load averages:  1.43,  0.77,  0.41    up 0+01:01:42  15:43:49
    128 processes: 4 running, 104 sleeping, 20 waiting
    CPU:  0.0% user,  0.0% nice,  100% system,  0.0% interrupt,  0.0% idle
    Mem: 76M Active, 257M Inact, 92M Wired, 12M Cache, 56M Buf, 13M Free
    Swap: 1024M Total, 1024M Free

    PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       0 root     -68    0     0K    64K -        2:03 62.26% kernel{em0 taskq}

    What was going on when you took this snapshot? The iperf run?

    Your vmstat output doesn't show excessive interrupt rates, but the rates shown are averaged since startup, not "instantaneous" rates. The top output shows a fully loaded CPU, most of the load apparently in em0 taskq which might be quite reasonable, depending on what the system was supposed to be doing. You have a single CPU, a 3GHz Celeron.

    If you want something to happen on the polling issue you most likely will have to report it to FreeBSD, say on web page: []http://www.freebsd.org/send-pr.html](http://www.freebsd.org/send-pr.html[url) and provide the output of pfSense shell command```
    pciconf -l

    
    I don't know the precise meaning of the iperf numbers reported but my gut feel is that they are possibly low for the type of CPU you have spending 100% in system "mode" (as reported by top). I am guessing you have an iperf server installed on your pfSense box, and that the iperf report segment
    
    > [  5]  8.0-10.0 sec  101 MBytes  423 Mbits/sec
    > [  4]  8.0-10.0 sec  23.9 MBytes  100 Mbits/sec
    
    means that in the interval 8 to 10 seconds from start of test, one stream sent and received at 423Mbps and another stream sent and received at 100Mbps which would make for a total of 2 x (423+100) Mbps or about 1Gbps processed by the box which would pretty much saturate a PCI bus (might be relevant because there are PCI em devices). A bit more context would be helpful in interpreting the numbers you have provided.


  • Yeah, the testing is using IPerf. I can get far better numbers with the same test on a linux machine running the IPerf server, with identical hardware. I have re-ran the tests on both to make sure. Also changed to a single thread.

    IPerf on Server

    pca@PCA-Linux ~ $ iperf -c 10.1.10.71 -i 2 -t 50
    –----------------------------------------------------------
    Client connecting to 10.1.10.71, TCP port 5001
    TCP window size: 22.9 KByte (default)

    [  3] local 10.1.10.121 port 34919 connected with 10.1.10.71 port 5001
    [ ID] Interval      Transfer    Bandwidth
    [  3]  0.0- 2.0 sec  205 MBytes  860 Mbits/sec
    [  3]  2.0- 4.0 sec  217 MBytes  911 Mbits/sec
    [  3]  4.0- 6.0 sec  219 MBytes  919 Mbits/sec
    [  3]  6.0- 8.0 sec  217 MBytes  911 Mbits/sec
    [  3]  8.0-10.0 sec  218 MBytes  915 Mbits/sec
    [  3] 10.0-12.0 sec  218 MBytes  916 Mbits/sec
    [  3] 12.0-14.0 sec  218 MBytes  915 Mbits/sec
    [  3] 14.0-16.0 sec  218 MBytes  914 Mbits/sec
    [  3] 16.0-18.0 sec  218 MBytes  914 Mbits/sec
    [  3] 18.0-20.0 sec  218 MBytes  915 Mbits/sec
    [  3] 20.0-22.0 sec  219 MBytes  918 Mbits/sec
    [  3] 22.0-24.0 sec  219 MBytes  917 Mbits/sec
    [  3] 24.0-26.0 sec  217 MBytes  912 Mbits/sec
    [  3] 26.0-28.0 sec  218 MBytes  914 Mbits/sec
    [  3] 28.0-30.0 sec  218 MBytes  915 Mbits/sec
    [  3] 30.0-32.0 sec  219 MBytes  920 Mbits/sec
    [  3] 32.0-34.0 sec  218 MBytes  913 Mbits/sec
    [  3] 34.0-36.0 sec  218 MBytes  913 Mbits/sec
    [  3] 36.0-38.0 sec  219 MBytes  920 Mbits/sec
    [  3] 38.0-40.0 sec  217 MBytes  910 Mbits/sec
    [  3] 40.0-42.0 sec  218 MBytes  915 Mbits/sec
    [  3] 42.0-44.0 sec  219 MBytes  920 Mbits/sec
    [  3] 44.0-46.0 sec  217 MBytes  910 Mbits/sec
    [  3] 46.0-48.0 sec  218 MBytes  915 Mbits/sec
    [  3] 48.0-50.0 sec  218 MBytes  916 Mbits/sec
    [  3]  0.0-50.0 sec  5.31 GBytes  913 Mbits/sec
    pca@PCA-Linux ~ $

    Top on Linux Box.

    root@cloud:~# top -SH

    top - 16:50:12 up 9 days,  1:03,  2 users,  load average: 0.00, 0.00, 0.00
    Tasks: 130 total,  1 running, 129 sleeping,  0 stopped,  0 zombie
    Cpu(s):  0.7%us, 11.3%sy,  0.0%ni, 86.4%id,  0.0%wa,  0.0%hi,  1.7%si,  0.0%st
    Mem:  1553960k total,  336700k used,  1217260k free,    23408k buffers
    Swap:  3034104k total,    31976k used,  3002128k free,  131940k cached

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
    6774 root      20  0 27848 1064  908 S 13.9  0.1  0:01.86 iperf

    IPerf on Router

    pca@PCA-Linux ~ $ iperf -c 10.1.10.1 -i 2 -t 50
    –----------------------------------------------------------
    Client connecting to 10.1.10.1, TCP port 5001
    TCP window size: 22.9 KByte (default)

    [  3] local 10.1.10.121 port 36129 connected with 10.1.10.1 port 5001
    [ ID] Interval      Transfer    Bandwidth
    [  3]  0.0- 2.0 sec  120 MBytes  502 Mbits/sec
    [  3]  2.0- 4.0 sec  120 MBytes  505 Mbits/sec
    [  3]  4.0- 6.0 sec  116 MBytes  485 Mbits/sec
    [  3]  6.0- 8.0 sec  117 MBytes  490 Mbits/sec
    [  3]  8.0-10.0 sec  117 MBytes  489 Mbits/sec
    [  3] 10.0-12.0 sec  117 MBytes  490 Mbits/sec
    [  3] 12.0-14.0 sec  120 MBytes  503 Mbits/sec
    [  3] 14.0-16.0 sec  120 MBytes  501 Mbits/sec
    [  3] 16.0-18.0 sec  118 MBytes  497 Mbits/sec
    [  3] 18.0-20.0 sec  119 MBytes  498 Mbits/sec
    [  3] 20.0-22.0 sec  120 MBytes  504 Mbits/sec
    [  3] 22.0-24.0 sec  120 MBytes  504 Mbits/sec
    [  3] 24.0-26.0 sec  120 MBytes  502 Mbits/sec
    [  3] 26.0-28.0 sec  121 MBytes  506 Mbits/sec
    [  3] 28.0-30.0 sec  121 MBytes  506 Mbits/sec
    [  3] 30.0-32.0 sec  123 MBytes  516 Mbits/sec
    [  3] 32.0-34.0 sec  122 MBytes  513 Mbits/sec
    [  3] 34.0-36.0 sec  123 MBytes  516 Mbits/sec
    [  3] 36.0-38.0 sec  123 MBytes  515 Mbits/sec
    [  3] 38.0-40.0 sec  124 MBytes  522 Mbits/sec
    [  3] 40.0-42.0 sec  124 MBytes  522 Mbits/sec
    [  3] 42.0-44.0 sec  125 MBytes  524 Mbits/sec
    [  3] 44.0-46.0 sec  125 MBytes  523 Mbits/sec
    [  3] 46.0-48.0 sec  124 MBytes  522 Mbits/sec
    [  3] 48.0-50.0 sec  124 MBytes  522 Mbits/sec
    [  3]  0.0-50.0 sec  2.95 GBytes  507 Mbits/sec
    pca@PCA-Linux ~ $

    Router TOP during test:

    last pid: 44285;  load averages:  0.84,  0.62,  0.30    up 0+01:57:56  16:40:03
    127 processes: 4 running, 103 sleeping, 20 waiting
    CPU:  6.1% user,  0.0% nice, 93.9% system,  0.0% interrupt,  0.0% idle
    Mem: 104M Active, 236M Inact, 81M Wired, 16M Cache, 56M Buf, 14M Free
    Swap: 1024M Total, 1024M Free

    PID USERNAME PRI NICE  SIZE    RES STATE    TIME  WCPU COMMAND
        0 root    -68    0    0K    64K -        4:48 67.29% kernel{em0 taskq}



  • Anything at all? This router is separating servers from the local network, and I cannot continue to use this router with the sub par performance. I would like to be able to, but not being able to utilize a gigabit network is a bit frustrating.



  • @cncking2000:

    Anything at all?

    A few things.

    1. As I pointed out in a previous reply, it is possible your iperf performance on your pfSense box is limited by the hardware. Please post the output of pfSense command```
    pciconf -l -v

    
    2\. Your description doesn't describe clearly the difference in configuration of the tests involving the Linux server and pfSense. Is exactly the same hardware used to run Linux and pfSense? Are the interfaces exactly the same down to the exact same bus type and NIC chipset?
    
    3\. iperf is a useful benchmark but you shouldn't extrapolate iperf performance to packet forwarding performance of a router because iperf moves data between user space and kernel space (an operation well known to be  high overhead operation) while routers forward packets between interfaces WITHOUT moving data between kernel and user space.
    
    @cncking2000:
    
    > This router is separating servers from the local network,
    
    I suggest the first thing would be to verify the hardware you have chosen is actually capable of moving data at gigabit speeds between two Gigabit interfaces. If the interface to the servers and the interface to the "local network" are both on the same "standard" PCI bus then it is a physical impossibility to get sustained gigabit speeds between the interfaces. Your pfSense startup output mentions AGP, SiS and doesn't mention SATA all of which suggests to me that the box is of a generation unlikely to have any more than a single standard PCI bus.


  • You're expecting gigabit wire speed from the sounds of it. Not going to get it with a Celeron. You're wanting Mustang performance out of a Pinto. The diff in cache between a Celeron and an equivalent Pentium 4 makes for a significant performance difference as a firewall. Commercial firewalls that will push a gigabit wire speed are upwards of $10K USD, and you're trying to use a box that probably wouldn't sell on ebay for $50.

    Ditch the sis NIC and Celeron proc and have nothing on a PCI bus, either PCI-X or PCI-e only. Don't use polling.



  • So, What you are suggesting is that I try a HT 3.2Ghz P4 and a 4x PCI-E Dual Intel network card in the 16x PCI-E slot since I have an onboard video in this other box? I can't convince management to spend much on this, but it sure the hell beats using the old Linksys. This facility is too large for a simple router.



  • @cncking2000:

    So, What you are suggesting is that I try a HT 3.2Ghz P4 and a 4x PCI-E Dual Intel network card in the 16x PCI-E slot since I have an onboard video in this other box?

    And run a more appropriate benchmark.


Log in to reply