Traffic between 2 interfaces

stephenw10

Huh, well that's just odd then. Perhaps one of those cards is sharing an IRQ with some thing that's misbehaving slightly. They are MSI/X capable but are they using that?

IP fastforwarding will break IPSec if you are using that. I've had it enabled on my home box for a good while now with no ill effects.

Steve

maverick_slo

MSI/X, how to verify if used?

maverick_slo

Furthermore:

dmesg | grep irq
ioapic0 <version 2.0=""> irqs 0-23 on motherboard
vgapci0: <vga-compatible display=""> port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1
uhci0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a=""> port 0x40a0-0x40bf irq 23 at device 29.0 on pci0
uhci1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b=""> port 0x4080-0x409f irq 19 at device 29.1 on pci0
uhci2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c=""> port 0x4060-0x407f irq 18 at device 29.2 on pci0
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d=""> port 0x4040-0x405f irq 16 at device 29.3 on pci0
ehci0: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller=""> mem 0xd0404400-0xd04047ff irq 23 at device 29.7 on pci0
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4=""> port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3
atapci0: <intel ich7="" sata300="" controller=""> port 0x40c8-0x40cf,0x40dc-0x40df,0x40c0-0x40c7,0x40d8-0x40db,0x4020-0x402f mem 0xd0404000-0xd04043ff irq 19 at device 31.2 on pci0
atrtc0: <at realtime="" clock=""> port 0x70-0x77 irq 8 on acpi0
ppc0: <parallel port=""> port 0x378-0x37f irq 7 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 4 on acpi0
uart2: <16550 or compatible> port 0x2e8-0x2ef irq 3 on acpi0
ata0: <ata channel=""> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1: <ata channel=""> at port 0x170-0x177,0x376 irq 15 on isa0
atkbd0: <at keyboard=""> irq 1 on atkbdc0</at></ata></ata></parallel></at></intel></intel(r)></intel></intel></intel></intel></intel></intel(r)></intel(r)></vga-compatible></version>

and snippet from above for em0 (problematic interface)


dmesg | grep irq\ 16
vgapci0: <vga-compatible display="">port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">port 0x4040-0x405f irq 16 at device 29.3 on pci0</intel></intel(r)></vga-compatible>

and em1:


dmesg | grep irq\ 17
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1</intel(r)>

and em2:


 dmesg | grep irq\ 20
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3</intel(r)>

So em0 is sharing IRQ with VGA and USB controller?
This would explain why em0 is dropping things in my opinion…
em1 and em2 are not sharing IRQ :)

stephenw10

Yes that seems like a likely suspect.

On a box I have here:

[2.1-RELEASE][root@pfsense.localdomain]/root(4): cat /var/log/dmesg.boot | grep MSI
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors
em2: Using MSIX interrupts with 3 vectors
em3: Using MSIX interrupts with 3 vectors
em4: Using MSIX interrupts with 3 vectors
em5: Using MSIX interrupts with 3 vectors

Yet at the same time:

[2.1-RELEASE][root@pfsense.localdomain]/root(14): cat /var/log/dmesg.boot | grep irq
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x9c00-0x9c1f mem 0xfe6e0000-0xfe6fffff,0xfe6dc000-0xfe6dffff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xac00-0xac1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 17 at device 0.0 on pci3
em2: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 18 at device 0.0 on pci4
em3: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 19 at device 0.0 on pci5
em4: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci6
em5: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci7</intel(r)></intel(r)></intel(r)></intel(r)></intel(r)></intel(r)>

I expect to see much higher numbered IRQs if it was really using them. More vmstat shows:

[2.1-RELEASE][root@pfsense.localdomain]/root(21): vmstat -i
interrupt                          total       rate
irq4: uart0                          515          0
irq14: ata0                        79091          0
irq20: fxp0                       847274          1
irq23: uhci0 ehci0             199142380        235
cpu0: timer                    335017795        396
irq265: em3:rx 0                  335218          0
irq266: em3:tx 0                  334731          0
irq267: em3:link                       2          0
cpu1: timer                    335017323        396
Total                          870774329       1029

Only em3 is connected on that box.

Steve

maverick_slo

I guess I use MSIX as well:


em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors


[2.1-RELEASE][root@gateway.rasca.local]/root(3): vmstat -i
interrupt                          total       rate
irq19: uhci1+                      20856         13
irq20: em2                        712364        453
cpu0: timer                      3125735       1990
irq256: em0:rx 0                 1179348        751
irq257: em0:tx 0                 1214662        773
irq258: em0:link                    1625          1
irq259: em1:rx 0                  758263        482
irq260: em1:tx 0                  871653        555
irq261: em1:link                    2919          1
cpu1: timer                      3105724       1978
Total                           10993149       7002

So both em0 and em1 are OK.
But IRQs are not OK in my opinion…

I have em2 as WAN, which is on FTTH 20/20 so max troughput is 40MBit/s.
I will move problematic em0 to WAN and use em2 instead of em0 for LAN.

I think this move should solve my problem.

I have to try 40 MBit/s limiter with iperf and if no errors em0 should handle my WAN just fine right?

maverick_slo

I will disable audio/usb/serial and this should give me few IRQs I need :)

stephenw10

Always a good idea.
Was that vmstat snap taken when you were hammering em0? I notice that irq16 doesn't even appear in the list, not interrupting at all.

Moving WAN to em0 seems like a good plan also.

Steve

maverick_slo

WOW :)

atom_net.PNG_thumb

maverick_slo

Even better (just tiiiny bit) when I enabled device polling :)

atom_net.PNG_thumb

stephenw10

Wow better than expected or wow there still a big difference in send vs receive speed? :P
What did you do to achieve that?

General advice is not to enable device polling unless you have a very good reason to do it. It will eat all your spare cpu cycles, sometime it slows the gui to a crawl.
640Mbps from an Atom is good though. :)

Steve

maverick_slo

I`ve swapped em0 and em2.
em0 and em1 are onboard and em2 is on PCI slot.

Now em0 = WAN and em1=LAN and em2=LAN2

So traffic is "spread" across NICs on different BUSes :)

I think that did the trick.

Why in one direction is slower I dont know and I really dont know how to approch it :P

stephenw10

em1 is PCIe and em2 is PCI?

Anyway looks like problem solved. :)

Steve

maverick_slo

em0 and em1 are onboard NICs and on PCIe BUS.
em2 is good old PCI :)

I still have a bunch of errors (missed packets) but speeds are good, somewhere or somewhat is lacking resources, but hey, I have 112 MByte/s transfer speed on same LAN and 50 Mbyte/s between LANs so I think is good :)

maverick_slo

Ta ta da ta taaaaaa :)

Figured it out :)

This mobo has something called: PCIe ASPM (http://en.wikipedia.org/wiki/Active_State_Power_Management)
Disabled it, set PowerD to hiadaptive (not sure if influences on NIC) and results are below.

Now DL/UL are virtually the same as far as Windows file copy is concerned :)

gateway.JPG_thumb

stephenw10

Ah, nice! Good find and good to know for future problems. I bet there's loads of people suffering from that. :)

Steve

maverick_slo

I will change back LAN and LAN2 to em0 and em1 (because of better NIC quality) and leave WAN to em2 again :)
Will see what will happen :)

maverick_slo

Errors are gone now, transfer speeds up to: 70 Mbytes/s.

PCIe ASPM was to blame, hope it helps some one :)

Thanks stephenw10, I really appreciate it!