Traffic between 2 interfaces

stephenw10

Yes that seems like a likely suspect.

On a box I have here:

[2.1-RELEASE][root@pfsense.localdomain]/root(4): cat /var/log/dmesg.boot | grep MSI
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors
em2: Using MSIX interrupts with 3 vectors
em3: Using MSIX interrupts with 3 vectors
em4: Using MSIX interrupts with 3 vectors
em5: Using MSIX interrupts with 3 vectors

Yet at the same time:

[2.1-RELEASE][root@pfsense.localdomain]/root(14): cat /var/log/dmesg.boot | grep irq
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x9c00-0x9c1f mem 0xfe6e0000-0xfe6fffff,0xfe6dc000-0xfe6dffff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xac00-0xac1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 17 at device 0.0 on pci3
em2: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 18 at device 0.0 on pci4
em3: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 19 at device 0.0 on pci5
em4: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci6
em5: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci7</intel(r)></intel(r)></intel(r)></intel(r)></intel(r)></intel(r)>

I expect to see much higher numbered IRQs if it was really using them. More vmstat shows:

[2.1-RELEASE][root@pfsense.localdomain]/root(21): vmstat -i
interrupt                          total       rate
irq4: uart0                          515          0
irq14: ata0                        79091          0
irq20: fxp0                       847274          1
irq23: uhci0 ehci0             199142380        235
cpu0: timer                    335017795        396
irq265: em3:rx 0                  335218          0
irq266: em3:tx 0                  334731          0
irq267: em3:link                       2          0
cpu1: timer                    335017323        396
Total                          870774329       1029

Only em3 is connected on that box.

Steve

maverick_slo

I guess I use MSIX as well:


em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors


[2.1-RELEASE][root@gateway.rasca.local]/root(3): vmstat -i
interrupt                          total       rate
irq19: uhci1+                      20856         13
irq20: em2                        712364        453
cpu0: timer                      3125735       1990
irq256: em0:rx 0                 1179348        751
irq257: em0:tx 0                 1214662        773
irq258: em0:link                    1625          1
irq259: em1:rx 0                  758263        482
irq260: em1:tx 0                  871653        555
irq261: em1:link                    2919          1
cpu1: timer                      3105724       1978
Total                           10993149       7002

So both em0 and em1 are OK.
But IRQs are not OK in my opinion…

I have em2 as WAN, which is on FTTH 20/20 so max troughput is 40MBit/s.
I will move problematic em0 to WAN and use em2 instead of em0 for LAN.

I think this move should solve my problem.

I have to try 40 MBit/s limiter with iperf and if no errors em0 should handle my WAN just fine right?

maverick_slo

I will disable audio/usb/serial and this should give me few IRQs I need :)

stephenw10

Always a good idea.
Was that vmstat snap taken when you were hammering em0? I notice that irq16 doesn't even appear in the list, not interrupting at all.

Moving WAN to em0 seems like a good plan also.

Steve

maverick_slo

WOW :)

atom_net.PNG_thumb

maverick_slo

Even better (just tiiiny bit) when I enabled device polling :)

atom_net.PNG_thumb

stephenw10

Wow better than expected or wow there still a big difference in send vs receive speed? :P
What did you do to achieve that?

General advice is not to enable device polling unless you have a very good reason to do it. It will eat all your spare cpu cycles, sometime it slows the gui to a crawl.
640Mbps from an Atom is good though. :)

Steve

maverick_slo

I`ve swapped em0 and em2.
em0 and em1 are onboard and em2 is on PCI slot.

Now em0 = WAN and em1=LAN and em2=LAN2

So traffic is "spread" across NICs on different BUSes :)

I think that did the trick.

Why in one direction is slower I dont know and I really dont know how to approch it :P

stephenw10

em1 is PCIe and em2 is PCI?

Anyway looks like problem solved. :)

Steve

maverick_slo

em0 and em1 are onboard NICs and on PCIe BUS.
em2 is good old PCI :)

I still have a bunch of errors (missed packets) but speeds are good, somewhere or somewhat is lacking resources, but hey, I have 112 MByte/s transfer speed on same LAN and 50 Mbyte/s between LANs so I think is good :)

maverick_slo

Ta ta da ta taaaaaa :)

Figured it out :)

This mobo has something called: PCIe ASPM (http://en.wikipedia.org/wiki/Active_State_Power_Management)
Disabled it, set PowerD to hiadaptive (not sure if influences on NIC) and results are below.

Now DL/UL are virtually the same as far as Windows file copy is concerned :)

gateway.JPG_thumb

stephenw10

Ah, nice! Good find and good to know for future problems. I bet there's loads of people suffering from that. :)

Steve

maverick_slo

I will change back LAN and LAN2 to em0 and em1 (because of better NIC quality) and leave WAN to em2 again :)
Will see what will happen :)

maverick_slo

Errors are gone now, transfer speeds up to: 70 Mbytes/s.

PCIe ASPM was to blame, hope it helps some one :)

Thanks stephenw10, I really appreciate it!