Traffic between 2 interfaces

maverick_slo

I guess I use MSIX as well:


em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors


[2.1-RELEASE][root@gateway.rasca.local]/root(3): vmstat -i
interrupt                          total       rate
irq19: uhci1+                      20856         13
irq20: em2                        712364        453
cpu0: timer                      3125735       1990
irq256: em0:rx 0                 1179348        751
irq257: em0:tx 0                 1214662        773
irq258: em0:link                    1625          1
irq259: em1:rx 0                  758263        482
irq260: em1:tx 0                  871653        555
irq261: em1:link                    2919          1
cpu1: timer                      3105724       1978
Total                           10993149       7002

So both em0 and em1 are OK.
But IRQs are not OK in my opinion…

I have em2 as WAN, which is on FTTH 20/20 so max troughput is 40MBit/s.
I will move problematic em0 to WAN and use em2 instead of em0 for LAN.

I think this move should solve my problem.

I have to try 40 MBit/s limiter with iperf and if no errors em0 should handle my WAN just fine right?

maverick_slo

I will disable audio/usb/serial and this should give me few IRQs I need :)

stephenw10

Always a good idea.
Was that vmstat snap taken when you were hammering em0? I notice that irq16 doesn't even appear in the list, not interrupting at all.

Moving WAN to em0 seems like a good plan also.

Steve

maverick_slo

WOW :)

atom_net.PNG_thumb

maverick_slo

Even better (just tiiiny bit) when I enabled device polling :)

atom_net.PNG_thumb

stephenw10

Wow better than expected or wow there still a big difference in send vs receive speed? :P
What did you do to achieve that?

General advice is not to enable device polling unless you have a very good reason to do it. It will eat all your spare cpu cycles, sometime it slows the gui to a crawl.
640Mbps from an Atom is good though. :)

Steve

maverick_slo

I`ve swapped em0 and em2.
em0 and em1 are onboard and em2 is on PCI slot.

Now em0 = WAN and em1=LAN and em2=LAN2

So traffic is "spread" across NICs on different BUSes :)

I think that did the trick.

Why in one direction is slower I dont know and I really dont know how to approch it :P

stephenw10

em1 is PCIe and em2 is PCI?

Anyway looks like problem solved. :)

Steve

maverick_slo

em0 and em1 are onboard NICs and on PCIe BUS.
em2 is good old PCI :)

I still have a bunch of errors (missed packets) but speeds are good, somewhere or somewhat is lacking resources, but hey, I have 112 MByte/s transfer speed on same LAN and 50 Mbyte/s between LANs so I think is good :)

maverick_slo

Ta ta da ta taaaaaa :)

Figured it out :)

This mobo has something called: PCIe ASPM (http://en.wikipedia.org/wiki/Active_State_Power_Management)
Disabled it, set PowerD to hiadaptive (not sure if influences on NIC) and results are below.

Now DL/UL are virtually the same as far as Windows file copy is concerned :)

gateway.JPG_thumb

stephenw10

Ah, nice! Good find and good to know for future problems. I bet there's loads of people suffering from that. :)

Steve

maverick_slo

I will change back LAN and LAN2 to em0 and em1 (because of better NIC quality) and leave WAN to em2 again :)
Will see what will happen :)

maverick_slo

Errors are gone now, transfer speeds up to: 70 Mbytes/s.

PCIe ASPM was to blame, hope it helps some one :)

Thanks stephenw10, I really appreciate it!