Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Traffic between 2 interfaces

    Scheduled Pinned Locked Moved General pfSense Questions
    31 Posts 4 Posters 9.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      maverick_slo
      last edited by

      MSI/X, how to verify if used?

      1 Reply Last reply Reply Quote 0
      • M
        maverick_slo
        last edited by

        Furthermore:

        dmesg | grep irq
ioapic0 <version 2.0=""> irqs 0-23 on motherboard
vgapci0: <vga-compatible display=""> port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1
uhci0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a=""> port 0x40a0-0x40bf irq 23 at device 29.0 on pci0
uhci1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b=""> port 0x4080-0x409f irq 19 at device 29.1 on pci0
uhci2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c=""> port 0x4060-0x407f irq 18 at device 29.2 on pci0
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d=""> port 0x4040-0x405f irq 16 at device 29.3 on pci0
ehci0: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller=""> mem 0xd0404400-0xd04047ff irq 23 at device 29.7 on pci0
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4=""> port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3
atapci0: <intel ich7="" sata300="" controller=""> port 0x40c8-0x40cf,0x40dc-0x40df,0x40c0-0x40c7,0x40d8-0x40db,0x4020-0x402f mem 0xd0404000-0xd04043ff irq 19 at device 31.2 on pci0
atrtc0: <at realtime="" clock=""> port 0x70-0x77 irq 8 on acpi0
ppc0: <parallel port=""> port 0x378-0x37f irq 7 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 4 on acpi0
uart2: <16550 or compatible> port 0x2e8-0x2ef irq 3 on acpi0
ata0: <ata channel=""> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1: <ata channel=""> at port 0x170-0x177,0x376 irq 15 on isa0
atkbd0: <at keyboard=""> irq 1 on atkbdc0</at></ata></ata></parallel></at></intel></intel(r)></intel></intel></intel></intel></intel></intel(r)></intel(r)></vga-compatible></version>

        and snippet from above for em0 (problematic interface)

        
dmesg | grep irq\ 16
vgapci0: <vga-compatible display="">port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">port 0x4040-0x405f irq 16 at device 29.3 on pci0</intel></intel(r)></vga-compatible>

        and em1:

        
dmesg | grep irq\ 17
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1</intel(r)>

        and em2:

        
 dmesg | grep irq\ 20
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3</intel(r)>

        So em0 is sharing IRQ with VGA and USB controller?
        This would explain why em0 is dropping things in my opinion…
        em1 and em2 are not sharing IRQ :)

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Yes that seems like a likely suspect.

          On a box I have here:

          [2.1-RELEASE][root@pfsense.localdomain]/root(4): cat /var/log/dmesg.boot | grep MSI
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors
em2: Using MSIX interrupts with 3 vectors
em3: Using MSIX interrupts with 3 vectors
em4: Using MSIX interrupts with 3 vectors
em5: Using MSIX interrupts with 3 vectors

          Yet at the same time:

          [2.1-RELEASE][root@pfsense.localdomain]/root(14): cat /var/log/dmesg.boot | grep irq
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x9c00-0x9c1f mem 0xfe6e0000-0xfe6fffff,0xfe6dc000-0xfe6dffff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xac00-0xac1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 17 at device 0.0 on pci3
em2: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 18 at device 0.0 on pci4
em3: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 19 at device 0.0 on pci5
em4: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci6
em5: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci7</intel(r)></intel(r)></intel(r)></intel(r)></intel(r)></intel(r)>

          I expect to see much higher numbered IRQs if it was really using them. More vmstat shows:

          [2.1-RELEASE][root@pfsense.localdomain]/root(21): vmstat -i
interrupt                          total       rate
irq4: uart0                          515          0
irq14: ata0                        79091          0
irq20: fxp0                       847274          1
irq23: uhci0 ehci0             199142380        235
cpu0: timer                    335017795        396
irq265: em3:rx 0                  335218          0
irq266: em3:tx 0                  334731          0
irq267: em3:link                       2          0
cpu1: timer                    335017323        396
Total                          870774329       1029

          Only em3 is connected on that box.

          Steve

          1 Reply Last reply Reply Quote 0
          • M
            maverick_slo
            last edited by

            I guess I use MSIX as well:

            
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors


            
[2.1-RELEASE][root@gateway.rasca.local]/root(3): vmstat -i
interrupt                          total       rate
irq19: uhci1+                      20856         13
irq20: em2                        712364        453
cpu0: timer                      3125735       1990
irq256: em0:rx 0                 1179348        751
irq257: em0:tx 0                 1214662        773
irq258: em0:link                    1625          1
irq259: em1:rx 0                  758263        482
irq260: em1:tx 0                  871653        555
irq261: em1:link                    2919          1
cpu1: timer                      3105724       1978
Total                           10993149       7002

            So both em0 and em1 are OK.
            But IRQs are not OK in my opinion…

            I have em2 as WAN, which is on FTTH 20/20 so max troughput is 40MBit/s.
            I will move problematic em0 to WAN and use em2 instead of em0 for LAN.

            I think this move should solve my problem.

            I have to try 40 MBit/s limiter with iperf and if no errors em0 should handle my WAN just fine right?

            1 Reply Last reply Reply Quote 0
            • M
              maverick_slo
              last edited by

              • I will disable audio/usb/serial and this should give me few IRQs I need :)
              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Always a good idea.
                Was that vmstat snap taken when you were hammering em0? I notice that irq16 doesn't even appear in the list, not interrupting at all.

                Moving WAN to em0 seems like a good plan also.

                Steve

                1 Reply Last reply Reply Quote 0
                • M
                  maverick_slo
                  last edited by

                  WOW :)

                  atom_net.PNG
                  atom_net.PNG_thumb

                  1 Reply Last reply Reply Quote 0
                  • M
                    maverick_slo
                    last edited by

                    Even better (just tiiiny bit) when I enabled device polling :)

                    atom_net.PNG
                    atom_net.PNG_thumb

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Wow better than expected or wow there still a big difference in send vs receive speed?  :P
                      What did you do to achieve that?

                      General advice is not to enable device polling unless you have a very good reason to do it. It will eat all your spare cpu cycles, sometime it slows the gui to a crawl.
                      640Mbps from an Atom is good though.  :)

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • M
                        maverick_slo
                        last edited by

                        I`ve swapped em0 and em2.
                        em0 and em1 are onboard and em2 is on PCI slot.

                        Now em0 = WAN and em1=LAN and em2=LAN2

                        So traffic is "spread" across NICs on different BUSes :)

                        I think that did the trick.

                        Why in one direction is slower I dont know and I really dont know how to approch it :P

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          em1 is PCIe and em2 is PCI?

                          Anyway looks like problem solved.  :)

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • M
                            maverick_slo
                            last edited by

                            em0 and em1 are onboard NICs and on PCIe BUS.
                            em2 is good old PCI :)

                            I still have a bunch of errors (missed packets) but speeds are good, somewhere or somewhat is lacking resources, but hey, I have 112 MByte/s transfer speed on same LAN and 50 Mbyte/s between LANs so I think is good :)

                            1 Reply Last reply Reply Quote 0
                            • M
                              maverick_slo
                              last edited by

                              Ta ta da ta taaaaaa :)

                              Figured it out :)

                              This mobo has something called: PCIe ASPM (http://en.wikipedia.org/wiki/Active_State_Power_Management)
                              Disabled it, set PowerD to hiadaptive (not sure if influences on NIC) and results are below.

                              Now DL/UL are virtually the same as far as Windows file copy is concerned :)

                              gateway.JPG
                              gateway.JPG_thumb

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Ah, nice! Good find and good to know for future problems. I bet there's loads of people suffering from that.  :)

                                Steve

                                1 Reply Last reply Reply Quote 0
                                • M
                                  maverick_slo
                                  last edited by

                                  I will change back LAN and LAN2 to em0 and em1 (because of better NIC quality) and leave WAN to em2 again :)
                                  Will see what will happen :)

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    maverick_slo
                                    last edited by

                                    Errors are gone now, transfer speeds up to: 70 Mbytes/s.

                                    PCIe ASPM was to blame, hope it helps some one :)

                                    Thanks stephenw10, I really appreciate it!

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.