Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Traffic between 2 interfaces

    Scheduled Pinned Locked Moved General pfSense Questions
    31 Posts 4 Posters 10.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      maverick_slo
      last edited by

      Hi!

      Cards are both on board Intel, they are the same:

      em0@pci0:2:0:0: class=0x020000 card=0x202c8086 chip=0x10d38086 rev=0x00 hdr=0x00
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled

em1@pci0:1:0:0: class=0x020000 card=0x202c8086 chip=0x10d38086 rev=0x00 hdr=0x00
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled

      I tried different combos of offloading on/off I even enabled device pooling and I also raised send and receive buffers…
      All same results. After fastforwarding = 1, things are working a lot faster...
      And I think that it hasn`t broke anything... I will have to wait and see :)

      I think I will have to live with it, as long packets are missed and not failed and as long there is no actual errors it should be fine :)

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Huh, well that's just odd then. Perhaps one of those cards is sharing an IRQ with some thing that's misbehaving slightly. They are MSI/X capable but are they using that?

        IP fastforwarding will break IPSec if you are using that. I've had it enabled on my home box for a good while now with no ill effects.

        Steve

        1 Reply Last reply Reply Quote 0
        • M
          maverick_slo
          last edited by

          MSI/X, how to verify if used?

          1 Reply Last reply Reply Quote 0
          • M
            maverick_slo
            last edited by

            Furthermore:

            dmesg | grep irq
ioapic0 <version 2.0=""> irqs 0-23 on motherboard
vgapci0: <vga-compatible display=""> port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1
uhci0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a=""> port 0x40a0-0x40bf irq 23 at device 29.0 on pci0
uhci1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b=""> port 0x4080-0x409f irq 19 at device 29.1 on pci0
uhci2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c=""> port 0x4060-0x407f irq 18 at device 29.2 on pci0
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d=""> port 0x4040-0x405f irq 16 at device 29.3 on pci0
ehci0: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller=""> mem 0xd0404400-0xd04047ff irq 23 at device 29.7 on pci0
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4=""> port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3
atapci0: <intel ich7="" sata300="" controller=""> port 0x40c8-0x40cf,0x40dc-0x40df,0x40c0-0x40c7,0x40d8-0x40db,0x4020-0x402f mem 0xd0404000-0xd04043ff irq 19 at device 31.2 on pci0
atrtc0: <at realtime="" clock=""> port 0x70-0x77 irq 8 on acpi0
ppc0: <parallel port=""> port 0x378-0x37f irq 7 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 4 on acpi0
uart2: <16550 or compatible> port 0x2e8-0x2ef irq 3 on acpi0
ata0: <ata channel=""> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1: <ata channel=""> at port 0x170-0x177,0x376 irq 15 on isa0
atkbd0: <at keyboard=""> irq 1 on atkbdc0</at></ata></ata></parallel></at></intel></intel(r)></intel></intel></intel></intel></intel></intel(r)></intel(r)></vga-compatible></version>

            and snippet from above for em0 (problematic interface)

            
dmesg | grep irq\ 16
vgapci0: <vga-compatible display="">port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">port 0x4040-0x405f irq 16 at device 29.3 on pci0</intel></intel(r)></vga-compatible>

            and em1:

            
dmesg | grep irq\ 17
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1</intel(r)>

            and em2:

            
 dmesg | grep irq\ 20
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3</intel(r)>

            So em0 is sharing IRQ with VGA and USB controller?
            This would explain why em0 is dropping things in my opinion…
            em1 and em2 are not sharing IRQ :)

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Yes that seems like a likely suspect.

              On a box I have here:

              [2.1-RELEASE][root@pfsense.localdomain]/root(4): cat /var/log/dmesg.boot | grep MSI
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors
em2: Using MSIX interrupts with 3 vectors
em3: Using MSIX interrupts with 3 vectors
em4: Using MSIX interrupts with 3 vectors
em5: Using MSIX interrupts with 3 vectors

              Yet at the same time:

              [2.1-RELEASE][root@pfsense.localdomain]/root(14): cat /var/log/dmesg.boot | grep irq
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x9c00-0x9c1f mem 0xfe6e0000-0xfe6fffff,0xfe6dc000-0xfe6dffff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xac00-0xac1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 17 at device 0.0 on pci3
em2: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 18 at device 0.0 on pci4
em3: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 19 at device 0.0 on pci5
em4: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci6
em5: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci7</intel(r)></intel(r)></intel(r)></intel(r)></intel(r)></intel(r)>

              I expect to see much higher numbered IRQs if it was really using them. More vmstat shows:

              [2.1-RELEASE][root@pfsense.localdomain]/root(21): vmstat -i
interrupt                          total       rate
irq4: uart0                          515          0
irq14: ata0                        79091          0
irq20: fxp0                       847274          1
irq23: uhci0 ehci0             199142380        235
cpu0: timer                    335017795        396
irq265: em3:rx 0                  335218          0
irq266: em3:tx 0                  334731          0
irq267: em3:link                       2          0
cpu1: timer                    335017323        396
Total                          870774329       1029

              Only em3 is connected on that box.

              Steve

              1 Reply Last reply Reply Quote 0
              • M
                maverick_slo
                last edited by

                I guess I use MSIX as well:

                
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors


                
[2.1-RELEASE][root@gateway.rasca.local]/root(3): vmstat -i
interrupt                          total       rate
irq19: uhci1+                      20856         13
irq20: em2                        712364        453
cpu0: timer                      3125735       1990
irq256: em0:rx 0                 1179348        751
irq257: em0:tx 0                 1214662        773
irq258: em0:link                    1625          1
irq259: em1:rx 0                  758263        482
irq260: em1:tx 0                  871653        555
irq261: em1:link                    2919          1
cpu1: timer                      3105724       1978
Total                           10993149       7002

                So both em0 and em1 are OK.
                But IRQs are not OK in my opinion…

                I have em2 as WAN, which is on FTTH 20/20 so max troughput is 40MBit/s.
                I will move problematic em0 to WAN and use em2 instead of em0 for LAN.

                I think this move should solve my problem.

                I have to try 40 MBit/s limiter with iperf and if no errors em0 should handle my WAN just fine right?

                1 Reply Last reply Reply Quote 0
                • M
                  maverick_slo
                  last edited by

                  • I will disable audio/usb/serial and this should give me few IRQs I need :)
                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Always a good idea.
                    Was that vmstat snap taken when you were hammering em0? I notice that irq16 doesn't even appear in the list, not interrupting at all.

                    Moving WAN to em0 seems like a good plan also.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • M
                      maverick_slo
                      last edited by

                      WOW :)

                      atom_net.PNG
                      atom_net.PNG_thumb

                      1 Reply Last reply Reply Quote 0
                      • M
                        maverick_slo
                        last edited by

                        Even better (just tiiiny bit) when I enabled device polling :)

                        atom_net.PNG
                        atom_net.PNG_thumb

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Wow better than expected or wow there still a big difference in send vs receive speed?  :P
                          What did you do to achieve that?

                          General advice is not to enable device polling unless you have a very good reason to do it. It will eat all your spare cpu cycles, sometime it slows the gui to a crawl.
                          640Mbps from an Atom is good though.  :)

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • M
                            maverick_slo
                            last edited by

                            I`ve swapped em0 and em2.
                            em0 and em1 are onboard and em2 is on PCI slot.

                            Now em0 = WAN and em1=LAN and em2=LAN2

                            So traffic is "spread" across NICs on different BUSes :)

                            I think that did the trick.

                            Why in one direction is slower I dont know and I really dont know how to approch it :P

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              em1 is PCIe and em2 is PCI?

                              Anyway looks like problem solved.  :)

                              Steve

                              1 Reply Last reply Reply Quote 0
                              • M
                                maverick_slo
                                last edited by

                                em0 and em1 are onboard NICs and on PCIe BUS.
                                em2 is good old PCI :)

                                I still have a bunch of errors (missed packets) but speeds are good, somewhere or somewhat is lacking resources, but hey, I have 112 MByte/s transfer speed on same LAN and 50 Mbyte/s between LANs so I think is good :)

                                1 Reply Last reply Reply Quote 0
                                • M
                                  maverick_slo
                                  last edited by

                                  Ta ta da ta taaaaaa :)

                                  Figured it out :)

                                  This mobo has something called: PCIe ASPM (http://en.wikipedia.org/wiki/Active_State_Power_Management)
                                  Disabled it, set PowerD to hiadaptive (not sure if influences on NIC) and results are below.

                                  Now DL/UL are virtually the same as far as Windows file copy is concerned :)

                                  gateway.JPG
                                  gateway.JPG_thumb

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Ah, nice! Good find and good to know for future problems. I bet there's loads of people suffering from that.  :)

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      maverick_slo
                                      last edited by

                                      I will change back LAN and LAN2 to em0 and em1 (because of better NIC quality) and leave WAN to em2 again :)
                                      Will see what will happen :)

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        maverick_slo
                                        last edited by

                                        Errors are gone now, transfer speeds up to: 70 Mbytes/s.

                                        PCIe ASPM was to blame, hope it helps some one :)

                                        Thanks stephenw10, I really appreciate it!

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.