Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Traffic between 2 interfaces

    Scheduled Pinned Locked Moved General pfSense Questions
    31 Posts 4 Posters 9.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Are they PCI or PCIe?

      Steve

      1 Reply Last reply Reply Quote 0
      • M
        maverick_slo
        last edited by

        This is the MB: http://ark.intel.com/products/56462/Intel-Desktop-Board-D2500CC

        I would say PCI + 1 PCI intel NIC (WAN PPPoE) is installed in 1 free slot…

        1 Reply Last reply Reply Quote 0
        • M
          mikeisfly
          last edited by

          I would check the cable, this would explain the errors in one direction. A poorly terminated cable can get very noise at the ends especially when there is a lot of traffic moving through that cable. Also if both your nics are Gigabit you should be able to get much higher transfer rates depending the read and write targets. If they are SSDs this should be no problem. You state that you are getting 50MB/s is that to say 400Mbps or do you mean 50Mbps? I know that there is some over head for routing but I would think that you could get closer to 1000Mbps. I have noticed from Windows to Windows speeds are much faster than from BSD to Windows or Linux to Windows. I would imagine this would be the same from BSD to BSD and Linux to Linux.
          Worst case senario if your NICs are PCI that means they are sharing bandwidth over the south bridge but the Legacy speed of the that would be no less than 133MB/s or 1064Mbps. They should be the only things using that bus. I would bet money thought that those NICs on your board are PCIe.

          1 Reply Last reply Reply Quote 0
          • M
            maverick_slo
            last edited by

            Yes, I`m getting around 500 Mbit/s which give me around 60 MB/s…
            I think that NICs are PCI and share 133 MB/s so one NIC upload 60 MB/s other download 60 MB/s that equals around 120 MB/s troughput...
            Which would make sense...

            Cable is brand new as well, I just tried 2 different ones.
            In same network from client to server and vice-versa I get 112 MB/s download or upload...
            So cables are fine, switch is fine all is fine :)

            What I will try to do is:

            switch on LAN iface with WAN (WAN is on another PCI card on its own bus) and then try again...

            BDW, you saw link about MB I posted?
            There is no mention of PCIe anywhere...

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Yes if they're both sharing the same bus then you could be hitting a limit there. I never quite got my head around what the maximum transfer speed between two NICs on a pci bus is.  :-
              Even with PCIe NICs you won't see more than 6-700Mbps through an Atom.

              Steve

              1 Reply Last reply Reply Quote 0
              • R
                razzfazz
                last edited by

                The on-board NICs are PCIe, not PCI.

                1 Reply Last reply Reply Quote 0
                • M
                  maverick_slo
                  last edited by

                  Ummm OK…

                  I tried also with iperf.

                  Lets say Computer A is on LAN1 iterface and Computer B on OPT1 interface.

                  Testing iperf - Computer A is server:

                  iperf -c 10.10.0.90 -t 60 -i 10 -f MB/s
                  –----------------------------------------------------------
                  Client connecting to 10.10.0.90, TCP port 5001
                  TCP window size: 0.06 MByte (default)

                  [156] local 172.16.16.230 port 49213 connected with 10.10.0.90 port 5001
                  [ ID] Interval      Transfer    Bandwidth
                  [156]  0.0-10.0 sec  559 MBytes  55.9 MBytes/sec
                  [156] 10.0-20.0 sec  561 MBytes  56.1 MBytes/sec
                  [156] 20.0-30.0 sec  561 MBytes  56.1 MBytes/sec
                  [156] 30.0-40.0 sec  560 MBytes  56.0 MBytes/sec
                  [156] 40.0-50.0 sec  560 MBytes  56.0 MBytes/sec
                  [156] 50.0-60.0 sec  561 MBytes  56.1 MBytes/sec
                  [156]  0.0-60.0 sec  3363 MBytes  56.0 MBytes/sec

                  Now Computer B is server:

                  
iperf -c 172.16.16.230 -t 60 -i 10 -f MB/s
------------------------------------------------------------
Client connecting to 172.16.16.230, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[156] local 10.10.0.90 port 59164 connected with 172.16.16.230 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   497 MBytes  49.7 MBytes/sec
[156] 10.0-20.0 sec   467 MBytes  46.7 MBytes/sec
[156] 20.0-30.0 sec   476 MBytes  47.6 MBytes/sec
[156] 30.0-40.0 sec   512 MBytes  51.2 MBytes/sec
[156] 40.0-50.0 sec   514 MBytes  51.4 MBytes/sec
[156] 50.0-60.0 sec   500 MBytes  50.0 MBytes/sec
[156]  0.0-60.0 sec  2966 MBytes  49.4 MBytes/sec

                  and

                  
iperf -c 172.16.16.230 -t 60 -i 10 -f MB/s
------------------------------------------------------------
Client connecting to 172.16.16.230, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[156] local 10.10.0.90 port 59202 connected with 172.16.16.230 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   536 MBytes  53.6 MBytes/sec
[156] 10.0-20.0 sec   506 MBytes  50.6 MBytes/sec
[156] 20.0-30.0 sec   491 MBytes  49.1 MBytes/sec
[156] 30.0-40.0 sec   455 MBytes  45.5 MBytes/sec
[156] 40.0-50.0 sec   537 MBytes  53.7 MBytes/sec
write failed: Connection reset by peer
read on server close failed: Connection reset by peer
[156]  0.0-77.4 sec  2928 MBytes  37.8 MBytes/sec

                  So if Computer B acts as a server I get different results and bunch of missed packets… If Computer A is server there are absolutely no errors...
                  Well I have to admit, I`m little confused here...

                  2 same NICs on same swtch, same settings for both of them (firewall rules are different) and in one direction OK in other not OK...
                  I just don`t get it :)

                  1 Reply Last reply Reply Quote 0
                  • M
                    maverick_slo
                    last edited by

                    :)
                    Enabled Fastforwarding here are results which are pretty good now :)
                    Close to Atom limit

                    
iperf -c 172.16.16.230 -t 60 -i 5 -f MB/s
------------------------------------------------------------
Client connecting to 172.16.16.230, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[176] local 10.10.0.60 port 58939 connected with 172.16.16.230 port 5001
[ ID] Interval       Transfer     Bandwidth
[176]  0.0- 5.0 sec   288 MBytes  57.6 MBytes/sec
[176]  5.0-10.0 sec   311 MBytes  62.2 MBytes/sec
[176] 10.0-15.0 sec   331 MBytes  66.3 MBytes/sec
[176] 15.0-20.0 sec   326 MBytes  65.2 MBytes/sec
[176] 20.0-25.0 sec   277 MBytes  55.4 MBytes/sec
[176] 25.0-30.0 sec   327 MBytes  65.3 MBytes/sec
[176] 30.0-35.0 sec   312 MBytes  62.4 MBytes/sec
[176] 35.0-40.0 sec   331 MBytes  66.1 MBytes/sec
[176] 40.0-45.0 sec   288 MBytes  57.7 MBytes/sec
[176] 45.0-50.0 sec   320 MBytes  64.0 MBytes/sec
[176] 50.0-55.0 sec   295 MBytes  59.1 MBytes/sec
[176] 55.0-60.0 sec   322 MBytes  64.3 MBytes/sec
[176]  0.0-60.0 sec  3728 MBytes  62.1 MBytes/sec
                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Interesting result in itself.  :)
                      The two cards you are using for LAN and OPT1 are different though yes?
                      They probably have either different or differently supported hardware off loading features. One of those cards is failing to keep up under load. You could try disabling various off-load settings in System: Advanced: Networking: and see what happens. You would expect thoughput to be reduced but if the support is broken as it says on that page then it may actually speed up.
                      Or, since it's doing you no harm, you could just live with it.  ;)

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • M
                        maverick_slo
                        last edited by

                        Hi!

                        Cards are both on board Intel, they are the same:

                        em0@pci0:2:0:0: class=0x020000 card=0x202c8086 chip=0x10d38086 rev=0x00 hdr=0x00
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled

em1@pci0:1:0:0: class=0x020000 card=0x202c8086 chip=0x10d38086 rev=0x00 hdr=0x00
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled

                        I tried different combos of offloading on/off I even enabled device pooling and I also raised send and receive buffers…
                        All same results. After fastforwarding = 1, things are working a lot faster...
                        And I think that it hasn`t broke anything... I will have to wait and see :)

                        I think I will have to live with it, as long packets are missed and not failed and as long there is no actual errors it should be fine :)

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Huh, well that's just odd then. Perhaps one of those cards is sharing an IRQ with some thing that's misbehaving slightly. They are MSI/X capable but are they using that?

                          IP fastforwarding will break IPSec if you are using that. I've had it enabled on my home box for a good while now with no ill effects.

                          Steve

                          1 Reply Last reply Reply Quote 0
                          • M
                            maverick_slo
                            last edited by

                            MSI/X, how to verify if used?

                            1 Reply Last reply Reply Quote 0
                            • M
                              maverick_slo
                              last edited by

                              Furthermore:

                              dmesg | grep irq
ioapic0 <version 2.0=""> irqs 0-23 on motherboard
vgapci0: <vga-compatible display=""> port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2=""> port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1
uhci0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a=""> port 0x40a0-0x40bf irq 23 at device 29.0 on pci0
uhci1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b=""> port 0x4080-0x409f irq 19 at device 29.1 on pci0
uhci2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c=""> port 0x4060-0x407f irq 18 at device 29.2 on pci0
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d=""> port 0x4040-0x405f irq 16 at device 29.3 on pci0
ehci0: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller=""> mem 0xd0404400-0xd04047ff irq 23 at device 29.7 on pci0
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4=""> port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3
atapci0: <intel ich7="" sata300="" controller=""> port 0x40c8-0x40cf,0x40dc-0x40df,0x40c0-0x40c7,0x40d8-0x40db,0x4020-0x402f mem 0xd0404000-0xd04043ff irq 19 at device 31.2 on pci0
atrtc0: <at realtime="" clock=""> port 0x70-0x77 irq 8 on acpi0
ppc0: <parallel port=""> port 0x378-0x37f irq 7 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 3 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 4 on acpi0
uart2: <16550 or compatible> port 0x2e8-0x2ef irq 3 on acpi0
ata0: <ata channel=""> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1: <ata channel=""> at port 0x170-0x177,0x376 irq 15 on isa0
atkbd0: <at keyboard=""> irq 1 on atkbdc0</at></ata></ata></parallel></at></intel></intel(r)></intel></intel></intel></intel></intel></intel(r)></intel(r)></vga-compatible></version>

                              and snippet from above for em0 (problematic interface)

                              
dmesg | grep irq\ 16
vgapci0: <vga-compatible display="">port 0x40d0-0x40d7 mem 0xd0300000-0xd03fffff irq 16 at device 2.0 on pci0
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x3000-0x301f mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff,0xd0240000-0xd0243fff irq 16 at device 0.0 on pci2
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">port 0x4040-0x405f irq 16 at device 29.3 on pci0</intel></intel(r)></vga-compatible>

                              and em1:

                              
dmesg | grep irq\ 17
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x2000-0x201f mem 0xd0120000-0xd013ffff,0xd0100000-0xd011ffff,0xd0140000-0xd0143fff irq 17 at device 0.0 on pci1</intel(r)>

                              and em2:

                              
 dmesg | grep irq\ 20
em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.4="">port 0x1000-0x103f mem 0xd0020000-0xd003ffff,0xd0000000-0xd001ffff irq 20 at device 0.0 on pci3</intel(r)>

                              So em0 is sharing IRQ with VGA and USB controller?
                              This would explain why em0 is dropping things in my opinion…
                              em1 and em2 are not sharing IRQ :)

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Yes that seems like a likely suspect.

                                On a box I have here:

                                [2.1-RELEASE][root@pfsense.localdomain]/root(4): cat /var/log/dmesg.boot | grep MSI
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors
em2: Using MSIX interrupts with 3 vectors
em3: Using MSIX interrupts with 3 vectors
em4: Using MSIX interrupts with 3 vectors
em5: Using MSIX interrupts with 3 vectors

                                Yet at the same time:

                                [2.1-RELEASE][root@pfsense.localdomain]/root(14): cat /var/log/dmesg.boot | grep irq
em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0x9c00-0x9c1f mem 0xfe6e0000-0xfe6fffff,0xfe6dc000-0xfe6dffff irq 16 at device 0.0 on pci2
em1: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xac00-0xac1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 17 at device 0.0 on pci3
em2: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 18 at device 0.0 on pci4
em3: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 19 at device 0.0 on pci5
em4: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci6
em5: <intel(r) 1000="" pro="" network="" connection="" 7.3.2="">port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 17 at device 0.0 on pci7</intel(r)></intel(r)></intel(r)></intel(r)></intel(r)></intel(r)>

                                I expect to see much higher numbered IRQs if it was really using them. More vmstat shows:

                                [2.1-RELEASE][root@pfsense.localdomain]/root(21): vmstat -i
interrupt                          total       rate
irq4: uart0                          515          0
irq14: ata0                        79091          0
irq20: fxp0                       847274          1
irq23: uhci0 ehci0             199142380        235
cpu0: timer                    335017795        396
irq265: em3:rx 0                  335218          0
irq266: em3:tx 0                  334731          0
irq267: em3:link                       2          0
cpu1: timer                    335017323        396
Total                          870774329       1029

                                Only em3 is connected on that box.

                                Steve

                                1 Reply Last reply Reply Quote 0
                                • M
                                  maverick_slo
                                  last edited by

                                  I guess I use MSIX as well:

                                  
em0: Using MSIX interrupts with 3 vectors
em1: Using MSIX interrupts with 3 vectors


                                  
[2.1-RELEASE][root@gateway.rasca.local]/root(3): vmstat -i
interrupt                          total       rate
irq19: uhci1+                      20856         13
irq20: em2                        712364        453
cpu0: timer                      3125735       1990
irq256: em0:rx 0                 1179348        751
irq257: em0:tx 0                 1214662        773
irq258: em0:link                    1625          1
irq259: em1:rx 0                  758263        482
irq260: em1:tx 0                  871653        555
irq261: em1:link                    2919          1
cpu1: timer                      3105724       1978
Total                           10993149       7002

                                  So both em0 and em1 are OK.
                                  But IRQs are not OK in my opinion…

                                  I have em2 as WAN, which is on FTTH 20/20 so max troughput is 40MBit/s.
                                  I will move problematic em0 to WAN and use em2 instead of em0 for LAN.

                                  I think this move should solve my problem.

                                  I have to try 40 MBit/s limiter with iperf and if no errors em0 should handle my WAN just fine right?

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    maverick_slo
                                    last edited by

                                    • I will disable audio/usb/serial and this should give me few IRQs I need :)
                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Always a good idea.
                                      Was that vmstat snap taken when you were hammering em0? I notice that irq16 doesn't even appear in the list, not interrupting at all.

                                      Moving WAN to em0 seems like a good plan also.

                                      Steve

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        maverick_slo
                                        last edited by

                                        WOW :)

                                        atom_net.PNG
                                        atom_net.PNG_thumb

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          maverick_slo
                                          last edited by

                                          Even better (just tiiiny bit) when I enabled device polling :)

                                          atom_net.PNG
                                          atom_net.PNG_thumb

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            Wow better than expected or wow there still a big difference in send vs receive speed?  :P
                                            What did you do to achieve that?

                                            General advice is not to enable device polling unless you have a very good reason to do it. It will eat all your spare cpu cycles, sometime it slows the gui to a crawl.
                                            640Mbps from an Atom is good though.  :)

                                            Steve

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.