Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Atom D525 with PCI Intel and PCIe Realtek iperf tests

    Scheduled Pinned Locked Moved Hardware
    10 Posts 2 Posters 1.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD
      Derelict LAYER 8 Netgate
      last edited by

      Another thread prompted me to do some iperf testing this weekend.  What I have is diagrammed in the attachment.

      Everything iperfs at 350-400Mbit/s except when re0 is the receiving interface.  I'm lucky to get 70Mbit/s in that case.  CPU doesn't go crazy or anything.  It just sucks.  Everything is 1000BaseTX and verified to have negotiated at 1000.

      192.168.223.6 (em0_vlan223): iperf –server
      172.29.225.65 (re0_vlan225): iperf -t 30 -c 192.168.223.6

      
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49463
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.9 sec   258 MBytes  69.9 Mbits/sec
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49466
[  4]  0.0-31.0 sec   256 MBytes  69.4 Mbits/sec

      172.29.225.65 (re0_vlan225): iperf –server
      192.168.223.6 (em0_vlan223): iperf -t 30 -c 172.29.225.65

      
[  4] local 192.168.223.6 port 61114 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.35 GBytes   386 Mbits/sec
[  4] local 192.168.223.6 port 61119 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.33 GBytes   381 Mbits/sec

      I then moved VLAN 225 to em1 and patched em1 into the same switch port and ran the same tests.

      192.168.223.6 (em0_vlan223): iperf –server
      172.29.225.65 (em1_vlan225): iperf -t 30 -c 192.168.223.6

      
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49467
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.39 GBytes   397 Mbits/sec
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49468
[  4]  0.0-30.0 sec  1.42 GBytes   407 Mbits/sec

      And finally:

      172.29.225.65 (em1_vlan225): iperf –server
      192.168.223.6 (em0_vlan223): iperf -t 30 -c 172.29.225.65

      
[  4] local 192.168.223.6 port 61369 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.33 GBytes   382 Mbits/sec
[  4] local 192.168.223.6 port 61394 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.27 GBytes   364 M

      Same switches, same switch ports, same clients and servers.  The only thing that sucks is when the realtek is receiving the traffic.
      iperf_re_em.png_thumb
      iperf_re_em.png

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • R
        razzfazz
        last edited by

        What do the options on each interface look like (ifconfig -m)? Any way you can re-test without VLANs?

        1 Reply Last reply Reply Quote 0
        • DerelictD
          Derelict LAYER 8 Netgate
          last edited by

          Not really without the VLANs. Familial dischord will occur. Resetting states so many times is bad enough.  I did disable HFSC on the realtek with no difference.

          em0_vlan223: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
          options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:a4:ec:72
          inet6 fe80::230:18ff:fead:7e4f%em0_vlan223 prefixlen 64 scopeid 0x9
          inet 192.168.223.1 netmask 0xffffff00 broadcast 192.168.223.255
          inet6 2001:656:34f4:223::1 prefixlen 64
          nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          supported media:
          media autoselect
          vlan: 223 vlanpcp: 0 parent interface: em0

          em1_vlan225: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
          options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:a4:ec:73
          inet6 fe80::230:18ff:fead:7e4f%em1_vlan225 prefixlen 64 scopeid 0x13
          inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
          inet6 2001:656:34f4:225::1 prefixlen 64
          nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          supported media:
          media autoselect
          vlan: 225 vlanpcp: 0 parent interface: em1

          And, moving my 172.29.225.0/24 network to re0_vlan225 I get this:

          re0_vlan225: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
          options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:ad:7e:4f
          inet6 fe80::230:18ff:fead:7e4f%re0_vlan225 prefixlen 64 scopeid 0x13
          inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
          inet6 2001:470:f00e:225::1 prefixlen 64
          nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          supported media:
          media autoselect
          vlan: 225 vlanpcp: 0 parent interface: re0

          While it was configured that way I ran the tests again:

          
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49517
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-35.0 sec   166 MBytes  39.8 Mbits/sec

[  4] local 192.168.223.6 port 62469 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.34 GBytes   382 Mbits/sec

          Actually, I can remove the VLAN from the realtek interface without much issue.  Here goes:

          re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
          options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=39db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic>ether 00:30:18:ad:7e:4f
          inet6 fe80::230:18ff:fead:7e4f%re0 prefixlen 64 scopeid 0x1
          inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
          inet6 2001:470:f00e:225::1 prefixlen 64
          nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
          status: active
          supported media:
          media autoselect mediaopt flowcontrol
          media autoselect
          media 1000baseT mediaopt full-duplex,flowcontrol,master
          media 1000baseT mediaopt full-duplex,flowcontrol
          media 1000baseT mediaopt full-duplex,master
          media 1000baseT mediaopt full-duplex
          media 1000baseT mediaopt master
          media 1000baseT
          media 100baseTX mediaopt full-duplex,flowcontrol
          media 100baseTX mediaopt full-duplex
          media 100baseTX
          media 10baseT/UTP mediaopt full-duplex,flowcontrol
          media 10baseT/UTP mediaopt full-duplex
          media 10baseT/UTP
          media none

          
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49558
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-31.0 sec   266 MBytes  72.0 Mbits/sec
[  4] local 192.168.223.6 port 62957 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.41 GBytes   403 Mbits/sec

          So that's both tagged and untagged on re0.</full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast>

          Chattanooga, Tennessee, USA
          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
          Do Not Chat For Help! NO_WAN_EGRESS(TM)

          1 Reply Last reply Reply Quote 0
          • R
            razzfazz
            last edited by

            Sorry, I meant the options on the parent interfaces. But if you're seeing the same issues without VLAN, it's probably not related to that (I figured the Intel interface is more likely to support e.g. VLAN_HWFILTER).

            You mentioned you're running traffic shaping; any chance you could try with that disabled completely? (Note that the actual queuing happens on the output side, so it's at least conceivable that bad settings might cause issues even with shaping disabled on the input side.)

            1 Reply Last reply Reply Quote 0
            • R
              razzfazz
              last edited by

              Also, what happens when you run iperf locally on your pfSense box and then point the client to 172.29.225.1?

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                That would involve installing iperf.  I don't install stuff on my pfsenses.

                em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
                options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=138db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic,vlan_hwfilter>ether 00:30:18:a4:ec:72
                inet6 fe80::230:18ff:fea4:ec72%em0 prefixlen 64 scopeid 0x2
                nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
                status: active
                supported media:
                media autoselect
                media 1000baseT
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP

                I've seen enough to stay away from realtek.  Anything more will be for those who don't heed the "stay away from realtek" warnings or the devs.  I didn't set out to bash it.  it just happened all by itself.</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic,vlan_hwfilter></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast>

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  @razzfazz:

                  Also, what happens when you run iperf locally on your pfSense box and then point the client to 172.29.225.1?

                  Stupid me didn't realize it was an installable package.  I'll do this tonight.  I'll also see if I can get rid of the VLANs for a bit.  My curiosity is piqued.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    Okay.  This is with no queues active:

                    192.168.223.6 (em0_vlan223): iperf -t 30 -c 192.168.223.6
                    172.29.225.65 (re0): iperf –server

                    
[  4] local 192.168.223.6 port 55778 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.62 GBytes   463 Mbits/sec
[  4] local 192.168.223.6 port 55796 connected with 172.29.225.65 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.56 GBytes   447 Mbits/sec

                    192.168.223.6 (em0_vlan223): iperf –server
                    172.29.225.65 (re0): iperf -t 30 -c 192.168.223.6

                    
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49647
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.2 sec   186 MBytes  51.6 Mbits/sec
[  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49652
[  4]  0.0-30.0 sec  79.9 MBytes  22.3 Mbits/sec

                    And this is running iperf on pfSense:

                    172.29.225.1 (pfSense): iperf –server
                    172.29.225.65 (re0): iperf -t 30 -c 172.29.225.1

                    
[  4] local 172.29.225.1 port 5001 connected with 172.29.225.65 port 49654
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec   983 MBytes   275 Mbits/sec
[  5] local 172.29.225.1 port 5001 connected with 172.29.225.65 port 49655
[  5]  0.0-30.0 sec   997 MBytes   279 Mbits/sec

                    Better, but still only about half of the em devices routed.

                    And one to an interface address on a different pfSense interface:

                    172.29.225.1 (pfSense): iperf –server
                    172.29.225.65 (re0): iperf -t 30 -c 172.29.225.1

                    
[  4] local 192.168.223.1 port 5001 connected with 172.29.225.65 port 49656
[  4]  0.0-30.0 sec   993 MBytes   278 Mbits/sec
[  5] local 192.168.223.1 port 5001 connected with 172.29.225.65 port 49657
[  5]  0.0-30.0 sec   982 MBytes   274 Mbits/sec

                    Edited to correct that this is all on the re0 untagged interface.

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • R
                      razzfazz
                      last edited by

                      OK, the ~280Mbps is roughly in line with what I'm seeing with a similar setup (Atom D510, RTL8111C). The 25-ish Mbps routed through don't make any sense to me, though… Did you clear all states after disabling queuing? If you're really interested in getting to the bottom of this, I'd look at the status->queues page and "top -SHIPz" (both on the pfSense box) during the transfers. If you just want to avoid the issue, assuming you don't have a 1Gbps uplink, maybe just use the re0 interface as your WAN and use the Intel interfaces for your LAN side interfaces?

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        It doesn't make any sense to me either.

                        I did not clear states before testing, but it's my understanding that when the queues are destroyed, there's nothing in the state that would affect it.

                        I have relegated re0 to my DSL backup WAN link. :)

                        Thanks for your help.  I can try other things but I'm left with the take-home that re in freebsd 8.3 needs some work.

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.