Atom D525 with PCI Intel and PCIe Realtek iperf tests


  • LAYER 8 Netgate

    Another thread prompted me to do some iperf testing this weekend.  What I have is diagrammed in the attachment.

    Everything iperfs at 350-400Mbit/s except when re0 is the receiving interface.  I'm lucky to get 70Mbit/s in that case.  CPU doesn't go crazy or anything.  It just sucks.  Everything is 1000BaseTX and verified to have negotiated at 1000.

    192.168.223.6 (em0_vlan223): iperf –server
    172.29.225.65 (re0_vlan225): iperf -t 30 -c 192.168.223.6

    
    Server listening on TCP port 5001
    TCP window size:  128 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49463
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.9 sec   258 MBytes  69.9 Mbits/sec
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49466
    [  4]  0.0-31.0 sec   256 MBytes  69.4 Mbits/sec
    
    

    172.29.225.65 (re0_vlan225): iperf –server
    192.168.223.6 (em0_vlan223): iperf -t 30 -c 172.29.225.65

    
    [  4] local 192.168.223.6 port 61114 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.35 GBytes   386 Mbits/sec
    [  4] local 192.168.223.6 port 61119 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.33 GBytes   381 Mbits/sec
    
    

    I then moved VLAN 225 to em1 and patched em1 into the same switch port and ran the same tests.

    192.168.223.6 (em0_vlan223): iperf –server
    172.29.225.65 (em1_vlan225): iperf -t 30 -c 192.168.223.6

    
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49467
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.39 GBytes   397 Mbits/sec
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49468
    [  4]  0.0-30.0 sec  1.42 GBytes   407 Mbits/sec
    
    

    And finally:

    172.29.225.65 (em1_vlan225): iperf –server
    192.168.223.6 (em0_vlan223): iperf -t 30 -c 172.29.225.65

    
    [  4] local 192.168.223.6 port 61369 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.33 GBytes   382 Mbits/sec
    [  4] local 192.168.223.6 port 61394 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.27 GBytes   364 M
    
    

    Same switches, same switch ports, same clients and servers.  The only thing that sucks is when the realtek is receiving the traffic.



  • What do the options on each interface look like (ifconfig -m)? Any way you can re-test without VLANs?


  • LAYER 8 Netgate

    Not really without the VLANs. Familial dischord will occur. Resetting states so many times is bad enough.  I did disable HFSC on the realtek with no difference.

    em0_vlan223: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:a4:ec:72
    inet6 fe80::230:18ff:fead:7e4f%em0_vlan223 prefixlen 64 scopeid 0x9
    inet 192.168.223.1 netmask 0xffffff00 broadcast 192.168.223.255
    inet6 2001:656:34f4:223::1 prefixlen 64
    nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    supported media:
    media autoselect
    vlan: 223 vlanpcp: 0 parent interface: em0

    em1_vlan225: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:a4:ec:73
    inet6 fe80::230:18ff:fead:7e4f%em1_vlan225 prefixlen 64 scopeid 0x13
    inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
    inet6 2001:656:34f4:225::1 prefixlen 64
    nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    supported media:
    media autoselect
    vlan: 225 vlanpcp: 0 parent interface: em1

    And, moving my 172.29.225.0/24 network to re0_vlan225 I get this:

    re0_vlan225: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:ad:7e:4f
    inet6 fe80::230:18ff:fead:7e4f%re0_vlan225 prefixlen 64 scopeid 0x13
    inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
    inet6 2001:470:f00e:225::1 prefixlen 64
    nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    supported media:
    media autoselect
    vlan: 225 vlanpcp: 0 parent interface: re0

    While it was configured that way I ran the tests again:

    
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49517
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-35.0 sec   166 MBytes  39.8 Mbits/sec
    
    [  4] local 192.168.223.6 port 62469 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.34 GBytes   382 Mbits/sec
    
    

    Actually, I can remove the VLAN from the realtek interface without much issue.  Here goes:

    re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=39db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic>ether 00:30:18:ad:7e:4f
    inet6 fe80::230:18ff:fead:7e4f%re0 prefixlen 64 scopeid 0x1
    inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
    inet6 2001:470:f00e:225::1 prefixlen 64
    nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    supported media:
    media autoselect mediaopt flowcontrol
    media autoselect
    media 1000baseT mediaopt full-duplex,flowcontrol,master
    media 1000baseT mediaopt full-duplex,flowcontrol
    media 1000baseT mediaopt full-duplex,master
    media 1000baseT mediaopt full-duplex
    media 1000baseT mediaopt master
    media 1000baseT
    media 100baseTX mediaopt full-duplex,flowcontrol
    media 100baseTX mediaopt full-duplex
    media 100baseTX
    media 10baseT/UTP mediaopt full-duplex,flowcontrol
    media 10baseT/UTP mediaopt full-duplex
    media 10baseT/UTP
    media none

    
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49558
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-31.0 sec   266 MBytes  72.0 Mbits/sec
    [  4] local 192.168.223.6 port 62957 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.41 GBytes   403 Mbits/sec
    
    

    So that's both tagged and untagged on re0.</full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast>



  • Sorry, I meant the options on the parent interfaces. But if you're seeing the same issues without VLAN, it's probably not related to that (I figured the Intel interface is more likely to support e.g. VLAN_HWFILTER).

    You mentioned you're running traffic shaping; any chance you could try with that disabled completely? (Note that the actual queuing happens on the output side, so it's at least conceivable that bad settings might cause issues even with shaping disabled on the input side.)



  • Also, what happens when you run iperf locally on your pfSense box and then point the client to 172.29.225.1?


  • LAYER 8 Netgate

    That would involve installing iperf.  I don't install stuff on my pfsenses.

    em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=138db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic,vlan_hwfilter>ether 00:30:18:a4:ec:72
    inet6 fe80::230:18ff:fea4:ec72%em0 prefixlen 64 scopeid 0x2
    nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    supported media:
    media autoselect
    media 1000baseT
    media 1000baseT mediaopt full-duplex
    media 100baseTX mediaopt full-duplex
    media 100baseTX
    media 10baseT/UTP mediaopt full-duplex
    media 10baseT/UTP

    I've seen enough to stay away from realtek.  Anything more will be for those who don't heed the "stay away from realtek" warnings or the devs.  I didn't set out to bash it.  it just happened all by itself.</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic,vlan_hwfilter></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast>


  • LAYER 8 Netgate

    @razzfazz:

    Also, what happens when you run iperf locally on your pfSense box and then point the client to 172.29.225.1?

    Stupid me didn't realize it was an installable package.  I'll do this tonight.  I'll also see if I can get rid of the VLANs for a bit.  My curiosity is piqued.


  • LAYER 8 Netgate

    Okay.  This is with no queues active:

    192.168.223.6 (em0_vlan223): iperf -t 30 -c 192.168.223.6
    172.29.225.65 (re0): iperf –server

    
    [  4] local 192.168.223.6 port 55778 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.62 GBytes   463 Mbits/sec
    [  4] local 192.168.223.6 port 55796 connected with 172.29.225.65 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec  1.56 GBytes   447 Mbits/sec
    
    

    192.168.223.6 (em0_vlan223): iperf –server
    172.29.225.65 (re0): iperf -t 30 -c 192.168.223.6

    
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49647
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.2 sec   186 MBytes  51.6 Mbits/sec
    [  4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49652
    [  4]  0.0-30.0 sec  79.9 MBytes  22.3 Mbits/sec
    
    

    And this is running iperf on pfSense:

    172.29.225.1 (pfSense): iperf –server
    172.29.225.65 (re0): iperf -t 30 -c 172.29.225.1

    
    [  4] local 172.29.225.1 port 5001 connected with 172.29.225.65 port 49654
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-30.0 sec   983 MBytes   275 Mbits/sec
    [  5] local 172.29.225.1 port 5001 connected with 172.29.225.65 port 49655
    [  5]  0.0-30.0 sec   997 MBytes   279 Mbits/sec
    
    

    Better, but still only about half of the em devices routed.

    And one to an interface address on a different pfSense interface:

    172.29.225.1 (pfSense): iperf –server
    172.29.225.65 (re0): iperf -t 30 -c 172.29.225.1

    
    [  4] local 192.168.223.1 port 5001 connected with 172.29.225.65 port 49656
    [  4]  0.0-30.0 sec   993 MBytes   278 Mbits/sec
    [  5] local 192.168.223.1 port 5001 connected with 172.29.225.65 port 49657
    [  5]  0.0-30.0 sec   982 MBytes   274 Mbits/sec
    
    

    Edited to correct that this is all on the re0 untagged interface.



  • OK, the ~280Mbps is roughly in line with what I'm seeing with a similar setup (Atom D510, RTL8111C). The 25-ish Mbps routed through don't make any sense to me, though… Did you clear all states after disabling queuing? If you're really interested in getting to the bottom of this, I'd look at the status->queues page and "top -SHIPz" (both on the pfSense box) during the transfers. If you just want to avoid the issue, assuming you don't have a 1Gbps uplink, maybe just use the re0 interface as your WAN and use the Intel interfaces for your LAN side interfaces?


  • LAYER 8 Netgate

    It doesn't make any sense to me either.

    I did not clear states before testing, but it's my understanding that when the queues are destroyed, there's nothing in the state that would affect it.

    I have relegated re0 to my DSL backup WAN link. :)

    Thanks for your help.  I can try other things but I'm left with the take-home that re in freebsd 8.3 needs some work.


Log in to reply