Atom D525 with PCI Intel and PCIe Realtek iperf tests
-
Another thread prompted me to do some iperf testing this weekend. What I have is diagrammed in the attachment.
Everything iperfs at 350-400Mbit/s except when re0 is the receiving interface. I'm lucky to get 70Mbit/s in that case. CPU doesn't go crazy or anything. It just sucks. Everything is 1000BaseTX and verified to have negotiated at 1000.
192.168.223.6 (em0_vlan223): iperf –server
172.29.225.65 (re0_vlan225): iperf -t 30 -c 192.168.223.6Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49463 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.9 sec 258 MBytes 69.9 Mbits/sec [ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49466 [ 4] 0.0-31.0 sec 256 MBytes 69.4 Mbits/sec
172.29.225.65 (re0_vlan225): iperf –server
192.168.223.6 (em0_vlan223): iperf -t 30 -c 172.29.225.65[ 4] local 192.168.223.6 port 61114 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.35 GBytes 386 Mbits/sec [ 4] local 192.168.223.6 port 61119 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.33 GBytes 381 Mbits/sec
I then moved VLAN 225 to em1 and patched em1 into the same switch port and ran the same tests.
192.168.223.6 (em0_vlan223): iperf –server
172.29.225.65 (em1_vlan225): iperf -t 30 -c 192.168.223.6[ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49467 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.39 GBytes 397 Mbits/sec [ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49468 [ 4] 0.0-30.0 sec 1.42 GBytes 407 Mbits/sec
And finally:
172.29.225.65 (em1_vlan225): iperf –server
192.168.223.6 (em0_vlan223): iperf -t 30 -c 172.29.225.65[ 4] local 192.168.223.6 port 61369 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.33 GBytes 382 Mbits/sec [ 4] local 192.168.223.6 port 61394 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.27 GBytes 364 M
Same switches, same switch ports, same clients and servers. The only thing that sucks is when the realtek is receiving the traffic.
-
What do the options on each interface look like (ifconfig -m)? Any way you can re-test without VLANs?
-
Not really without the VLANs. Familial dischord will occur. Resetting states so many times is bad enough. I did disable HFSC on the realtek with no difference.
em0_vlan223: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:a4:ec:72
inet6 fe80::230:18ff:fead:7e4f%em0_vlan223 prefixlen 64 scopeid 0x9
inet 192.168.223.1 netmask 0xffffff00 broadcast 192.168.223.255
inet6 2001:656:34f4:223::1 prefixlen 64
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
vlan: 223 vlanpcp: 0 parent interface: em0em1_vlan225: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:a4:ec:73
inet6 fe80::230:18ff:fead:7e4f%em1_vlan225 prefixlen 64 scopeid 0x13
inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
inet6 2001:656:34f4:225::1 prefixlen 64
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
vlan: 225 vlanpcp: 0 parent interface: em1And, moving my 172.29.225.0/24 network to re0_vlan225 I get this:
re0_vlan225: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>capabilities=3 <rxcsum,txcsum>ether 00:30:18:ad:7e:4f
inet6 fe80::230:18ff:fead:7e4f%re0_vlan225 prefixlen 64 scopeid 0x13
inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
inet6 2001:470:f00e:225::1 prefixlen 64
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
vlan: 225 vlanpcp: 0 parent interface: re0While it was configured that way I ran the tests again:
[ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49517 [ ID] Interval Transfer Bandwidth [ 4] 0.0-35.0 sec 166 MBytes 39.8 Mbits/sec [ 4] local 192.168.223.6 port 62469 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.34 GBytes 382 Mbits/sec
Actually, I can remove the VLAN from the realtek interface without much issue. Here goes:
re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=39db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic>ether 00:30:18:ad:7e:4f
inet6 fe80::230:18ff:fead:7e4f%re0 prefixlen 64 scopeid 0x1
inet 172.29.225.1 netmask 0xffffff00 broadcast 172.29.225.255
inet6 2001:470:f00e:225::1 prefixlen 64
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect mediaopt flowcontrol
media autoselect
media 1000baseT mediaopt full-duplex,flowcontrol,master
media 1000baseT mediaopt full-duplex,flowcontrol
media 1000baseT mediaopt full-duplex,master
media 1000baseT mediaopt full-duplex
media 1000baseT mediaopt master
media 1000baseT
media 100baseTX mediaopt full-duplex,flowcontrol
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex,flowcontrol
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
media none[ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49558 [ ID] Interval Transfer Bandwidth [ 4] 0.0-31.0 sec 266 MBytes 72.0 Mbits/sec [ 4] local 192.168.223.6 port 62957 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.41 GBytes 403 Mbits/sec
So that's both tagged and untagged on re0.</full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum></rxcsum,txcsum></up,broadcast,running,simplex,multicast>
-
Sorry, I meant the options on the parent interfaces. But if you're seeing the same issues without VLAN, it's probably not related to that (I figured the Intel interface is more likely to support e.g. VLAN_HWFILTER).
You mentioned you're running traffic shaping; any chance you could try with that disabled completely? (Note that the actual queuing happens on the output side, so it's at least conceivable that bad settings might cause issues even with shaping disabled on the input side.)
-
Also, what happens when you run iperf locally on your pfSense box and then point the client to 172.29.225.1?
-
That would involve installing iperf. I don't install stuff on my pfsenses.
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=138db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic,vlan_hwfilter>ether 00:30:18:a4:ec:72
inet6 fe80::230:18ff:fea4:ec72%em0 prefixlen 64 scopeid 0x2
nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
media 1000baseT
media 1000baseT mediaopt full-duplex
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTPI've seen enough to stay away from realtek. Anything more will be for those who don't heed the "stay away from realtek" warnings or the devs. I didn't set out to bash it. it just happened all by itself.</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic,vlan_hwfilter></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast>
-
Also, what happens when you run iperf locally on your pfSense box and then point the client to 172.29.225.1?
Stupid me didn't realize it was an installable package. I'll do this tonight. I'll also see if I can get rid of the VLANs for a bit. My curiosity is piqued.
-
Okay. This is with no queues active:
192.168.223.6 (em0_vlan223): iperf -t 30 -c 192.168.223.6
172.29.225.65 (re0): iperf –server[ 4] local 192.168.223.6 port 55778 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.62 GBytes 463 Mbits/sec [ 4] local 192.168.223.6 port 55796 connected with 172.29.225.65 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 1.56 GBytes 447 Mbits/sec
192.168.223.6 (em0_vlan223): iperf –server
172.29.225.65 (re0): iperf -t 30 -c 192.168.223.6[ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49647 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.2 sec 186 MBytes 51.6 Mbits/sec [ 4] local 192.168.223.6 port 5001 connected with 172.29.225.65 port 49652 [ 4] 0.0-30.0 sec 79.9 MBytes 22.3 Mbits/sec
And this is running iperf on pfSense:
172.29.225.1 (pfSense): iperf –server
172.29.225.65 (re0): iperf -t 30 -c 172.29.225.1[ 4] local 172.29.225.1 port 5001 connected with 172.29.225.65 port 49654 [ ID] Interval Transfer Bandwidth [ 4] 0.0-30.0 sec 983 MBytes 275 Mbits/sec [ 5] local 172.29.225.1 port 5001 connected with 172.29.225.65 port 49655 [ 5] 0.0-30.0 sec 997 MBytes 279 Mbits/sec
Better, but still only about half of the em devices routed.
And one to an interface address on a different pfSense interface:
172.29.225.1 (pfSense): iperf –server
172.29.225.65 (re0): iperf -t 30 -c 172.29.225.1[ 4] local 192.168.223.1 port 5001 connected with 172.29.225.65 port 49656 [ 4] 0.0-30.0 sec 993 MBytes 278 Mbits/sec [ 5] local 192.168.223.1 port 5001 connected with 172.29.225.65 port 49657 [ 5] 0.0-30.0 sec 982 MBytes 274 Mbits/sec
Edited to correct that this is all on the re0 untagged interface.
-
OK, the ~280Mbps is roughly in line with what I'm seeing with a similar setup (Atom D510, RTL8111C). The 25-ish Mbps routed through don't make any sense to me, though… Did you clear all states after disabling queuing? If you're really interested in getting to the bottom of this, I'd look at the status->queues page and "top -SHIPz" (both on the pfSense box) during the transfers. If you just want to avoid the issue, assuming you don't have a 1Gbps uplink, maybe just use the re0 interface as your WAN and use the Intel interfaces for your LAN side interfaces?
-
It doesn't make any sense to me either.
I did not clear states before testing, but it's my understanding that when the queues are destroyed, there's nothing in the state that would affect it.
I have relegated re0 to my DSL backup WAN link. :)
Thanks for your help. I can try other things but I'm left with the take-home that re in freebsd 8.3 needs some work.