[updated with new ifconfs/iperf] config. causing 300Mbps drop in lan performance

cmcdonald · Jun 4, 2013, 3:15 AM

Link to Dmesg: http://pastie.org/7995476 Both systems have identical hardware.

Update: I have posted my most recent ifconfigs and iperf tests. This time using a multi-threaded test instead of single threaded (as per a suggestion offered on another forum).

Left Box

$ ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether 00:15:17:20:d5:00
	inet xxx.xxx.xxx.226 netmask 0xfffffff8 broadcast xxx.xxx.xxx.231
	inet6 fe80::215:17ff:fe20:d500%em0 prefixlen 64 scopeid 0x1 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=4019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso>ether 00:15:17:20:d5:01
	media: Ethernet autoselect
	status: no carrier
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=5209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:24:b9
	inet6 fe80::7254:d2ff:feab:24b9%em2 prefixlen 64 scopeid 0x3 
	inet 192.168.1.1 netmask 0xfffffffc broadcast 192.168.1.3
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=5219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:24:ba
	inet6 fe80::7254:d2ff:feab:24ba%em3 prefixlen 64 scopeid 0x4 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	nd6 options=43 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33664
em3_vlan10: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan10 prefixlen 64 scopeid 0x9 
	inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 10 parent interface: em3
em3_vlan20: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan20 prefixlen 64 scopeid 0xa 
	inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 20 parent interface: em3
em3_vlan30: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan30 prefixlen 64 scopeid 0xb 
	inet 172.16.0.1 netmask 0xfffffc00 broadcast 172.16.3.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 30 parent interface: em3
pptpd0: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
em2_vlan100: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 70:54:d2:ab:24:b9
	inet6 fe80::215:17ff:fe20:d500%em2_vlan100 prefixlen 64 scopeid 0xd 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 100 parent interface: em2
em2_vlan200: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 70:54:d2:ab:24:b9
	inet6 fe80::215:17ff:fe20:d500%em2_vlan200 prefixlen 64 scopeid 0xe 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 200 parent interface: em2
em2_vlan300: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 70:54:d2:ab:24:b9
	inet6 fe80::215:17ff:fe20:d500%em2_vlan300 prefixlen 64 scopeid 0xf 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 300 parent interface: em2</full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></pointopoint,noarp,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso></broadcast,oactive,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast>

Right Box:

$ ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether 00:15:17:6a:df:b8
	inet 162.17.46.227 netmask 0xfffffff8 broadcast 162.17.46.231
	inet6 fe80::215:17ff:fe6a:dfb8%em0 prefixlen 64 scopeid 0x1 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=4019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso>ether 00:15:17:6a:df:b9
	media: Ethernet autoselect
	status: no carrier
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=5209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:1e:32
	inet6 fe80::7254:d2ff:feab:1e32%em2 prefixlen 64 scopeid 0x3 
	inet 192.168.1.2 netmask 0xfffffffc broadcast 192.168.1.3
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=5219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:1e:33
	inet6 fe80::7254:d2ff:feab:1e33%em3 prefixlen 64 scopeid 0x4 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
enc0: flags=0<> metric 0 mtu 1536
pflog0: flags=100 <promisc>metric 0 mtu 33144
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
em3_vlan10: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan10 prefixlen 64 scopeid 0x9 
	inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 10 vlanpcp: 0 parent interface: em3
em3_vlan20: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan20 prefixlen 64 scopeid 0xa 
	inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 20 vlanpcp: 0 parent interface: em3
em3_vlan30: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan30 prefixlen 64 scopeid 0xb 
	inet 172.16.0.2 netmask 0xfffffc00 broadcast 172.16.3.255
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 30 vlanpcp: 0 parent interface: em3</full-duplex></performnud></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso></broadcast,oactive,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast>

Path #1

Path #2

I am floored as to what could be causing this… Any ideas?

This is across the same interface, same switch, same ports. Just alternating which machine is acting as server and client.

I know this isn't a hardware/driver issue because my sync cards use the same NIC and they are operating at full ~945 Mbps. Albeit, they are directly connected together. However, I do have three VLANs on the interface in question here. Could this be the culprit? All of my interfaces are Intel Server cards, so they should be able handle it

cmcdonald · Jun 2, 2013, 9:43 AM

Okay, I've done some regressing of my network configuration and discovered this is caused by VLANs. Is this expected or is this a driver issue? This occurs when no other traffic is transversing the network.

stephenw10 · Jun 2, 2013, 11:22 AM

Do you have VLAN hardware offloading enabled? I thought these two systems were identical, his does the vlan setup vary?

Steve

cmcdonald · Jun 2, 2013, 2:55 PM

@stephenw10:

Do you have VLAN hardware offloading enabled? I thought these two systems were identical, his does the vlan setup vary?

Steve

Are you referring to TCP segmentation offloading?

And no, both TSO and LRO are disabled (checked) on both machines. I also tried enabling them on both machines and the problem still exists…

stephenw10 · Jun 2, 2013, 5:06 PM

Nope, I meant VLAN hardware tagging. It's not available on all NICs.
However it's also possibly a problem with where the off loading features are set. As they can also be set (or not) on the VLAN interface.

So I had taken some info from other threads of yours, perhaps incorrectly. Please detail the hardware in each machine. Also give the output of ifconfig for each.

Steve

cmcdonald · Jun 2, 2013, 8:00 PM

@stephenw10:

Nope, I meant VLAN hardware tagging. It's not available on all NICs.
However it's also possibly a problem with where the off loading features are set. As they can also be set (or not) on the VLAN interface.

So I had taken some info from other threads of yours, perhaps incorrectly. Please detail the hardware in each machine. Also give the output of ifconfig for each.

Steve

I'll post the ifconfigs when I get off work. But here Ill post the hardware for each machine

Mobo: http://www.newegg.com/Product/Product.aspx?Item=N82E16813121743

Processor: Core i3 3220 ivy bridge

The NICs in question are onboard:

1x 82574
1x 82579

I use the onboard nics for sync and lan

I also have a dual port Intel pro/1000 server adapter which I use for wan, and one for future use.

Left box

$ ifconfig
em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=4009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether 00:15:17:20:d5:00
	inet xxx.xxx.xxx.226 netmask 0xfffffff8 broadcast xxx.xxx.xxx.231
	inet6 fe80::215:17ff:fe20:d500%em0 prefixlen 64 scopeid 0x1 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=4019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso>ether 00:15:17:20:d5:01
	media: Ethernet autoselect
	status: no carrier
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 70:54:d2:ab:24:b9
	inet 192.168.1.1 netmask 0xfffffffc broadcast 192.168.1.3
	inet6 fe80::7254:d2ff:feab:24b9%em2 prefixlen 64 scopeid 0x3 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=5219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:24:ba
	inet6 fe80::7254:d2ff:feab:24ba%em3 prefixlen 64 scopeid 0x4 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	nd6 options=43 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33664
em3_vlan10: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan10 prefixlen 64 scopeid 0x9 
	inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 10 parent interface: em3
em3_vlan20: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan20 prefixlen 64 scopeid 0xa 
	inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 20 parent interface: em3
em3_vlan30: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan30 prefixlen 64 scopeid 0xb 
	inet 172.16.0.2 netmask 0xfffffc00 broadcast 172.16.3.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 30 parent interface: em3
vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet xxx.xxx..xxx.225 netmask 0xfffffff8 
	carp: MASTER vhid 1 advbase 1 advskew 0
vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.10.1 netmask 0xffffff00 
	carp: MASTER vhid 2 advbase 1 advskew 0
vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.20.1 netmask 0xffffff00 
	carp: MASTER vhid 3 advbase 1 advskew 0
vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 172.16.0.1 netmask 0xfffffc00 
	carp: MASTER vhid 4 advbase 1 advskew 0
pptpd0: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500</pointopoint,noarp,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,promisc,simplex,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso></broadcast,oactive,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>

Right box


$ ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether 00:15:17:6a:df:b8
	inet xxx.xxx.xxx.227 netmask 0xfffffff8 broadcast xxx.xxx.xxx.231
	inet6 fe80::215:17ff:fe6a:dfb8%em0 prefixlen 64 scopeid 0x1 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=4019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso>ether 00:15:17:6a:df:b9
	media: Ethernet autoselect
	status: no carrier
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 70:54:d2:ab:1e:32
	inet 192.168.1.2 netmask 0xfffffffc broadcast 192.168.1.3
	inet6 fe80::7254:d2ff:feab:1e32%em2 prefixlen 64 scopeid 0x3 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=5219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:1e:33
	inet6 fe80::7254:d2ff:feab:1e33%em3 prefixlen 64 scopeid 0x4 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	nd6 options=43 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33664
em3_vlan10: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan10 prefixlen 64 scopeid 0x9 
	inet 192.168.10.3 netmask 0xffffff00 broadcast 192.168.10.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 10 parent interface: em3
em3_vlan20: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan20 prefixlen 64 scopeid 0xa 
	inet 192.168.20.3 netmask 0xffffff00 broadcast 192.168.20.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 20 parent interface: em3
em3_vlan30: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan30 prefixlen 64 scopeid 0xb 
	inet 172.16.0.3 netmask 0xfffffc00 broadcast 172.16.3.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 30 parent interface: em3</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso></broadcast,oactive,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast>

stephenw10 · Jun 2, 2013, 9:10 PM

Hmm, OK
The left box has em3 and all it's VLAN interfaces set with the 'promisc' flag and the right box does not. That could be causing some large overhead as the NIC/CPU has to respond to all packets. This could be the result of some test you have run or maybe something else.

However the test is faster when the iperf server is running on the left box.

Which of the interfaces, em0-3, is which hardware NIC?

Steve

cmcdonald · Jun 3, 2013, 4:59 AM

~~Okay I've discovered that upon assigning CARP VIPS, the respective interfaces (and parent interface) automatically enable promiscuous mode.~~

Ugh nvm… I really want to figure this out >:(

em0-1 is dual port server card (WAN and unassigned)
em2-3 is mobo nics (LAN and Sync, repsectively)

Edit:

I am now seeing the speed drop ~~consistent~~ in both directions:

Left box:

$ ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether 00:15:17:20:d5:00
	inet xxx.xxx.xxx.226 netmask 0xfffffff8 broadcast xxx.xxx.xxx.231
	inet6 fe80::215:17ff:fe20:d500%em0 prefixlen 64 scopeid 0x1 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=4019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso>ether 00:15:17:20:d5:01
	media: Ethernet autoselect
	status: no carrier
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 70:54:d2:ab:24:b9
	inet 192.168.1.1 netmask 0xfffffffc broadcast 192.168.1.3
	inet6 fe80::7254:d2ff:feab:24b9%em2 prefixlen 64 scopeid 0x3 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=5219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:24:ba
	inet6 fe80::7254:d2ff:feab:24ba%em3 prefixlen 64 scopeid 0x4 
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=0<> metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	nd6 options=43 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33664
em3_vlan10: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan10 prefixlen 64 scopeid 0x9 
	inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 10 parent interface: em3
em3_vlan20: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan20 prefixlen 64 scopeid 0xa 
	inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 20 parent interface: em3
em3_vlan30: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:24:ba
	inet6 fe80::215:17ff:fe20:d500%em3_vlan30 prefixlen 64 scopeid 0xb 
	inet 172.16.0.1 netmask 0xfffffc00 broadcast 172.16.3.255
	nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 30 parent interface: em3
pptpd0: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500</pointopoint,noarp,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso></broadcast,oactive,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast>

Right box:

$ ifconfig
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether 00:15:17:6a:df:b8
	inet xxx.xxx.xxx.227 netmask 0xfffffff8 broadcast xxx.xxx.xxx.231
	inet6 fe80::215:17ff:fe6a:dfb8%em0 prefixlen 64 scopeid 0x1 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=4019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso>ether 00:15:17:6a:df:b9
	media: Ethernet autoselect
	status: no carrier
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 70:54:d2:ab:1e:32
	inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
	inet6 fe80::7254:d2ff:feab:1e32%em2 prefixlen 64 scopeid 0x3 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=5219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso>ether 70:54:d2:ab:1e:33
	inet6 fe80::7254:d2ff:feab:1e33%em3 prefixlen 64 scopeid 0x4 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
enc0: flags=0<> metric 0 mtu 1536
pflog0: flags=100 <promisc>metric 0 mtu 33144
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
em3_vlan10: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan10 prefixlen 64 scopeid 0x9 
	inet 192.168.10.2 netmask 0xffffff00 broadcast 192.168.10.255
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 10 vlanpcp: 0 parent interface: em3
em3_vlan20: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan20 prefixlen 64 scopeid 0xa 
	inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 20 vlanpcp: 0 parent interface: em3
em3_vlan30: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=103 <rxcsum,txcsum,tso4>ether 70:54:d2:ab:1e:33
	inet6 fe80::215:17ff:fe6a:dfb8%em3_vlan30 prefixlen 64 scopeid 0xb 
	inet 172.16.0.2 netmask 0xfffffc00 broadcast 172.16.3.255
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 30 vlanpcp: 0 parent interface: em3</full-duplex></performnud></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,tso4></up,broadcast,running,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso></broadcast,oactive,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast>

cmcdonald · Jun 3, 2013, 4:51 AM

------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[  8] local 192.168.10.2 port 44835 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  8]  0.0-10.0 sec   935 MBytes   784 Mbits/sec

------------------------------------------------------------
Client connecting to 192.168.10.2, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[  8] local 192.168.10.1 port 29825 connected with 192.168.10.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  8]  0.0-10.0 sec   828 MBytes   695 Mbits/sec

stephenw10 · Jun 3, 2013, 8:09 AM

What change did you make to remove the promiscuous flag from left em3?

Since that seems to have had the opposite effect what happens if you enable promiscuous mode on both em3 NICs?

Steve

cmcdonald · Jun 3, 2013, 5:45 PM

@stephenw10:

What change did you make to remove the promiscuous flag from left em3?

Since that seems to have had the opposite effect what happens if you enable promiscuous mode on both em3 NICs?

Steve

Here is what I've done. Removed all interfaces (sans WAN interface, I need this to vpn into my boxes). Removed all VLAN interfaces. and Removed all CARP VIPs on all interfaces. I noticed that simply adding CARP VIPs forces all respective interfaces into promiscuous mode. Then I went and reconfigured everything.

stephenw10 · Jun 3, 2013, 7:36 PM

Enabling any sort of virtual interface on a physical one necessitates promiscuous mode as it has to respond to more than one MAC address.

I have no idea why that would speed up the throughput though. Anyone else? :-\

Steve

cmcdonald · Jun 4, 2013, 3:01 AM

@stephenw10:

Enabling any sort of virtual interface on a physical one necessitates promiscuous mode as it has to respond to more than one MAC address.

I have no idea why that would speed up the throughput though. Anyone else? :-\

Steve

Thanks for the help! I have posted my most recent ifconfigs and tests. Also, I also committed one of my boxes to the most recent 2.1 build to see if any difference would be made. The version difference now adds another variable to the mix but I had no more ideas…

Anybody?

cmcdonald · Jun 6, 2013, 3:34 AM

Correct me if I am wrong, but as I understand it, VLAN tagging occurs when packets leave an interface. Therefore, my results can be interpreted in the following way:

First, FreeBSD 8.3 em drivers tag packets "faster" than 8.1 drivers. This can be seen by observing the performance difference between vlan tagging on traffic originating from a 2.1 and a 2.0.3 box (see my first post). Path #1 reveals a performance metric slightly higher than Path #2. Path #1 traffic is originating from a pfSense 2.1 box, Path #2 traffic is originating from a pfSense 2.0.3 box. I would imagine that if I updated my "left" box to 2.1, that I would obtain slightly higher throughput.

Second, I performed another test by disabling all VLANs on the "right" box and simply untagged its LAN switch port to VLAN 10 (Management). Simply, ALL VLAN TAGGING IS DISABLED on the "right" box. Here are the results:
Path #1

Path #2

Notice that when a box is not tagging VLANS, the outbound performance shoots up again! This tells me a few things:

First, my switch isn't the problem. As these packets are having to be tagged before approaching the "left" box, or untagged when approaching the "right" box. My switch is handling the tagging/untagging just fine. Second, this further supports my first interpretation that VLAN tagging on pfSense is what is causing the performance degradation.

Now, is this simply a limitation to my pfSense hardware? My hardware is already "overkill" by most pfSense standards. I like to look "ahead" when making hardware purchases. So this was intentional. However, is it safe to put this to rest and call it a hardware limitation or is there something I am overlooking. Some configuration problem that is causing the performance degradation?

stephenw10 · Jun 6, 2013, 8:54 AM

To test your tagging theory you should have disabled the VLANs on the left box. As is stands the results above are the same as with VLAN tagging on the box, left-to-right 700Mb and right-to-left 900Mb.

Also I'm pretty sure both boxes were running 2.0.3 when I last looked at this thread, did you upgrade one? Try upgrading both. The newer drivers in 8.3 might well be making better use of the hardware features of the NIC.

Steve

cmcdonald · Jun 6, 2013, 8:40 PM

@stephenw10:

To test your tagging theory you should have disabled the VLANs on the left box. As is stands the results above are the same as with VLAN tagging on the box, left-to-right 700Mb and right-to-left 900Mb.

Also I'm pretty sure both boxes were running 2.0.3 when I last looked at this thread, did you upgrade one? Try upgrading both. The newer drivers in 8.3 might well be making better use of the hardware features of the NIC.

Steve

You are correct, but notice that in the original post, the "right-to-left" test was performed from a 2.1 box. The speeds are clearly slower than usual. However, upon disabling VLANS on the right box, "right-to-left" performance jumped back up to 900+ Mbps (which is about as fast as you'll ever see gigabit flow). My left box is acting as my production box at the moment. Although they will both be configured for CARP soon, I will have to get my "right" box assuming the production roles before attempting such a test. I might take down the VLANS on the production box tonight and run tests.

Simply, the "right-to-left" vs "left-to-right" difference seen in my original post is simply attributed to more efficient driver support in 8.3 (as opposed to 8.1).

What I would be very curious in seeing other people's iperf results with similar conditions. Basically, running the client behind an interface that is tagging multiple VLANs.

cmcdonald · Jun 7, 2013, 4:26 AM

Update: I decided to figure out which of the two OEM cards I am tagging on.

I simple command revealed the following:


$ pciconf -lv
em0@pci0:1:0:0: class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06 hdr=0x00
    class      = network
    subclass   = ethernet
em1@pci0:1:0:1: class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06 hdr=0x00
    class      = network
    subclass   = ethernet
em2@pci0:0:25:0:        class=0x020000 card=0x202d8086 chip=0x15028086 rev=0x05 hdr=0x00
    class      = network
    subclass   = ethernet
em3@pci0:3:0:0: class=0x020000 card=0x202d8086 chip=0x10d38086 rev=0x00 hdr=0x00
    class      = network
    subclass   = ethernet

I know that the em0/1 cards are identical (hence the same chip value 0x[DevID][VendorID]) A simple search of the list of retail cards http://www.intel.com/support/network/sb/cs-012904.htm reveals that it is indeed a Intel Pro/1000 PT dual card. As expected.

The manual isn't very clear which of the two board NICs is the 82579LM or the 82574L. However, after bouncing the DeviceID and VendorID against the OEM database, I have discovered that my LAN interface is using the 82574L card and not the 82579LM card.

Here are their respective ARK sheets:
http://ark.intel.com/products/47620/Intel-82579LM-Gigabit-Ethernet-PHY
http://ark.intel.com/products/32209/Intel-82574L-Gigabit-Ethernet-Controller

Some interesting finds:

First, the 82574L Controller is also used the Pro/1000 CT Desktop retail boards.
Second, the 82579LM is also ~3 years newer.
Third, clearly I should either be using the second port on my PT board or the 82579LM. I believe I am using the "worst" of the three controllers in my boxes for my LAN interface.

These boxes are about 900 miles away from my present location. I do all of my work remotely. I will be getting one of my guys on site to move some cables around in the morning and report back with new metrics.

I have both boxes connected directly through the 82579LM (this is my dedicated sync interface). Is it possible to tag and trunk vlans over a direct connection without a switch? I might give this a try as well to see how performance goes over the 8259LM (instead of having to move cables around).

Edit #1:
Okay here is the current testing setup:

I am now tagging on the 8259LM card on both boxes. I have three VLANS and static IPs are assigned across the board.
VLAN 100, 200, 300
left 100: 192.168.1.1/24
left 200: 192.168.2.1/24
left 300: 192.168.3.1/24

right 100: 192.168.1.2/24
right 200: 192.168.2.2/24
right 300: 192.168.3.2/24

Here are the results:
Path #1:

Path #2:

Now, unless my switch is causing these issues (which is very unlikely, I have already consulted my switch manufacturers message boards), the issue has got to be the 82574L (CT Desktop card).

Edit #2:

I just did a serious hardcore test using the setup in Edit #1.

Here is what I did. Ran an iPerf across VLAN 100 from "right-to-left" for 60 seconds. While running that test, I then ran another iperf across VLAN 200 from "left-to-right". Even saturating the bandwidth both tests averaged at around 880 Mbps. That isn't bad at all! Considering packets were flying like mad.

stephenw10 · Jun 7, 2013, 9:32 AM

Ah, that's an interesting result. :)
I will say that the 82574 was until very recently one of Intel's most popular NIC chips, used just about everywhere. I would be very surprised if it was a hardware problem. That's the chip that suffered the 'packet of death' and the reason it was so worrying was that chip is everywhere. PoD not an issue by the way. ;)

Is it possible to tag and trunk vlans over a direct connection without a switch?

I take it you have discovered it is possible.

Steve

cmcdonald · Jun 7, 2013, 2:08 PM

@stephenw10:

Ah, that's an interesting result. :)
I will say that the 82574 was until very recently one of Intel's most popular NIC chips, used just about everywhere. I would be very surprised if it was a hardware problem. That's the chip that suffered the 'packet of death' and the reason it was so worrying was that chip is everywhere. PoD not an issue by the way. ;)

Is it possible to tag and trunk vlans over a direct connection without a switch?

I take it you have discovered it is possible.

Steve

Okay I can confirm that moving my LAN connection to the 82579LM (and changing the parent interface of the three VLANs) results in 915-935 Mbps speeds in both directions! Sweet! :) I had my contact move the LAN connection to the 82579LM on both boxes and I just made the configuration change. So my findings were correct. The 82574L just doesn't handle VLAN tagging as well as it could. Clearly from the ARK sheets, the 82574L controller was designed more for endpoint devices, not server hardware. However, the 82579LM is a much more capable controller.

Thanks

stephenw10 · Jun 7, 2013, 2:33 PM

You could try disabling the hardware VLAN tagging on the 82574 and do it in software instead. I believe the command for that would be:

ifconfig em3 -VLAN_HWTAGGING

Interestingly I see that it also supports VLAN_HWFILTER which em2 does not. Perhaps the FreeBSD driver in 2.0.X attempts to use this feature and it ends up actually slowing the connection?

Stve