APU.1C LAN latency
-
I have WAN (DHCP) plus LAN (10.52.0.1/24) and OPT1 (10.52.1.1/24). When I plug in to LAN and do stuff (move around the webGUI, or download data or…) I get very high ping times from my laptop to LAN IP 10.52.0.1 - e.g.
LAN re2
WAN re1
OPT1 re0[2.2-BETA][root@apu22.localdomain]/root: ifconfig re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate>ether 00:0d:b9:33:88:88 inet6 fe80::20d:b9ff:fe33:8888%re0 prefixlen 64 scopeid 0x1 inet 10.52.1.1 netmask 0xffffff00 broadcast 10.52.1.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (none) status: no carrier re1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate>ether 00:0d:b9:33:88:89 inet6 fe80::20d:b9ff:fe33:8889%re1 prefixlen 64 scopeid 0x2 inet 192.168.1.107 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>) status: active re2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate>ether 00:0d:b9:33:88:8a inet6 fe80::20d:b9ff:fe33:888a%re2 prefixlen 64 scopeid 0x3 inet 10.52.0.1 netmask 0xffffff00 broadcast 10.52.0.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex,master>) status: active pflog0: flags=100 <promisc>metric 0 mtu 33144 pfsync0: flags=0<> metric 0 mtu 1500 syncpeer: 224.0.0.240 maxupd: 128 defer: on syncok: 1 lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 nd6 options=21 <performnud,auto_linklocal>enc0: flags=0<> metric 0 mtu 1536 nd6 options=21 <performnud,auto_linklocal>ovpnc1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500 options=80000 <linkstate>inet6 fe80::20d:b9ff:fe33:8888%ovpnc1 prefixlen 64 scopeid 0x8 inet 10.99.0.2 --> 10.99.0.1 netmask 0xffffffff nd6 options=21 <performnud,auto_linklocal>Opened by PID 18077</performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></promisc></full-duplex,master></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate></up,broadcast,running,simplex,multicast>
And when generating a bits of traffic on LAN, ping times go like:
Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time=1408ms TTL=64 Reply from 10.52.1.1: bytes=32 time=510ms TTL=64 Reply from 10.52.1.1: bytes=32 time=765ms TTL=64 Reply from 10.52.1.1: bytes=32 time=557ms TTL=64 Reply from 10.52.1.1: bytes=32 time=1022ms TTL=64 Reply from 10.52.1.1: bytes=32 time=952ms TTL=64 Reply from 10.52.1.1: bytes=32 time=911ms TTL=64 Reply from 10.52.1.1: bytes=32 time=382ms TTL=64 Reply from 10.52.1.1: bytes=32 time=908ms TTL=64 Reply from 10.52.1.1: bytes=32 time=662ms TTL=64 Reply from 10.52.1.1: bytes=32 time=49ms TTL=64 Reply from 10.52.1.1: bytes=32 time=1410ms TTL=64 Reply from 10.52.1.1: bytes=32 time=71ms TTL=64 Reply from 10.52.1.1: bytes=32 time=1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time=2070ms TTL=64 Reply from 10.52.1.1: bytes=32 time=594ms TTL=64 Reply from 10.52.1.1: bytes=32 time=1105ms TTL=64 Reply from 10.52.1.1: bytes=32 time=692ms TTL=64 Reply from 10.52.1.1: bytes=32 time=750ms TTL=64 Reply from 10.52.1.1: bytes=32 time=666ms TTL=64 Reply from 10.52.1.1: bytes=32 time=577ms TTL=64 Reply from 10.52.1.1: bytes=32 time=325ms TTL=64 Reply from 10.52.1.1: bytes=32 time=719ms TTL=64 Reply from 10.52.1.1: bytes=32 time=180ms TTL=64 Reply from 10.52.1.1: bytes=32 time=528ms TTL=64 Reply from 10.52.1.1: bytes=32 time=338ms TTL=64 Reply from 10.52.1.1: bytes=32 time=371ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64
Then I move the cable from LAN (re2) to OPT1 (re0) and all is well, I can generate traffic and there is no latency issue:
[2.2-BETA][root@apu22.localdomain]/root: ifconfig re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate>ether 00:0d:b9:33:88:88 inet6 fe80::20d:b9ff:fe33:8888%re0 prefixlen 64 scopeid 0x1 inet 10.52.1.1 netmask 0xffffff00 broadcast 10.52.1.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>) status: active re1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate>ether 00:0d:b9:33:88:89 inet6 fe80::20d:b9ff:fe33:8889%re1 prefixlen 64 scopeid 0x2 inet 192.168.1.107 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>) status: active re2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate>ether 00:0d:b9:33:88:8a inet6 fe80::20d:b9ff:fe33:888a%re2 prefixlen 64 scopeid 0x3 inet 10.52.0.1 netmask 0xffffff00 broadcast 10.52.0.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (none) status: no carrier pflog0: flags=100 <promisc>metric 0 mtu 33144 pfsync0: flags=0<> metric 0 mtu 1500 syncpeer: 224.0.0.240 maxupd: 128 defer: on syncok: 1 lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 nd6 options=21 <performnud,auto_linklocal>enc0: flags=0<> metric 0 mtu 1536 nd6 options=21 <performnud,auto_linklocal>ovpnc1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500 options=80000 <linkstate>inet6 fe80::20d:b9ff:fe33:8888%ovpnc1 prefixlen 64 scopeid 0x8 inet 10.99.0.2 --> 10.99.0.1 netmask 0xffffffff nd6 options=21 <performnud,auto_linklocal>Opened by PID 27419</performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></promisc></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate></up,broadcast,running,simplex,multicast>
Ping times are fine:
Reply from 10.52.1.1: bytes=32 time=6ms TTL=64 Reply from 10.52.1.1: bytes=32 time=4ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time=12ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time=2ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time=13ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64 Reply from 10.52.1.1: bytes=32 time=1ms TTL=64 Reply from 10.52.1.1: bytes=32 time<1ms TTL=64
Then I re-assigned the devices - LAN to re0 and OPT1 to re2, and rebooted. The problem followed LAN - now on re0 - and OPT1 on re2 worked fine. So not a hardware problem with the physical port.
Then I disabled LAN and removed it from the assigned list. Now the problem was on OPT1.
Put LAN back and set it all up again, and rebooted. Problem is back on LAN.
I can't see anything special in the config about LAN (or OPT1 when LAN did not exist).It does seem to be that whichever device has "master" in ifconfig has the problem:
media: Ethernet autoselect (1000baseT <full-duplex,master>)</full-duplex,master>
and when the device does not have "master" in the media properties, it works fine:
media: Ethernet autoselect (1000baseT <full-duplex>)</full-duplex>
Whatever "master" is about seems to be the issue. I suspect that if I can get rid of that "master" setting in ifconfig it will all be happy.
I can't say if this happens on other hardware - the APU is the only Gbit hardware I have!
Added: here is a link to a reasonable article about link auto-negotiation: http://www.dell.com/content/topics/global.aspx/power/en/ps1q01_hernan?c=us&l=en&cs=04
Something is making the "re" device become "master" for LAN but not for OPT1. And perhaps "re" device, or the realtek hardware itself does not function well as the master of a Gbit link. -
Have you tried turning off hardware checksum offload, hardware TCP segmentation offload, and hardware large receive offload?
Any In/Out errors, or collisions show up on the interface status page?
Not sure if you've seen this, either: https://redmine.pfsense.org/issues/3870. Is everything connecting to the APU set to autonegotiate?
-
I have now checked System->Advanced, Networking, Disable hardware checksum offload, and rebooted.
Problem symptoms are the same - when I plug a network cable from my laptop into whichever physical port (re2 or re0) is assigned to LAN then when I do real network activity the local ping time goes up crazy. The latency is real because just navigating around the webGUI is very slow.
When I plug the cable into whichever physical port is assigned to OPT1 then everything is snappy - local ping time remains very low, webGUI navigates from page to page in a flash.
It is the same laptop client and cat6 cable in all cases. I have used a different cable also with same results.I am really wondering what is special about the "logical" LAN in pfSense that can effect how the real hardware is setup, vs using OPT1 as a LAN-style interface. (in theory there should be no difference, but I am seeing this very measurable different behaviour).
Also, if I can sort out the hardware, I will try an APU.1C with 2.1.5 and see if I can induce the same symptoms.
-
You have traffic shaping or limiters configured on there?
-
Apologies, I am an idiot.
I reset to factory defaults and set up basic stuff and it all worked fine.
Then put back the original config and then looked hard at the config - there were some traffic shaper queues set up on LAN! One had "default queue" checked and limited to 111 Kbps. Removed all traffic shaper settings and now everything is fine. There is no software, firmware or hardware issue, it did exactly what it was told.
I had completely forgotten that I had played with Traffic Shaper a while ago.
Sorry for the time waste - to my friends, and to me.