New install on NUC12 - lots of missed packets and slow upload

mtis

Hi all.

I've just installed pfSense on my NUC12 which has 1x I225-V, used for WAN and 1x I225-V used for LAN. During my testing I've noticed on the dashboard for "errors in" on the LAN interface it reads 37391. sysctl shows this is all missed packets (dev.igc.1.mac_stats.missed_packets: 37391). What is the cause of this? Is this normal? I have been running speedtests. I have a 1GB symmetric line and I get over 900Mbps down but only ~70Mbps up. Related?

I've disabled hardware checksum offload.

Thanks in advance.

stephenw10

No that's not normal. What's it connected to on the LAN side? What does it show the link status as? Is it actually linked at the expected rate?

MTU mismatch? flow-control mismatch perhaps?

Steve

mtis

@stephenw10 It's connected to a Unifi US 8 60W. Link is showing as 1000baseT <full-duplex>. Flow control disabled on the switch and I believe flow control disabled on pfSense but I'm not entirely sure. sysctl dev.igc.1.fc: 0. MTU default 1500.

stephenw10

What does ifconfig -vvvm igc1 show?

You might need to enable flow control if it's dropping packets.

mtis

igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=48020b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,HWSTATS,MEXTPG>
        capabilities=4f43fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 48:21:0b:xx:xx:xx
        inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
        inet6 fe80::4a21:bff:fexx:xxxx%igc1 prefixlen 64 scopeid 0x2
        inet6 fe80::1:1%igc1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 2500Base-T
                media 1000baseT
                media 1000baseT mediaopt full-duplex
                media 100baseTX mediaopt full-duplex
                media 100baseTX
                media 10baseT/UTP mediaopt full-duplex
                media 10baseT/UTP
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

Mmm, nothing unusual there. Does netstat -i show drops or errors (or neither)?

mtis

Yes it shows the same number under Ierrs for igc1, 0 under Idrop

stephenw10

No other errors in the mac stat counters?

I normally expect 'missed packets' to be the result of no CPU cycles to service the queue but in a NUC I imagine it's massively over-powered. Hard to think it could exhaust anything at 1Gbps.

Since you're seeing only 70Mbps upload I'd check the basics, cable, ports at each end etc.

mtis

I'll rule out the basics. I'll swap the LAN and WAN ports round too in case it's a hardware issue with the I226-V (which is a NUC expansion board).

stephenw10

We have flow-control enabled by default for the i226-V NICs in the 4200. I would try enabling it.

mtis

@stephenw10 Interesting results! I've enabled flow control on the unifi switch and the errors have disappeared, and upload is now full speed. Changing flow control on pfSense has no affect whatsoever so I guess I will leave it as default. I've also found I can't change flow control for the 2 interfaces independently, I wanted to disable for igc0 (WAN) and enable for igc1 (LAN) but having just one (e.g. dev.igc.0.fc 0) disables it for both.

@stephenw10 thank you for the suggestion!

TheNarc

@mtis @stephenw10 Just been following this out of curiosity since I've got a box with I226-V NICs as well, although I disabled flow control on them. It's only been running for a month or two but gets heavy usage, and I haven't seen any errors like this. Is there a particular reason that flow control is enabled for the 4200? I wish I could remember what/where I read that disabling it might be a good idea . . .

stephenw10

It's enabled by default on all our devices. That usually doesn't make much difference but it's really dependent on what it's connected to. Some things require it be enabled or disabled to get best performance.

TheNarc

@stephenw10 Makes sense, thank you!

mtis

Yes I’m not really clear why I need it enabled. I have 1GbE on all links, WAN to ISP ONT, LAN to switch, and then all devices connected to the switch. Why do I need flow control for this? I never needed it with OpenWrt.

mtis

I’m worried that enabling flow control is masking the symptom but not resolving the underlying issue itself. Is it possible there is a bigger problem here, or is flow control sometimes required in a setup like this.

TheNarc

@mtis I would defer to those more knowledgeable than myself on the subject, but from what I can tell, if enabling it got rid of the errors you were getting and did not introduce any other unexpected issues like reduced throughput, I probably wouldn't worry about it. Most of the advice I could find about whether to enable flow control or not seemed to just boil down to "try it both ways and see which works best for you", which is deeply unsatisfying for those of us who enjoy understanding why one thing works better than another. But it seems that to get there in this case would require a much deeper level of understanding and/or analysis of your particular network topology than I am capable of, at least.

stephenw10

The one thing it could be masking is that for some reason the LAN interface is unable to keep up with the rate the traffic is arriving. You would not expect that to be possible on a 1G link into a 2.5G NIC.

Check the the CPU usage graphs in Status > Monitoring. Is it ever getting close to 100%/

mtis

Update from me. Something didn't feel right. And I found it. Intel ASPM. Once I disabled this on the NUC in BIOS settings everything worked as expected.

stephenw10

Nice! Yeah we've seen ASPM cause all sorts of issues.