2100 Errors in on WAN

Lifelike

I'm having trouble with errors in, on WAN. I've swapped cables, connected directly to ONT vs through ISP modem, placed a switch in front of pfsense box directly from ONT, and all three have the same issue. The only difference being how quickly the errors pile up. Directly from ONT is significantly faster pile up of errors, usually with a slower DL (5-100mb/s) but fast UL (~700 mb/s). With modem / switch in front it's around ~700/700 mb/s.

Have ran the speedtest directly on pfsense taking out LAN and have the same results as below. It could be on the provider side, but I'm not sure what to look at to determine that's the case. I've been unsuccessful when trying to find information on the specific errors. WAN is currently on autoselect in pfsense, and inside ISP modem bypassing pfsense all together lists as "Full-Duplex 1,000 Mbps". I've tried forcing to all 1000bases in pfsense without any difference.

If I start a speedtest, the errors increase by ~100 per run.

What would mac crc and rx errors indicate is the problem? Physically a mac address of pfSense that my ISP doesn't like? I've also tried spoofing pfsense mac address to match what my ISP provided router says with the same result.

Current ISP, if it's of any help, is Verizon Fios.

The box has been mostly idle when these were pulled, the packets in / out can be significantly lower with the same amount of errors. Usually resulting in >20% packet loss taking down the interface.

[22.05-RELEASE][admin@pfsense.localdomain]/root: sysctl dev.mvneta.0
dev.mvneta.0.lpi: 0
dev.mvneta.0.flow_control: 0
dev.mvneta.0.mib.reset: 0
dev.mvneta.0.mib.watchdog: 0
dev.mvneta.0.mib.overrun: 0
dev.mvneta.0.mib.rx_discard: 19313
dev.mvneta.0.mib.mac_late_collision: 0
dev.mvneta.0.mib.mac_collision: 0
dev.mvneta.0.mib.mac_crc_err: 4332
dev.mvneta.0.mib.mac_rx_err: 8682
dev.mvneta.0.mib.pkt_jabber: 0
dev.mvneta.0.mib.pkt_oversize: 0
dev.mvneta.0.mib.pkt_fragment: 3
dev.mvneta.0.mib.pkt_undersize: 0
dev.mvneta.0.mib.fc_rx_bad: 0
dev.mvneta.0.mib.fc_rx_good: 260
dev.mvneta.0.mib.fc_tx: 0
dev.mvneta.0.mib.tx_mac_ctl_err: 0
dev.mvneta.0.mib.tx_bcast_frame: 2188
dev.mvneta.0.mib.tx_mcast_frame: 159
dev.mvneta.0.mib.tx_exces_collision: 0
dev.mvneta.0.mib.tx_good_frame: 4624717
dev.mvneta.0.mib.tx_good_oct: 4503029218
dev.mvneta.0.mib.rx_fame_1024_max: 7308566
dev.mvneta.0.mib.rx_frame_512_1023: 15213
dev.mvneta.0.mib.rx_frame_256_511: 20290
dev.mvneta.0.mib.rx_frame_128_255: 69964
dev.mvneta.0.mib.rx_frame_65_127: 2080976
dev.mvneta.0.mib.rx_frame_1_64: 532139
dev.mvneta.0.mib.rx_mcast_frame: 242455
dev.mvneta.0.mib.rx_bcast_frame: 42819
dev.mvneta.0.mib.rx_bad_frame: 13017
dev.mvneta.0.mib.rx_good_frame: 5721410
dev.mvneta.0.mib.tx_mac_err: 0
dev.mvneta.0.mib.rx_bad_oct: 5263671
dev.mvneta.0.mib.rx_good_oct: 6500423677
dev.mvneta.0.rx.queue0.threshold_timer_us: 100
dev.mvneta.0.%parent: simplebus1
dev.mvneta.0.%pnpinfo: name=ethernet@30000 compat=marvell,armada-3700-neta
dev.mvneta.0.%location: 
dev.mvneta.0.%driver: mvneta
dev.mvneta.0.%desc: NETA controller

From the above, pkt_framgmenet of 3 seems valid + mac_crc_err (4332) + mac_rx_err (8682) = rx_bad_frame (13017)

stephenw10

One thing you could do here is re-assign one of the LAN ports as the WAN and see if you still see the same error rate there:
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/configuring-the-switch-ports.html

Steve

Lifelike

Thanks for the suggestion, that seems to work at least through the modem which would have >1000+ errors for the amount of tests I ran.

I had this problem a couple years ago after purchasing, but was quick to blame comcast, and let it collect dust until moving to a new ISP and location so it is well out of warranty...

Two questions, does that mean the WAN nic is bad (mvneta0)?
If so, and this would be the permanent WAN slot, am I now limited to a bottleneck pushing packets through LAN? As all speedtests from multiple locations were ~500/~500 or is that just a coincidence and I could still theoretically get 900up/900down?

stephenw10

Using WAN and LAN on the same NIC does restrict the throughput because each mvneta NIC is only since a single queue.

However I would test using mvneta0 as a LAN and see if you still see errors. You may not when connecting to something different. If you do though that does point to a hardware issue.

Steve

Lifelike

Seeing about the same, 1k per test when using mvneta0 as a LAN port. That's disappointing, oh well, at least it can be ruled out that it isn't an issue down the line of my ISP outside of my direct control. I guess I'll deal with splitting on LAN side for the time being on the 2100 and when I get some time, run pfSense off some old server parts I have around.

Appreciate your help Steve!

stephenw10

Mmm, that is disappointing. Probably nothing we can do though.