IN/OUT errors on every pfSense interfaces. What would you do according to your experience?
-
Dear Users,
during the last days we are experiencing some intermittent network perfomance degradation.
We checked everything, but we noticed only some in/out errors on our border router interfaces.Below you can find the interfaces statistics.
Please note that our router (based on pfSense 2.7.2) has 2 P2P links to the ISP (WAN1 nd WAN2) managed using BGP.
Currently, WAN2 is the primary link.
PUBLIC interface is the interface used to present our public subnet to ISP.If I'm not wrong, all the 3 interfaces have in/out errors. Do you think that they are related to faulty cables?
In my opinion it is very strange: 3 cables or 3 SFP+ transceivers damaged at the same time.What is your point of view about this issue?
Additional notes:
- router WAN1 and WAN2 10Gbps interfaces are directly connected to the ISP via two different border switches (provided and managed by the ISP)
- router PUBLIC 10Gbps interface is connected to our border switch (where public hosts are connected)
WAN1
MTU 1500
Media 10Gbase-SR <full-duplex>
In/out packets 1153688/9860472 (42.83 MiB/1.08 GiB)
In/out packets (pass) 1153688/9860472 (42.83 MiB/1.08 GiB)
In/out packets (block) 51789/0 (2.88 MiB/0 B)
In/out errors 10272/0
Collisions 0
Interrupts 11023076 (24/s)WAN2
MTU 1500
Media 10Gbase-SR <full-duplex>
In/out packets 84734635666/31887450415 (108.79 TiB/31.46 TiB)
In/out packets (pass) 84734635666/31887450415 (108.79 TiB/31.46 TiB)
In/out packets (block) 4794229/22 (222.20 MiB/1 KiB)
In/out errors 122459/0
Collisions 0
Interrupts 16197060089 (35140/s)PUBLIC
MTU 9000
Media 10Gbase-SR <full-duplex>
In/out packets 31742772795/84658044844 (31.33 TiB/108.72 TiB)
In/out packets (pass) 31742772795/84658044844 (31.33 TiB/108.72 TiB)
In/out packets (block) 571298/16 (21.78 MiB/960 B)
In/out errors 1683731/0
Collisions 0
Interrupts 14916949041 (32364/s)Thank you in advance,
Mauro -
@mauro-tridici those numbers are pretty minuscule compared to the packets in..
In my experience I wouldn't be concerned.. Unless say those number went from 0 to those number very quickly..
Those are wan interfaces? You have no idea what sort of odd stuff that interface could see..
-
Those are all input errors. What NICs are those?
Are those errors new?
What CPU are you using? How much traffic is passing?
It could just be maxing out the available cycles so the NICs cannot accept packets.Steve
-
@stephenw10 @johnpoz thank you for sharing with me your experience.
Yes, they are all input errors and they are new. They have been detected and notified by Zabbix which is monitoring the router.
You can find below the info you need:
Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
40 CPUs: 2 package(s) x 10 core(s) x 2 hardware threads2 x network adapters dual port Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet
I'm not an expert, but I can say that when we received the notification the traffic in input in WAN2 interface (the primary ISP link) was about 8Gbps (available bandwidth is 10Gbps)
Tomorrow morning I will ask the ISP to check the ISP switch port connected to WAN1 and WAN2. But I don't understand why I see errors on the PUBLIC interface also. They are input errors, but in this case they are related to the traffic that from LAN goes to internet.
This is a simplified traffic flow (excluding the backup route WAN1):
WAN2 <-> router <-> PUBLIC <-> firewall <-> LAN hostsThank you for the time you are dedicating to my case.
Mauro
-
Hmm, then it feels like something else must have changed at that time. Was pfSense upgraded? Maybe some other config change?
There are no other components shared between those connections as I understand it. Separate switches on each one.
Do you log CPU usage with Zabbix? Did that change when the errors started?
