[SOLVED] Errors In, but no Errors Out and No Collisions on both WAN

[Sergei_Shablovsky]

Hi. pfSense Gurus!

Recently see the Errors in errors on a WAN Statistic Dashboard.

No “Errors Out” and no “Collisions”.

Both WAN ports are on physically different NICs, and both WAN ports are Speed & Duplex forced, MTU 1350.

[Gertjan]

@sergei_shablovsky

Strange.
I started to thinks that these erros/collisions were not shown any more, as I only have seen them being '0' :

login-to-view

What is a "Errors In" ? It's the result between the checksum present in the packet, put in there by the sending NIC, and the checksum build by the local driver (or NIC-chip).
If "Errors Out" start to be non zero, then the drivers is adding wrong checksums to the packets ? ( ! )
Or the driver knows that the packet will get altered further along on the line (in the future ?)
Or is it the receiving NIC who sends back to our NIC that he sends packets with errors.

"Error in" / "Error out" =
Failing NIC chip,
Failing Transceiver,
Failing connector,
Failing uplink cable(s)
Failing upstream devices.
And many other reasons.

[stephenw10]

How many errors? Out of how much total data?

Has the WAN been disconnected at all?

It's common to see some errors if the cable was removed or something upstream was rebooted etc. That's not normally a problem.

Steve

[Sergei_Shablovsky]

@gertjan said in Errors In, but no Errors Out and No Collisions on both WAN:

@sergei_shablovsky

Strange.
I started to thinks that these erros/collisions were not shown any more, as I only have seen them being '0' :

login-to-view

What is a "Errors In" ? It's the result between the checksum present in the packet, put in there by the sending NIC, and the checksum build by the local driver (or NIC-chip).
If "Errors Out" start to be non zero, then the drivers is adding wrong checksums to the packets ? ( ! )
Or the driver knows that the packet will get altered further along on the line (in the future ?)
Or is it the receiving NIC who sends back to our NIC that he sends packets with errors.

Thank You for a beautiful explanation!
Appreciate Your help here!

"Error in" / "Error out" =
Failing NIC chip,
Failing Transceiver,
Failing connector,

Check all of this. Sure on 99,999% all are ok.

But when coming to this stage

Failing uplink cable(s)
Failing upstream devices.

and especially this stage

And many other reasons.

something interesting happened,- when ISP's engineer coming, I receive a lot of news from inside ISP's life:

• recent engineer goes to another company due low income ;)
• ISP's start to shorting the stuff because need to redirect financing on new applience and PR;
• new youth(!) engineer come to site to exchange/upgrade ISP's applience, and of course because of lack of knowledge unable to properly upload and tuning settings on a applience, and no one from Service Dept able to help him;

Due all of this my settings pushed to 100TX full-duplex on WANs interfaces on this pfSense become wrong and after setting to “100TX half-duplex” the numbers of “Errors In” dramatically reduced, but “Errors Out” and “Collisions” come in.

When I try to set all WANs speed&control to “automatic”, physically link exist and up, but no DHCP received, so nothing working at all...

[stephenw10]

If the ISP device you're connected to is set at fixed 100Mb you would need to match that setting in the pfSense NIC. If it's a 1G NIC and you leave it as auto-select it will fail to negotiate a link speed and fall back to it's default which it sounds like is 100M half-duplex.
With that duplex mismatch you will certainly see errors/collisions.

Steve

[Sergei_Shablovsky]

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

How many errors? Out of how much total data?

I reboot the system, and make some measurements. Please look:
(data from pfSense GUI Status -> Interfaces)

Uptime for this data 5h 20min.
Network loading are quite small for this branch today. (Just nothing)

WAN_1
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
490641/359474 (541.09 MiB/33.24 MiB)
In/out packets (pass)
490641/359474 (541.09 MiB/33.24 MiB)
In/out packets (block)
4864/803 (663 KiB/144 KiB)
In/out errors
107/107
Collisions
56365

WAN_2
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
170677/150218 (101.52 MiB/13.74 MiB)
In/out packets (pass)
170677/150218 (101.52 MiB/13.74 MiB)
In/out packets (block)
4827/1650 (826 KiB/211 KiB)
In/out errors
14/14
Collisions
13684

[stephenw10]

Yeah, you can't have that running at half-duplex. The number of collisions there are expected in that situation.

[Sergei_Shablovsky]

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

Yeah, you can't have that running at half-duplex. The number of collisions there are expected in that situation.

Stephen, thank You for patience! ;)

So, I doing some tests and here the result:

Both WANs (mean 1st port on each of NICs) set in pfSense In Interfaces -> WAN_X in
100baseTX <full-duplex>

The output in **Status -> Interfaces”

WAN_1
MTU
1350
Media
100baseTX <full-duplex>
In/out packets
266980/253695 (299.20 MiB/34.30 MiB)
In/out packets (pass)
266980/253695 (299.20 MiB/34.30 MiB)
In/out packets (block)
740/285 (73 KiB/46 KiB)
In/out errors
12123/0
Collisions
0

WAN_2
MTU
1350
Media
100baseTX <full-duplex>
In/out packets
15759/16785 (1.62 MiB/777 KiB)
In/out packets (pass)
15759/16785 (1.62 MiB/777 KiB)
In/out packets (block)
1301/0 (181 KiB/0 B)
In/out errors
281/0
Collisions
0

Both WANs (mean 1st port on each of NICs) set in pfSense In Interfaces -> WAN_X in
100baseTX**

The output in **Status -> Interfaces”

WAN_1
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
58191/43464 (55.42 MiB/7.11 MiB)
In/out packets (pass)
58191/43464 (55.42 MiB/7.11 MiB)
In/out packets (block)
208/0 (13 KiB/0 B)
In/out errors
2/2
Collisions
6407

WAN_2
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
5461/5977 (726 KiB/320 KiB)
In/out packets (pass)
5461/5977 (726 KiB/320 KiB)
In/out packets (block)
177/0 (8 KiB/0 B)
In/out errors
0/0
Collisions
217

Need to add, that on any other interfaces any other NICs (which are identical and we buy at one time from one supplier) there are no In/Out Errors or Collisions

[stephenw10]

The MTU there is very low.

What changed here? Were those ever linked at 1G?

If you set a fixed speed it must match the other end of the link.
If you set half duplex you will always get collisions even if the other end does match.

Steve

[Sergei_Shablovsky]

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

The MTU there is very low.

We need this value for uplink, just ignore it.
Anyway, I make the test for both 100TX and 100TX full duplex, with MTU 1350, 1500 and 1420, and looks like MTU no such involved in this issue...

What changed here? Were those ever linked at 1G?

I try each settings for interface from automatic to highest 1000TX full duplex on Interfaces - WAN_X webpage of pfSense GUI, the result You able to see above...

Shortly to say, only on 100TX (mean 100TX half-duplex), all pfSense updates are ok, no connection reset by server due 1,5-2h connect time, all working.
But I still have In Errors.

[stephenw10]

With half duplex you mat well get errors, you will definitely see collisions.

Nothing is half-duplex these days not since hubs and 10base2! What is it connected to? It seems like something must be configured wrong.

Steve

[Sergei_Shablovsky]

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

With half duplex you mat well get errors, you will definitely see collisions.

Nothing is half-duplex these days not since hubs and 10base2!

Heh ;)

What is it connected to?

I start thinking “connected to a piece of sh****t” ;)
(Some Huawei applience)

It seems like something must be configured wrong.

Also make the same conclusion. But anyway before to start complaining the ISP, I need to be sure that all ok on my end of cable ;)
So, if You have some suggestions, I would be thankful to read it :)

[Gertjan]

@sergei_shablovsky
Put a switch between the ISP device, and pfSense.
First, hook up ,the switch to WAN-pfSense, and see that the connection negotiates a connection using 'auto on both side (the switch will only have the auto mode).

Then make the connection between the switch and the ISP device.

[stephenw10]

Yup, that would be a good test.

[Sergei_Shablovsky]

@gertjan said in Errors In, but no Errors Out and No Collisions on both WAN:

@sergei_shablovsky
Put a switch between the ISP device, and pfSense.
First, hook up ,the switch to WAN-pfSense, and see that the connection negotiates a connection using 'auto on both side (the switch will only have the auto mode).

Then make the connection between the switch and the ISP device.

Thank You, I'l try to doing this today. Really great idea, that I miss. (I little bit tied, may be need some sort of small vacation;)

[jimp]

For future reference you can find out more detail about the cause of these errors in most cases from sysctl. The amount of information varies by driver.

For example if you see errors on ix0, check sysctl dev.ix.0 and look at the counters, there are some which are obviously errors (marked with names that say error/err) and others that are less obvious like no_desc_avail. There will be counters under places like mac_stats as well as under various queues. Again, varying by driver.

All of these should total up to the number you see in the status page for errors, but it varies too much for the GUI to attempt displaying the detail.

[Sergei_Shablovsky]

@gertjan said in Errors In, but no Errors Out and No Collisions on both WAN:

@sergei_shablovsky
Put a switch between the ISP device, and pfSense.
First, hook up ,the switch to WAN-pfSense, and see that the connection negotiates a connection using 'auto on both side (the switch will only have the auto mode).

Then make the connection between the switch and the ISP device.

Thank You for all suggestions and help here!

So, the conclusion on testing: on a short (<30m) distance the NIC able to work on 1000Tx <full duplex> but on long distance (>80m) only 100TX <full duplex> possible + with a lot of “CRC In Errors”.

[Sergei_Shablovsky]

@jimp said in Errors In, but no Errors Out and No Collisions on both WAN:

For future reference you can find out more detail about the cause of these errors in most cases from sysctl. The amount of information varies by driver.

For example if you see errors on ix0, check sysctl dev.ix.0 and look at the counters, there are some which are obviously errors (marked with names that say error/err) and others that are less obvious like no_desc_avail. There will be counters under places like mac_stats as well as under various queues. Again, varying by driver.

Thanks You for suggestions and help here!

During testing various combinations NIC/cable/other appliance we see that “CRC In Errors” caused by impossibility of exact NIC to handle long (mean >80m) FTP Cat6e cable connection at 1000TX <full duplex> (1Gb/s).

All of these should total up to the number you see in the status page for errors, but it varies too much for the GUI to attempt displaying the detail.

Anyway the pfSense have a great GUI Dashboard.

Only one I need from pfSense Dashboard: dividing on groups (similar to Grafana with options to Expand/Collapse):
Section 1 - for frequently used info like WANs/LANs information, DNS info, VPNs info, netflow real-time graphics;
Section 2 - for more static, or rarely changing information (whole system info, installed packets, services status, RSS news, etc);

[Sergei_Shablovsky]

Issue was resolved by replacing FTP Cat6 connection to ISP's applience on fiber 4-mode cable: no any collisions, no any errors in sending/receiving packets and in addition delay was decreased from 7-8ms to 3-4ms.

[mer]

@sergei_shablovsky Amazing how often physical connectivity is the problem. Copper that works fine for 10M could have all kinds of problems at 10G. Fiber? Need to make sure everything matches.

Lesson to take away:
Check physical first :)