[SOLVED] Errors In, but no Errors Out and No Collisions on both WAN

stephenw10 · Nov 5, 2021, 12:29 PM

If the ISP device you're connected to is set at fixed 100Mb you would need to match that setting in the pfSense NIC. If it's a 1G NIC and you leave it as auto-select it will fail to negotiate a link speed and fall back to it's default which it sounds like is 100M half-duplex.
With that duplex mismatch you will certainly see errors/collisions.

Steve

Sergei_Shablovsky · Nov 5, 2021, 3:50 PM

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

How many errors? Out of how much total data?

I reboot the system, and make some measurements. Please look:
(data from pfSense GUI Status -> Interfaces)

Uptime for this data 5h 20min.
Network loading are quite small for this branch today. (Just nothing)

WAN_1
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
490641/359474 (541.09 MiB/33.24 MiB)
In/out packets (pass)
490641/359474 (541.09 MiB/33.24 MiB)
In/out packets (block)
4864/803 (663 KiB/144 KiB)
In/out errors
107/107
Collisions
56365

WAN_2
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
170677/150218 (101.52 MiB/13.74 MiB)
In/out packets (pass)
170677/150218 (101.52 MiB/13.74 MiB)
In/out packets (block)
4827/1650 (826 KiB/211 KiB)
In/out errors
14/14
Collisions
13684

stephenw10 · Nov 5, 2021, 4:04 PM

Yeah, you can't have that running at half-duplex. The number of collisions there are expected in that situation.

Sergei_Shablovsky · Nov 5, 2021, 9:57 PM

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

Yeah, you can't have that running at half-duplex. The number of collisions there are expected in that situation.

Stephen, thank You for patience! ;)

So, I doing some tests and here the result:

Both WANs (mean 1st port on each of NICs) set in pfSense In Interfaces -> WAN_X in
100baseTX <full-duplex>

The output in **Status -> Interfaces”

WAN_1
MTU
1350
Media
100baseTX <full-duplex>
In/out packets
266980/253695 (299.20 MiB/34.30 MiB)
In/out packets (pass)
266980/253695 (299.20 MiB/34.30 MiB)
In/out packets (block)
740/285 (73 KiB/46 KiB)
In/out errors
12123/0
Collisions
0

WAN_2
MTU
1350
Media
100baseTX <full-duplex>
In/out packets
15759/16785 (1.62 MiB/777 KiB)
In/out packets (pass)
15759/16785 (1.62 MiB/777 KiB)
In/out packets (block)
1301/0 (181 KiB/0 B)
In/out errors
281/0
Collisions
0

Both WANs (mean 1st port on each of NICs) set in pfSense In Interfaces -> WAN_X in
100baseTX**

The output in **Status -> Interfaces”

WAN_1
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
58191/43464 (55.42 MiB/7.11 MiB)
In/out packets (pass)
58191/43464 (55.42 MiB/7.11 MiB)
In/out packets (block)
208/0 (13 KiB/0 B)
In/out errors
2/2
Collisions
6407

WAN_2
MTU
1350
Media
100baseTX <half-duplex>
In/out packets
5461/5977 (726 KiB/320 KiB)
In/out packets (pass)
5461/5977 (726 KiB/320 KiB)
In/out packets (block)
177/0 (8 KiB/0 B)
In/out errors
0/0
Collisions
217

Need to add, that on any other interfaces any other NICs (which are identical and we buy at one time from one supplier) there are no In/Out Errors or Collisions

stephenw10 · Nov 5, 2021, 9:36 PM

The MTU there is very low.

What changed here? Were those ever linked at 1G?

If you set a fixed speed it must match the other end of the link.
If you set half duplex you will always get collisions even if the other end does match.

Steve

Sergei_Shablovsky · Nov 5, 2021, 10:37 PM

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

The MTU there is very low.

We need this value for uplink, just ignore it.
Anyway, I make the test for both 100TX and 100TX full duplex, with MTU 1350, 1500 and 1420, and looks like MTU no such involved in this issue...

What changed here? Were those ever linked at 1G?

I try each settings for interface from automatic to highest 1000TX full duplex on Interfaces - WAN_X webpage of pfSense GUI, the result You able to see above...

Shortly to say, only on 100TX (mean 100TX half-duplex), all pfSense updates are ok, no connection reset by server due 1,5-2h connect time, all working.
But I still have In Errors.

stephenw10 · Nov 6, 2021, 1:02 AM

With half duplex you mat well get errors, you will definitely see collisions.

Nothing is half-duplex these days not since hubs and 10base2! What is it connected to? It seems like something must be configured wrong.

Steve

Sergei_Shablovsky · Nov 6, 2021, 4:26 AM

@stephenw10 said in Errors In, but no Errors Out and No Collisions on both WAN:

With half duplex you mat well get errors, you will definitely see collisions.

Nothing is half-duplex these days not since hubs and 10base2!

Heh ;)

What is it connected to?

I start thinking “connected to a piece of sh****t” ;)
(Some Huawei applience)

It seems like something must be configured wrong.

Also make the same conclusion. But anyway before to start complaining the ISP, I need to be sure that all ok on my end of cable ;)
So, if You have some suggestions, I would be thankful to read it :)

Gertjan · Nov 6, 2021, 7:13 PM

@sergei_shablovsky
Put a switch between the ISP device, and pfSense.
First, hook up ,the switch to WAN-pfSense, and see that the connection negotiates a connection using 'auto on both side (the switch will only have the auto mode).

Then make the connection between the switch and the ISP device.

stephenw10 · Nov 6, 2021, 10:07 AM

Yup, that would be a good test.

Sergei_Shablovsky · Nov 6, 2021, 7:13 PM

@gertjan said in Errors In, but no Errors Out and No Collisions on both WAN:

@sergei_shablovsky
Put a switch between the ISP device, and pfSense.
First, hook up ,the switch to WAN-pfSense, and see that the connection negotiates a connection using 'auto on both side (the switch will only have the auto mode).

Then make the connection between the switch and the ISP device.

Thank You, I'l try to doing this today. Really great idea, that I miss. (I little bit tied, may be need some sort of small vacation;)

jimp · Nov 9, 2021, 3:02 PM

For future reference you can find out more detail about the cause of these errors in most cases from sysctl. The amount of information varies by driver.

For example if you see errors on ix0, check sysctl dev.ix.0 and look at the counters, there are some which are obviously errors (marked with names that say error/err) and others that are less obvious like no_desc_avail. There will be counters under places like mac_stats as well as under various queues. Again, varying by driver.

All of these should total up to the number you see in the status page for errors, but it varies too much for the GUI to attempt displaying the detail.

Sergei_Shablovsky · Nov 17, 2021, 7:36 AM

@gertjan said in Errors In, but no Errors Out and No Collisions on both WAN:

@sergei_shablovsky
Put a switch between the ISP device, and pfSense.
First, hook up ,the switch to WAN-pfSense, and see that the connection negotiates a connection using 'auto on both side (the switch will only have the auto mode).

Then make the connection between the switch and the ISP device.

Thank You for all suggestions and help here!

So, the conclusion on testing: on a short (<30m) distance the NIC able to work on 1000Tx <full duplex> but on long distance (>80m) only 100TX <full duplex> possible + with a lot of “CRC In Errors”.

Sergei_Shablovsky · Nov 17, 2021, 7:53 AM

@jimp said in Errors In, but no Errors Out and No Collisions on both WAN:

For future reference you can find out more detail about the cause of these errors in most cases from sysctl. The amount of information varies by driver.

For example if you see errors on ix0, check sysctl dev.ix.0 and look at the counters, there are some which are obviously errors (marked with names that say error/err) and others that are less obvious like no_desc_avail. There will be counters under places like mac_stats as well as under various queues. Again, varying by driver.

Thanks You for suggestions and help here!

During testing various combinations NIC/cable/other appliance we see that “CRC In Errors” caused by impossibility of exact NIC to handle long (mean >80m) FTP Cat6e cable connection at 1000TX <full duplex> (1Gb/s).

All of these should total up to the number you see in the status page for errors, but it varies too much for the GUI to attempt displaying the detail.

Anyway the pfSense have a great GUI Dashboard.

Only one I need from pfSense Dashboard: dividing on groups (similar to Grafana with options to Expand/Collapse):
Section 1 - for frequently used info like WANs/LANs information, DNS info, VPNs info, netflow real-time graphics;
Section 2 - for more static, or rarely changing information (whole system info, installed packets, services status, RSS news, etc);

Sergei_Shablovsky · Nov 17, 2021, 7:58 AM

Issue was resolved by replacing FTP Cat6 connection to ISP's applience on fiber 4-mode cable: no any collisions, no any errors in sending/receiving packets and in addition delay was decreased from 7-8ms to 3-4ms.

mer · Nov 17, 2021, 1:10 PM

@sergei_shablovsky Amazing how often physical connectivity is the problem. Copper that works fine for 10M could have all kinds of problems at 10G. Fiber? Need to make sure everything matches.

Lesson to take away:
Check physical first :)

Sergei_Shablovsky · Nov 17, 2021, 1:10 PM

@mer said in [SOLVED] Errors In, but no Errors Out and No Collisions on both WAN:

@sergei_shablovsky Amazing how often physical connectivity is the problem. Copper that works fine for 10M could have all kinds of problems at 10G. Fiber? Need to make sure everything matches.

In exactly this case both the misconfiguration / changes in ISP work and length of cable impact.

Lesson to take away:
Check physical first :)

Need to re-read our “Resolving uplink physical connection problems” checklist ;)

mer · Nov 17, 2021, 1:17 PM

@sergei_shablovsky Another thing with copper cables that I've run into: cheap ones "seem" to work, but will cause problems. Especially at higher speeds and longer distances. The distance is part that everyone overlooks. Oh that 110m cat-4 should be fine, it worked before.

Glad that it was simple to correct.

Sergei_Shablovsky · Nov 17, 2021, 4:09 PM

@mer said in [SOLVED] Errors In, but no Errors Out and No Collisions on both WAN:

Another thing with copper cables that I've run into: cheap ones "seem" to work, but will cause problems. Especially at higher speeds and longer distances. The distance is part that everyone overlooks.

Thank You, I forgot to mention this.

Of course, our rules would be Using for copper link as much quality as available from local distributor.
Because You make cabling one time in 2-3-5-8 years, up to the next applience upgrade ;)

mer · Nov 17, 2021, 4:11 PM

@sergei_shablovsky said in [SOLVED] Errors In, but no Errors Out and No Collisions on both WAN:

Of course, our rules would be Using for copper link as much quality as available from local distributor.
Because You make cabling one time in 2-3-5-8 years, up to the next applience upgrade ;)

That is a very good rule. Even for your home network.