WAN drops from 1gbit to 100mbit

ledj

Hi all

Running latest stable release:
2.3.2-RELEASE-p1 (amd64)
built on Tue Sep 27 12:13:07 CDT 2016
FreeBSD 10.3-RELEASE-p9

On this hardware:
https://www.supermicro.nl/products/system/1U/5018/SYS-5018D-FN4T.cfm

Yesterday I discovered that the WAN speed was only 100mbit, so I went to "Interfaces / WAN" and set "Speed and Duplex" to "autoselect" (was "Default"), that made the speed go to 1000mbit for about 3 hours.

Later in the evening this happened:
Dec 20 23:04:18 firewall check_reload_status: Linkup starting igb0
Dec 20 23:04:18 firewall kernel: igb0: link state changed to DOWN
Dec 20 23:04:19 firewall php-fpm[59816]: /rc.linkup: Hotplug event detected for WAN(wan) static IP (#.#.#.8 )
Dec 20 23:04:19 firewall check_reload_status: Reloading filter
Dec 20 23:04:20 firewall xinetd[15263]: Starting reconfiguration
Dec 20 23:04:20 firewall xinetd[15263]: Swapping defaults
Dec 20 23:04:20 firewall xinetd[15263]: readjusting service 6969-udp
Dec 20 23:04:20 firewall xinetd[15263]: Reconfigured: new=0 old=1 dropped=0 (services)
Dec 20 23:04:22 firewall check_reload_status: Linkup starting igb0
Dec 20 23:04:22 firewall kernel: igb0: link state changed to UP
Dec 20 23:04:23 firewall php-fpm[11610]: /rc.linkup: Hotplug event detected for WAN(wan) static IP (#.#.#.8 )
Dec 20 23:04:23 firewall check_reload_status: rc.newwanip starting igb0
Dec 20 23:04:23 firewall check_reload_status: Reloading filter
Dec 20 23:04:24 firewall php-fpm[11610]: /rc.newwanip: rc.newwanip: Info: starting on igb0.
Dec 20 23:04:24 firewall php-fpm[11610]: /rc.newwanip: rc.newwanip: on (IP address: #.#.#.8) (interface: WAN[wan]) (real interface: igb0).
Dec 20 23:04:24 firewall check_reload_status: Reloading filter

And back to 100mbit. This degradation is causing some troubles.

So I've checked cables and switch to uplink, but can't identify the issue. The switch that the WAN is connected to also falls back to 100mbit on the switchport (autonegotiation). Switch is also connected to another pfsense server that doesn't have issues and running 1gbit (although the pfsense version is in the 2.2 branch).

Flow control is enabled. I have the option to try the ix0 port on the server (a 10gbit RJ45-port).

Anybody aware of something that can cause this behavior or tweaks to avoid this ?

Guest

Hello,

in often at supermicro servers the wan port is also linked to the IPMI port as fall back port!
This can really be changed in the IPMI or BIOS software, if this might be causing your problem!
Please check this settings against the manual and then try out to disable that settings that the IPMI
port will not longer the fall back port shared together with the wan port.

ledj

The IPMI port WAS empty when the server booted first time, so that might be it!

LAN Interface was "Failover", I've set it to "Dedicate". Although it doesn't help on-the-fly/instant, a system reboot maybe change the link state. Also the only options of "Link" is 100M, (see screenshot) even though speed shows 1G.

ipmisc.png_thumb

ipmisc2.png_thumb

ledj

A reboot didn't help. But I suspect a big power cable ("datacenter big") running very close to the network cable could cause degradation (or at least I hope!), will try to move the network cable away from the power cable and post an update later.

ledj

Changing cable path fixed the issue!

Harvy66

Same cable, just routed the path differently? Could the cable be bad or was near electrical noise?

Nullity

@Harvy66:

Same cable, just routed the path differently? Could the cable be bad or was near electrical noise?

I'd like to know more as well…

ledj

2x32A power cables (top of rack) were going very near the cable, which was unshielded btw. Changed to shielded cable, and also different path to switch, to make sure. Since then it's been on 1000 mbit not falling back to 100 mbit. So not a fault of pfsense! :-)