Does not connect to switch after switch reset

rikinn55

@stephenw10
PfSense Scale
23.01-RELEASE (amd64)
built on Fri Feb 10 20:06:33 UTC 2023
FreeBSD 14.0-CURRENT

stephenw10

What NICs are they? What is returned by: pciconf -lv?

rikinn55

@stephenw10 said in Does not connect to switch after switch reset:

pciconf -lv

hostb0@pci0:0:0:0:	class=0x060000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31f0 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:	class=0x030000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x3185 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'GeminiLake [UHD Graphics 600]'
    class      = display
    subclass   = VGA
none0@pci0:0:15:0:	class=0x078000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x319a subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor Trusted Execution Engine Interface'
    class      = simple comms
ahci0@pci0:0:18:0:	class=0x010601 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31e3 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor SATA Controller'
    class      = mass storage
    subclass   = SATA
pcib1@pci0:0:19:0:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:19:1:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d9 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:19:2:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31da subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:19:3:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31db subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:0:20:0:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d6 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:0:20:1:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d7 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
xhci0@pci0:0:21:0:	class=0x0c0330 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31a8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31e8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
ichsmb0@pci0:0:31:1:	class=0x0c0500 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31d4 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor Gaussian Mixture Model'
    class      = serial bus
    subclass   = SMBus
igc0@pci0:1:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc1@pci0:2:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc2@pci0:3:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc3@pci0:4:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc4@pci0:5:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet

stephenw10

Ok, so 5 igb NICs. But you say it has 8 ports total? If it has some sprt of internal switch that could be a problem.

rikinn55

@stephenw10
The Tp link has 8 and the qotom has 5

stephenw10

@rikinn55 said in Does not connect to switch after switch reset:

Qotom Mini PC w/8 ports

OK, that was just a typo then?

So what's logged when it disconnects and then re-links?

What does ifconfig -vvvm igc0 show? Assuming igc0 is one of the ports that's flapping?

rikinn55

@stephenw10
PfS_Gen_Log.txt
igc1 flags=8863UP,BROADCAST,RUNNING.txt

Hope these are the two items you want. If not please let me know where to find what your looking for.

Thanks

stephenw10

Hmm, OK so all the igc ports behave the same?

Before the reboot in that log we can see igc flapping every 5s or so.

After rebooting we see it flap once during the boot process then it stays up for ~2mins. Does it remain linked correctly after that?

rikinn55

@stephenw10
Yes it stays up every time. Seems a reboot of the Qotom is what is required. I tried reboot the TP Link but that failed getting it to relink.

rikinn55

@stephenw10
Didn't answer the first question. Keep in mind PfS isn't doing anything with ports 2-4. So if I pulg and unplug from them I always get a link right away. I have not set PfS to use any of those ports as LAN yet.

stephenw10

Ok, well first try re-assigning LAN as a different NIC and make sure that behaves the same.

The only time I've anything like this was when setting igc to 'autoselect' rather than 'default' in the config after having linked at 1G.

I note I do have hardware checksum off-loading disabled on test devices here. I wouldn't expect that to make any difference to linking but wort trying.

You can check the sysctls for igc1 like:

23.01-RELEASE][root@4100.stevew.lan]/root: sysctl hw.igc
hw.igc.max_interrupt_rate: 8000
hw.igc.eee_setting: 1
hw.igc.rx_process_limit: 100
hw.igc.sbp: 1
hw.igc.smart_pwr_down: 0
hw.igc.rx_abs_int_delay: 66
hw.igc.tx_abs_int_delay: 66
hw.igc.rx_int_delay: 0
hw.igc.tx_int_delay: 66
hw.igc.disable_crc_stripping: 0
[23.01-RELEASE][root@4100.stevew.lan]/root: sysctl dev.igc.0
dev.igc.0.wake: 0
dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 4030
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 0
dev.igc.0.mac_stats.tx_frames_512_1023: 0
dev.igc.0.mac_stats.tx_frames_256_511: 3163
dev.igc.0.mac_stats.tx_frames_128_255: 0
dev.igc.0.mac_stats.tx_frames_65_127: 0
dev.igc.0.mac_stats.tx_frames_64: 22
dev.igc.0.mac_stats.mcast_pkts_txd: 22
dev.igc.0.mac_stats.bcast_pkts_txd: 3163
dev.igc.0.mac_stats.good_pkts_txd: 3185
dev.igc.0.mac_stats.total_pkts_txd: 3185
dev.igc.0.mac_stats.good_octets_txd: 1108458
dev.igc.0.mac_stats.good_octets_recvd: 0
dev.igc.0.mac_stats.rx_frames_1024_1522: 0
dev.igc.0.mac_stats.rx_frames_512_1023: 0
dev.igc.0.mac_stats.rx_frames_256_511: 0
dev.igc.0.mac_stats.rx_frames_128_255: 0
dev.igc.0.mac_stats.rx_frames_65_127: 0
dev.igc.0.mac_stats.rx_frames_64: 0
dev.igc.0.mac_stats.mcast_pkts_recvd: 0
dev.igc.0.mac_stats.bcast_pkts_recvd: 0
dev.igc.0.mac_stats.good_pkts_recvd: 0
dev.igc.0.mac_stats.total_pkts_recvd: 0
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.missed_packets: 0
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 128
dev.igc.0.queue_rx_1.rxd_head: 0
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 128
dev.igc.0.queue_rx_0.rxd_head: 0
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 0
dev.igc.0.queue_tx_1.txd_head: 0
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 222
dev.igc.0.queue_tx_0.txd_head: 222
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335966
dev.igc.0.device_control: 1478231617
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 66
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.itr: 488
dev.igc.0.tx_abs_int_delay: 66
dev.igc.0.rx_abs_int_delay: 66
dev.igc.0.tx_int_delay: 66
dev.igc.0.rx_int_delay: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
	CTRL	 581c0641
	STATUS	 40780683
	CTRL_EXIT	 10000040

Interrupt Registers
	ICR	 00000000

RX Registers
	RCTL	 0440801e
	RDLEN	 00004000
	RDH	 00000000
	RDT	 00000080
	RXDCTL	 02040808
	RDBAL	 02392000
	RDBAH	 00000000

TX Registers
	TCTL	 a503f0fa
	TDBAL	 02376000
	TDBAH	 00000000
	TDLEN	 00004000
	TDH	 000000de
	TDT	 000000de
	TXDCTL	 0201011f
	TDFH	 00000000
	TDFT	 00000000
	TDFHS	 00000000
	TDFPC	 00000000


dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 128
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 0
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 128
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 128
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 0
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 128
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 0
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 0
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 0
dev.igc.0.iflib.txq1.txq_processed: 0
dev.igc.0.iflib.txq1.txq_in_use: 0
dev.igc.0.iflib.txq1.txq_cidx_processed: 0
dev.igc.0.iflib.txq1.txq_cidx: 0
dev.igc.0.iflib.txq1.txq_pidx: 0
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 3185
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 3185
dev.igc.0.iflib.txq0.ring_state: pidx_head: 1137 pidx_tail: 1137 cidx: 1137 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 6322
dev.igc.0.iflib.txq0.txq_processed: 6362
dev.igc.0.iflib.txq0.txq_in_use: 44
dev.igc.0.iflib.txq0.txq_cidx_processed: 218
dev.igc.0.iflib.txq0.txq_cidx: 178
dev.igc.0.iflib.txq0.txq_pidx: 222
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 7
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%parent: pci4
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:4:0:0 handle=\_SB_.PCI0.PEX3.PXSX
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I225-V

rikinn55

@stephenw10
Here are the sysctls:
sysctl hw.igc.txt

Now I did a test by changing PfS to use igc.2 in place of igc.1 attached are resulting files.
After reboot.txt
After unplug igc.2 and plug back in.txt

As you will notice I experienced the same issue.

Thanks for all your help.

stephenw10

Hmm, looks near identical to any igc port I have here.

It does that when connected to anything? A host directly for example?

rikinn55

@stephenw10
I will check that in the morning.

rikinn55

@stephenw10
Ran the test with my laptop (intel nic) same result. See attached
Log 12APR23-a.txt

bingo600

I'm no FreeBSD/pfSense boot-log guru.
But wasn't it an 8-port Qotom ?

I only see it detect igc0..igc4 ... where are the last 3 net IF's ?

Could it be bad hw ?

/Bingo

rikinn55

@bingo600
I think that's what I said early on but it is a 5 port.

stephenw10

Can you get the log covering it start to fail after it has been stable?

So showing what is logged when you first disconnect and reconnect the cable.

rikinn55

@stephenw10
The file show a fresh boot through disconnect and reconnect then another fresh boot to bring everything back up.
Log 12APR23-b.txt

stephenw10

Hmm, the only thing of any significance I see there is that when the interface comes up after boot it does so before check_reload_status runs. Later the order is reversed and the link bounces. However that's no different to the igc NICs in the 6100 or 4100.

Are you able to try a 2.7 snapshot there? It has slightly more recent code. I'm not aware of any updates that would affect igc but there may be something.

I can only imagine it has a different PHY and that is somehow seeing the link change.