Does not connect to switch after switch reset

rikinn55 · Apr 11, 2023, 7:15 PM

@stephenw10
Didn't answer the first question. Keep in mind PfS isn't doing anything with ports 2-4. So if I pulg and unplug from them I always get a link right away. I have not set PfS to use any of those ports as LAN yet.

stephenw10 · Apr 11, 2023, 7:53 PM

Ok, well first try re-assigning LAN as a different NIC and make sure that behaves the same.

The only time I've anything like this was when setting igc to 'autoselect' rather than 'default' in the config after having linked at 1G.

I note I do have hardware checksum off-loading disabled on test devices here. I wouldn't expect that to make any difference to linking but wort trying.

You can check the sysctls for igc1 like:

23.01-RELEASE][root@4100.stevew.lan]/root: sysctl hw.igc
hw.igc.max_interrupt_rate: 8000
hw.igc.eee_setting: 1
hw.igc.rx_process_limit: 100
hw.igc.sbp: 1
hw.igc.smart_pwr_down: 0
hw.igc.rx_abs_int_delay: 66
hw.igc.tx_abs_int_delay: 66
hw.igc.rx_int_delay: 0
hw.igc.tx_int_delay: 66
hw.igc.disable_crc_stripping: 0
[23.01-RELEASE][root@4100.stevew.lan]/root: sysctl dev.igc.0
dev.igc.0.wake: 0
dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 4030
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 0
dev.igc.0.mac_stats.tx_frames_512_1023: 0
dev.igc.0.mac_stats.tx_frames_256_511: 3163
dev.igc.0.mac_stats.tx_frames_128_255: 0
dev.igc.0.mac_stats.tx_frames_65_127: 0
dev.igc.0.mac_stats.tx_frames_64: 22
dev.igc.0.mac_stats.mcast_pkts_txd: 22
dev.igc.0.mac_stats.bcast_pkts_txd: 3163
dev.igc.0.mac_stats.good_pkts_txd: 3185
dev.igc.0.mac_stats.total_pkts_txd: 3185
dev.igc.0.mac_stats.good_octets_txd: 1108458
dev.igc.0.mac_stats.good_octets_recvd: 0
dev.igc.0.mac_stats.rx_frames_1024_1522: 0
dev.igc.0.mac_stats.rx_frames_512_1023: 0
dev.igc.0.mac_stats.rx_frames_256_511: 0
dev.igc.0.mac_stats.rx_frames_128_255: 0
dev.igc.0.mac_stats.rx_frames_65_127: 0
dev.igc.0.mac_stats.rx_frames_64: 0
dev.igc.0.mac_stats.mcast_pkts_recvd: 0
dev.igc.0.mac_stats.bcast_pkts_recvd: 0
dev.igc.0.mac_stats.good_pkts_recvd: 0
dev.igc.0.mac_stats.total_pkts_recvd: 0
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.missed_packets: 0
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 128
dev.igc.0.queue_rx_1.rxd_head: 0
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 128
dev.igc.0.queue_rx_0.rxd_head: 0
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 0
dev.igc.0.queue_tx_1.txd_head: 0
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 222
dev.igc.0.queue_tx_0.txd_head: 222
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335966
dev.igc.0.device_control: 1478231617
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 66
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.itr: 488
dev.igc.0.tx_abs_int_delay: 66
dev.igc.0.rx_abs_int_delay: 66
dev.igc.0.tx_int_delay: 66
dev.igc.0.rx_int_delay: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
	CTRL	 581c0641
	STATUS	 40780683
	CTRL_EXIT	 10000040

Interrupt Registers
	ICR	 00000000

RX Registers
	RCTL	 0440801e
	RDLEN	 00004000
	RDH	 00000000
	RDT	 00000080
	RXDCTL	 02040808
	RDBAL	 02392000
	RDBAH	 00000000

TX Registers
	TCTL	 a503f0fa
	TDBAL	 02376000
	TDBAH	 00000000
	TDLEN	 00004000
	TDH	 000000de
	TDT	 000000de
	TXDCTL	 0201011f
	TDFH	 00000000
	TDFT	 00000000
	TDFHS	 00000000
	TDFPC	 00000000


dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 128
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 0
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 128
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 128
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 0
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 128
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 0
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 0
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 0
dev.igc.0.iflib.txq1.txq_processed: 0
dev.igc.0.iflib.txq1.txq_in_use: 0
dev.igc.0.iflib.txq1.txq_cidx_processed: 0
dev.igc.0.iflib.txq1.txq_cidx: 0
dev.igc.0.iflib.txq1.txq_pidx: 0
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 3185
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 3185
dev.igc.0.iflib.txq0.ring_state: pidx_head: 1137 pidx_tail: 1137 cidx: 1137 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 6322
dev.igc.0.iflib.txq0.txq_processed: 6362
dev.igc.0.iflib.txq0.txq_in_use: 44
dev.igc.0.iflib.txq0.txq_cidx_processed: 218
dev.igc.0.iflib.txq0.txq_cidx: 178
dev.igc.0.iflib.txq0.txq_pidx: 222
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 7
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%parent: pci4
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:4:0:0 handle=\_SB_.PCI0.PEX3.PXSX
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I225-V

rikinn55 · Apr 11, 2023, 10:20 PM

@stephenw10
Here are the sysctls:
sysctl hw.igc.txt

Now I did a test by changing PfS to use igc.2 in place of igc.1 attached are resulting files.
After reboot.txt
After unplug igc.2 and plug back in.txt

As you will notice I experienced the same issue.

Thanks for all your help.

stephenw10 · Apr 12, 2023, 4:28 AM

Hmm, looks near identical to any igc port I have here.

It does that when connected to anything? A host directly for example?

rikinn55 · Apr 12, 2023, 4:28 AM

@stephenw10
I will check that in the morning.

rikinn55 · Apr 12, 2023, 6:19 PM

@stephenw10
Ran the test with my laptop (intel nic) same result. See attached
Log 12APR23-a.txt

bingo600 · Apr 12, 2023, 6:42 PM

I'm no FreeBSD/pfSense boot-log guru.
But wasn't it an 8-port Qotom ?

I only see it detect igc0..igc4 ... where are the last 3 net IF's ?

Could it be bad hw ?

/Bingo

rikinn55 · Apr 12, 2023, 8:53 PM

@bingo600
I think that's what I said early on but it is a 5 port.

stephenw10 · Apr 12, 2023, 9:22 PM

Can you get the log covering it start to fail after it has been stable?

So showing what is logged when you first disconnect and reconnect the cable.

rikinn55 · Apr 12, 2023, 9:48 PM

@stephenw10
The file show a fresh boot through disconnect and reconnect then another fresh boot to bring everything back up.
Log 12APR23-b.txt

stephenw10 · Apr 17, 2023, 4:18 PM

Hmm, the only thing of any significance I see there is that when the interface comes up after boot it does so before check_reload_status runs. Later the order is reversed and the link bounces. However that's no different to the igc NICs in the 6100 or 4100.

Are you able to try a 2.7 snapshot there? It has slightly more recent code. I'm not aware of any updates that would affect igc but there may be something.

I can only imagine it has a different PHY and that is somehow seeing the link change.

rikinn55 · Apr 17, 2023, 4:18 PM

@stephenw10
I haven't given up. I got really busy and haven't had time to get back to fixing the problem. What I have decided to do is to format the drive and start over completely fresh. I didn't have anything setup that was beyond just getting on the internet so I have nothing to lose. If I remember right the problem started when I did a reset of the system and had it load factory defaults. But those defaults didn't include a few firewall settings which I entered myself. I can only imagine when I put in the firewall settings I messed something up. I believe we have more than proven the hardware is not the problem.

I hope I can get that done this week. I'm still swamped but just maybe I can get an hour and get this done.

rikinn55 · Apr 28, 2023, 3:47 PM

@stephenw10
Sorry it took so long to get back to this. I did a complete reinstall of pfsense and now everything works as it should. No more port not reconnecting after a physical disconnect and reconnect. So, as I believed previously there was something not right in the settings when I tried to have pfsense go back to factory settings which wiped out all the firewall rules. My attempt to recreate those rules I believe is what broke things.

Thank you so much stephenw10 for your help. I learned alot working with you.

stephenw10 · Apr 28, 2023, 5:33 PM

Ah, good result.