Does not connect to switch after switch reset

rikinn55 · Apr 8, 2023, 9:12 PM

I'm new to PfSense and don't know if this is a software or hardware issue. here is what happens. When the connected switch is rebooted, the link between the PfSense computer (Qotom Mini PC w/8 ports) and the switch (TP-Link TL-SG3210XHP-M2) will not reconnect. The ports just keep trying to sync. Both machines are new and so I'm in the beginnings of setup. The switch is a blank canvas virtually no setup except for a time-server and one entry to forward all traffic to anywhere it wants to go (0.0.0.0 0.0.0.0). It does not matter what port I plug the PfSense computer into I have the same problem. On the switch I can plug and unplug any other network device and things link up as expected.

Now on the PfSense machine. In bios all ports are on just your basic make everything work setup. PfSense is install the way it comes and have gone through the startup wizard. So nothing has been setup for rules or anything else. Keep in mind that if I reboot the computer after the switch is back up then everything connects and works just fine. The only issue is no reconnection if switch is rebooted. Or is the computer is up and running before the switch is powered up.

Ideas?

Thanks in advance

bingo600 · Apr 9, 2023, 6:20 AM

@rikinn55
There could be several causes here.

Are we talking about the pfSense WAN port ... Does it get the ip address via DHCP ?

My guess is one of these, could be the cause.

1:
The pfSense NIC & the switch has ethernet L1 problems , negotiating the "link".

2:
The pfSense interface is getting the IP via DHCP, and the switch/router is "slower" to come up (giving ethernet link) than the pfSense.

My money would be on 2:
I have had that 2: issue on pfSense WAN. Where the pfSense did not get a valid interface "link up" when the interface was ready. That caused a it to give up on getting a DHCP ip address, within the timeout period, and silently "gave up".

My issue was caused by a "Wan ISP router" that was slower to boot then the pfSense.
I would expect a "simple switch" to be ... much faster that a Qotom/pfSense, and give link when the pfSense interface is ready.

I had to switch to "Static IP" on the WAN port, to solve the issue. Then there's no DHCP timeout.
And the pfSense "Wan" interface "just connected", when the switch were ready.

If it's the same DHCP/timeout related, issue that i had.
A manual disconnect/connect of the pfSense interface cable would solve it (the switch must be up & ready, before doing this).
But that is not a feasible solution. Just a way of testing if it could be DHCP/Timeout related.

If you have a "Dumb switch ... aka fast booting switch" , try to insert it between the pfSense & the tp-link, on the problem interface.

A console connection to the Qotom during boot would be helpfull here.
And looking in the log files would clearly show the DHCP failure during boot.

/Bingo

rikinn55 · Apr 9, 2023, 6:47 AM

@bingo600
Thank you for your reply.The problem is connecting LAN1 on the Qotom/offense to port 1 on the tp link. offense is providing duck and vlan1 on tp link is static.

Generally when the create the link the port lights on the tp link go solid and the same on the a qotom/pfsense. Based on how the lights are acting the link is continually dropped. Is it possible the tp link is blocking the connection because I haven't setup a rule in the switch? I will look at what default settings if any are in the switch firewall. The switch has more features than I wanted but it offered what I wanted in number of ports running 2.5gb.

rikinn55 · Apr 9, 2023, 6:50 AM

@rikinn55
I really auto correct, it messes up the words I want used. Sorry I did catch all of them.

bingo600 · Apr 9, 2023, 7:02 AM

@rikinn55
If you have a ethernet L1 (link problem) , you should easily see that in the pfSense logs.
My guess is that it would do a lot of interface link errors.

Is the pfSense (Qotom) 2.5Gb capable ?

Does the "Disconect/Connect" pfSense interface cable solve the issue ?

I don't have TP-Link switches .... I have been avoiding them like the plague.
They might have improved, since their VLAN1 issues...

Do you have another switch you could try to replicate it with ?
Ie. an old 1Gb switch or the likes ....

Have you checkked for a switch firmware upgrade ??

stephenw10 · Apr 9, 2023, 11:56 PM

Yes, what are the NIC types in the firewall?

I have seen link negotiation issues in igc if the NIC is set to autoselect rather than default for some reason. That wouldn't be the case unless you had set it though.

rikinn55 · Apr 10, 2023, 4:05 PM

@bingo600
I did already update the switch firmware.
The Qotom is 2.5gb and is the reason for the TP Link purchase.
Replication: Here is what I did.
Disconnect from TP Link to the Ubiquity 1gb switch = same issue.
Reboot the Qotom get a link.
Disconnect from Ubiquity and attach to Netgear dumb switch = same issue.
Reboot the Qotom get a link.
Disconnect and plug into the TP Link = same issue.
Turn off the Qotom.
When the Qotom is not running the port show connections to switch
So I disconnect from TP Link port 1 and attached to port 6 lights linked up.
Disconnected from TP Link port 6 attached to Ubiquity ports light up.
Reattach to TP-Link and boot the Qotom
Note: Keep in mind the Qotom has 5 ports and only ports 1 (WAN) and ports 2 (LAN) are being monitored by PfSense. Ports 3, 4, and 5 are not even set in PfSense. With that in mind after the Qotom was booted 1:
Disconnected LAN and pluged into Qotom port 3 linked up with TP Link.
Disconnect port 3 and plugged back to LAN no link established.
Disconnected LAN plugged into Qotom port 4 linked up with TP Link
Disconnected port 4 plugged into LAN no link.
My conclusion is that the problem is not hardware. I believe the last test I did proved that it is something in the PfSense settings, but what.
How can I get a print out of the settings in PfSense so I can share them?

Rikinn

stephenw10 · Apr 10, 2023, 4:17 PM

So they are Intel igc(4) NICs?

Have you set a link speed as I described above?

rikinn55 · Apr 10, 2023, 5:46 PM

@stephenw10
I forgot to do that. I will do it and get back on what happens.

stephenw10 · Apr 10, 2023, 6:10 PM

I'm asking if you have already set anything because that could affect the link negotiation. By default most NICs should link OK.

Also please confirm what NICs the firewall is using because there are several 2.5G NIC types it could be.

rikinn55 · Apr 10, 2023, 6:14 PM

@stephenw10
All nic's are auto negotiate. None are set to static link speed. The Qotom has all Intel nics. I was very careful on that when I was looking for a unit. I had already experience problems when not an Intel nic.

stephenw10 · Apr 10, 2023, 6:34 PM

Ok, so they are all Intel igc(4)?

Are you running 2.6?

rikinn55 · Apr 10, 2023, 7:36 PM

@stephenw10
PfSense Scale
23.01-RELEASE (amd64)
built on Fri Feb 10 20:06:33 UTC 2023
FreeBSD 14.0-CURRENT

stephenw10 · Apr 10, 2023, 8:49 PM

What NICs are they? What is returned by: pciconf -lv?

rikinn55 · Apr 11, 2023, 2:46 AM

@stephenw10 said in Does not connect to switch after switch reset:

pciconf -lv

hostb0@pci0:0:0:0:	class=0x060000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31f0 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:0:2:0:	class=0x030000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x3185 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'GeminiLake [UHD Graphics 600]'
    class      = display
    subclass   = VGA
none0@pci0:0:15:0:	class=0x078000 rev=0x06 hdr=0x00 vendor=0x8086 device=0x319a subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor Trusted Execution Engine Interface'
    class      = simple comms
ahci0@pci0:0:18:0:	class=0x010601 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31e3 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor SATA Controller'
    class      = mass storage
    subclass   = SATA
pcib1@pci0:0:19:0:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:19:1:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d9 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:19:2:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31da subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:19:3:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31db subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:0:20:0:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d6 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:0:20:1:	class=0x060400 rev=0xf6 hdr=0x01 vendor=0x8086 device=0x31d7 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Gemini Lake PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
xhci0@pci0:0:21:0:	class=0x0c0330 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31a8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31e8 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
ichsmb0@pci0:0:31:1:	class=0x0c0500 rev=0x06 hdr=0x00 vendor=0x8086 device=0x31d4 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Celeron/Pentium Silver Processor Gaussian Mixture Model'
    class      = serial bus
    subclass   = SMBus
igc0@pci0:1:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc1@pci0:2:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc2@pci0:3:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc3@pci0:4:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc4@pci0:5:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet

stephenw10 · Apr 11, 2023, 2:48 AM

Ok, so 5 igb NICs. But you say it has 8 ports total? If it has some sprt of internal switch that could be a problem.

rikinn55 · Apr 11, 2023, 2:56 AM

@stephenw10
The Tp link has 8 and the qotom has 5

stephenw10 · Apr 11, 2023, 11:54 AM

@rikinn55 said in Does not connect to switch after switch reset:

Qotom Mini PC w/8 ports

OK, that was just a typo then?

So what's logged when it disconnects and then re-links?

What does ifconfig -vvvm igc0 show? Assuming igc0 is one of the ports that's flapping?

rikinn55 · Apr 11, 2023, 5:19 PM

@stephenw10
PfS_Gen_Log.txt
igc1 flags=8863UP,BROADCAST,RUNNING.txt

Hope these are the two items you want. If not please let me know where to find what your looking for.

Thanks

stephenw10 · Apr 11, 2023, 5:35 PM

Hmm, OK so all the igc ports behave the same?

Before the reboot in that log we can see igc flapping every 5s or so.

After rebooting we see it flap once during the boot process then it stays up for ~2mins. Does it remain linked correctly after that?