pfSense not enabling port

Gblenn · Oct 31, 2024, 2:50 PM

@georgelza So it looks like there is something happening at least. But it states auto-negotiation : no.

So what if you set the Unifi side to 10G and not Autonegotiate?

georgelza · Oct 31, 2024, 2:50 PM

have to pop out for hour, daddy duties. bbl.

G

georgelza · Oct 31, 2024, 2:54 PM

@Gblenn let me try.
it was set to 10 GBS FDX.

switched it to Auto, still same.

G

stephenw10 · Oct 31, 2024, 3:12 PM

It has to be auto at both ends to use that. Otherwise it should link at 10G fixed on both ends.

Probably need to query the module to make sure it at least sees signal on the fiber. I don't have anything to test that with but maybe: ethtool --show-module enp4s0f0

stephenw10 · Oct 31, 2024, 3:13 PM

Just to confirm this is Proxmox installed on the same Topton hardware that pfSense couldn't get link on?

Gblenn · Oct 31, 2024, 3:37 PM

@stephenw10 said in pfSense not enabling port:

Just to confirm this is Proxmox installed on the same Topton hardware that pfSense couldn't get link on?

No it's a separate machine, thinking it should work in Proxmox at least...

It has to be auto at both ends to use that. Otherwise it should link at 10G fixed on both ends.

Probably need to query the module to make sure it at least sees signal on the fiber. I don't have anything to test that >with but maybe: ethtool --show-module enp4s0f0

It did list this at the end as per the post further up:

    Link detected: yes

And Proxmox log was showing interface going up and down...

Oct 31 16:46:17 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
Oct 31 16:46:17 pmox1 kernel: vmbr30: port 1(enp4s0f0) entered disabled state
Oct 31 16:46:17 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Oct 31 16:46:17 pmox1 kernel: vmbr30: port 1(enp4s0f0) entered blocking state
Oct 31 16:46:17 pmox1 kernel: vmbr30: port 1(enp4s0f0) entered forwarding state
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Perhaps time to try a different module, or set of modules / fiber...

georgelza · Oct 31, 2024, 3:41 PM

@Gblenn

I've already swopped the fiber out.

And tried the pfSense which was 2 modules there... and that fiber. then ran into issues and thought if we think this is pfSense not liking it lets me jst confirm things work, in a more compatible environment, so installed transceivers into my 2nd Topton that runs pmox. patched that into my Unifi Pro Max port 26 which SFP+ (SFP1), port 25 is (SFP0).

and ye. we not getting much further.

My unifi switch is reporting the correct SFP+ transceivers so can't image there being a problem with them, and well, they brand brand new.

G

stephenw10 · Oct 31, 2024, 3:43 PM

Ok but it's the same model Topton device? Same NIC?

Yeah, seeing the NIC logged as changing link state like that sure looks like it should be working....

Gblenn · Oct 31, 2024, 4:25 PM

@georgelza Just realized one more thing that you might try in Proxmox...

What do you see from this command?
dmesg | grep ixgbe

georgelza · Oct 31, 2024, 5:04 PM

@Gblenn said in pfSense not enabling port:

dmesg | grep ixgbe

[175483.486743] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175484.110776] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175484.214719] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175485.462768] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175485.670734] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175485.879762] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175485.982711] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175486.398837] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175486.502809] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175486.606757] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175486.710853] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175487.230749] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175487.438681] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175489.830832] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175489.934714] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175498.150577] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175498.255548] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175502.414503] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175502.518579] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down

georgelza · Oct 31, 2024, 5:07 PM

@stephenw10

pretty much yes, cpu and memory and nvme size differs, thats all.

G

stephenw10 · Oct 31, 2024, 5:12 PM

Link flapping every few seconds like that sure looks like a low level link negotiation issue.

georgelza · Oct 31, 2024, 5:22 PM

@stephenw10

so thesis the setting on the Unifi switch atm.

login-to-view

stephenw10 · Oct 31, 2024, 5:24 PM

I would try setting both ends to 10G FDX fixed since it seems to be struggling to negotiate.

georgelza · Oct 31, 2024, 5:27 PM

@stephenw10

well first noticed is the native VLAN was wrong... fixed that bit did not change anything. ok so setting to 10GbE FDX on switch is easy, how do I accomplish the same on proxmox ?

G

Gblenn · Oct 31, 2024, 5:35 PM

@georgelza I thought it was set to manual already, otherwise something like

ethtool -s enp4s0f0 speed 10000 duplex full autoneg off

stephenw10 · Oct 31, 2024, 5:42 PM

Yup at least as test just set it manually at the CLI. That won't survive a reboot of course.

georgelza · Oct 31, 2024, 5:45 PM

@stephenw10
well it seems I can now ping 172.16.30.1, from both the pmox shell, I also created a ubuntu guest vm, i can from inside here ping the 30.1 also

so we def making progress.

i've down a ifconfig <interface> down and a back up, not picking up a ip yet.

G

Gblenn · Oct 31, 2024, 5:45 PM

@georgelza said in pfSense not enabling port:

@Gblenn said in pfSense not enabling port:

dmesg | grep ixgbe

[175483.486743] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175484.110776] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175484.214719] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175485.462768] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175485.670734] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175485.879762] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175485.982711] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175486.398837] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175486.502809] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175486.606757] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175486.710853] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175487.230749] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175487.438681] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175489.830832] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175489.934714] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175498.150577] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175498.255548] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
[175502.414503] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[175502.518579] ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down

It would be interesting to see what it sais before it start flapping like that... Of there is anything that could provide a clue whether there is some module inocmpatibility for example. Some cards check the module to see if it is in the list of "supported devices", which it will announce here if that is the case...

I wonder if you remove the module, issue dmesg | grep ixgbe again.
Then plug it in and immediately run dmesg | grep ixgbe

Gblenn · Oct 31, 2024, 5:49 PM

@georgelza said in pfSense not enabling port:

@stephenw10
well it seems I can now ping 172.16.30.1, from both the pmox shell, I also created a ubuntu guest vm, i can from inside here ping the 30.1 also

so we def making progress.

i've down a ifconfig <interface> down and a back up, not picking up a ip yet.

G

Proxmox should be able to ping 172.16.30.1, since that is allowed according to your rules, and it is pfsense responding on that IP. You are likely pinging from 172.16.10 which is the Proxmox host, and not the ix0 interface you are trying to fix.
From the VM however, if you are able to ping, you should be good to go.

What does dmesg | grep ixgbe on Proxmox host show you now?