pfSense not enabling port

Gblenn

@georgelza said in pfSense not enabling port:

As per suggestion, see below. Looks good, vmbr30 which sows up

root@pmox1:~# ip a
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN group default qlen 1000
link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff
10: vmbr30: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
inet 172.16.30.11/24 scope global vmbr30
valid_lft forever preferred_lft forever
inet6 fe80::aab8:e0ff:fe05:f091/64 scope link
valid_lft forever preferred_lft forever

Hmm, perhaps it isn't working after all. Just noticed that you have the NO-CARRIER notification = No cable connected?! As well as it is not enabled as in "noop state DOWN group".

Further on the vmbr30 there is no LOWER_UP as in the physical connection at the link layer is not there?? Compare vmbr0 to vmbr30:

vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP>
vmbr30: <BROADCAST,MULTICAST,UP>

What does it look like in the Proxmox UI, for the host and the listings under network?

georgelza

@Gblenn see above.

The UI implies it's up...

guess i need to say i wait... this is rightly not a netgate problem... well until i install the DAC cable from the Topton running pfSense into y core switch.

This atm is more unifi/Topton comm...

netgate/pfsense related though, at the moment it's not allowing me to specify a default gw or simply a gw to use for the 172.16.30.0 network.

i can ping from the pmox my 172.16.10.1 gw, but that's going via the 2.5GbE copper link, to switch and onwards to pfSense to the igc0 port

G

Gblenn

@georgelza said in pfSense not enabling port:

@Gblenn see above.

The UI implies it's up...
G

No, the UI only sais that it is administratively activated.

Here's what it looks like for me if I disable the switchport that my 10G link is connected to, same as you have: NO-CARRIER and no LOWER_UP.

3: enp10s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000

And this is what the UI is showing me...

georgelza

@Gblenn
ok, interesting... figured that implied it was working...
even though as you said ip a actually said otherwise.

so it's starting to look like the Pmox host is also not linked to the Unifi pro max...

wondering if this is caused by the switch... think i might need to "engage" some patience and wait for that 2nd SFP+ to arrive and then the DAC cable and then see from there.
Will first try them on the pmox host, as it's more compatible with anything and confirm they work, then if all good there then move them to the pfSense host.

G

Gblenn

@georgelza said in pfSense not enabling port:

Hmm, do you always assign IP from Proxmox? I'm not sure what Proxmox will do in this case... as I would imagine it is Proxmox handling the ICMP request within it's virtualization environment.

If you had a VM that you assigned vmbr30 to (leaving the ipv4 part empty), it would be assigned an IP from pfsense instead. You don't need to put it in any VLAN, as that is only complicating things when testing... But if you want to, it's just a matter of entering the VLAN tag in the field for the VM's interface instead.

But still, the fact that it shows the link is not UP at the interface level, makes me wonder...

georgelza

@Gblenn I figured i'd give the host a ip on the network the card lives.
i will then give the guest vm's their own ip's on that network also.

can easily remove that 30.11

G

stephenw10

The status of the bridge device is not really important compared to that of the actual NIC. The bridge could appear up even if the NIC is not.

I would check the NIC stats and see if you see any incoming packets on it. ip -s link show enp4s0f0

Gblenn

@georgelza said in pfSense not enabling port:

@Gblenn I figured i'd give the host a ip on the network the card lives.
i will then give the guest vm's their own ip's on that network also.

can easily remove that 30.11

G

Leave all that to pfsense instead, that's where you want to control all those things, including all your static IP's. If you have set up the Unifi SFP+ port as VLAN 30 Untagged, then anything on the Proxmox side will of course get an IP from that range.

georgelza

@stephenw10

root@pmox1:~# ip -s link show enp4s0f0
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN mode DEFAULT group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
        236040    3934    282       0       0    3934 
    TX:  bytes packets errors dropped carrier collsns           
      11447434   67461      0       0       0       0 
root@pmox1:~#

Gblenn

@stephenw10 It's already in the info pasted above, showing NO-CARRIER and DOWN so it doesn't seem to be working unfortunately...

georgelza

@Gblenn

I normally do dhcp reserve assignment on pfSense for all devices... aka control/manage i from that side.

G

stephenw10

Ah, yes. try: ethtool enp4s0f0

georgelza

@stephenw10 said in pfSense not enabling port:

ethtool enp4s0f0

root@pmox1:~#  ethtool enp4s0f0
Settings for enp4s0f0:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
root@pmox1:~#

Gblenn

@georgelza Yes, so no need to set the IP, it's one of the things that made me think it was working when I saw it...

So when you ping from Proxmox, I guess in this case you were pinging inside the virtual switch of Proxmox and got that ICMP response. Meaning it never left Promxox...

And when you then ping from the PC, you can see that it times out. The response from 172.16.30.1 comes from pfsense obviously.

georgelza

and from the logs

Oct 31 16:46:17 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
Oct 31 16:46:17 pmox1 kernel: vmbr30: port 1(enp4s0f0) entered disabled state
Oct 31 16:46:17 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Oct 31 16:46:17 pmox1 kernel: vmbr30: port 1(enp4s0f0) entered blocking state
Oct 31 16:46:17 pmox1 kernel: vmbr30: port 1(enp4s0f0) entered forwarding state
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Down
Oct 31 16:46:18 pmox1 kernel: ixgbe 0000:04:00.0 enp4s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Gblenn

@georgelza So it looks like there is something happening at least. But it states auto-negotiation : no.

So what if you set the Unifi side to 10G and not Autonegotiate?

georgelza

have to pop out for hour, daddy duties. bbl.

G

georgelza

@Gblenn let me try.
it was set to 10 GBS FDX.

switched it to Auto, still same.

G

stephenw10

It has to be auto at both ends to use that. Otherwise it should link at 10G fixed on both ends.

Probably need to query the module to make sure it at least sees signal on the fiber. I don't have anything to test that with but maybe: ethtool --show-module enp4s0f0

stephenw10

Just to confirm this is Proxmox installed on the same Topton hardware that pfSense couldn't get link on?