pfSense not enabling port

georgelza · Oct 30, 2024, 1:24 PM

Need to figure out how to get this done... as there is allot of common bits here.
and sharing...

I got a 2nd topton with Proxmox on it... patched that into the Unifi Pro Max, SFP2.
configure Proxmox to use the fiber port ix0 as a 2nd bridge. the port is alive, i can ping the port from local. but i can't ping out, which tells me something is wrong more somewhere... the port is active as far as proxmox is concerned, if I can get this working then at least I know the hw is compatible on both sides... aka (problem sits inside pfSense).

I have ordered a DAC cable and a 2nd SFP+, different brand, allot cheaper than these enterprise level Dell/EMC's.

G

Gblenn · Oct 30, 2024, 2:13 PM

@georgelza said in pfSense not enabling port:

@Gblenn

Need to figure out how to get this done... as there is allot of common bits here.
and sharing...

I got a 2nd topton with Proxmox on it... patched that into the Unifi Pro Max, SFP2.
configure Proxmox to use the fiber port ix0 as a 2nd bridge. the port is alive, i can ping the port from local. but i can't ping out

So the bridge you created, how are you using it for thist testing / pinging?
Like, do you have a VM running on Proxmox that has the SFP assigned to it?

Here is what it looks like from a machine of mine, where enp9s0 is the motherboard NIC and enp10s0f0, f1 are the two ports on my X520 card, which I have assigned vmbr1 and 2 respectively.

login-to-view

So if you SSH in to Proxmox and do 'ip a', do you see the connected NIC reading something like this:
login-to-view

Key here is UP, meaning that my port has linked up with something, my switch in this case.

From the Proxmox host perspective I only have an IP assigned to vmbr0, which is where I access the Proxmox host interface (UI and SSH). When I start a VM which has vmbr1 assigned however, I will be able to see the IP from within the VM, and ping from it...

I have ordered a DAC cable and a 2nd SFP+, different brand, allot cheaper than these enterprise level Dell/EMC's.

G

Yes there are plenty available, although sometimes a good idea to check the compatibility list, or from someone who has tested already.

georgelza · Oct 31, 2024, 1:07 PM

@Gblenn said in pfSense not enabling port:

Hi
What I did was click on the pmox1 and click on shell

As per suggestion, see below. Looks good, vmbr30 which sows up

root@pmox1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:72 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:73 brd ff:ff:ff:ff:ff:ff
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:74 brd ff:ff:ff:ff:ff:ff
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.51/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link 
       valid_lft forever preferred_lft forever
10: vmbr30: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    inet 172.16.30.11/24 scope global vmbr30
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe05:f091/64 scope link 
       valid_lft forever preferred_lft forever

If I ping 172.16.30.1 however. also if i ping from my laptop to the 172.16.30.11 ip which is suppose to be assigned on the pmox1 host it fails.

root@pmox1:~# ping 172.16.30.1
PING 172.16.30.1 (172.16.30.1) 56(84) bytes of data.
From 172.16.30.11 icmp_seq=4 Destination Host Unreachable
From 172.16.30.11 icmp_seq=5 Destination Host Unreachable
From 172.16.30.11 icmp_seq=6 Destination Host Unreachable
From 172.16.30.11 icmp_seq=9 Destination Host Unreachable
From 172.16.30.11 icmp_seq=10 Destination Host Unreachable
From 172.16.30.11 icmp_seq=11 Destination Host Unreachable
^C
--- 172.16.30.1 ping statistics ---

Gblenn · Oct 31, 2024, 1:16 PM

@georgelza Ok but the Proxmox host, vmbr0 and the vmbr30 are on different subnets. So unless you have rules set up to allow them to communicate with each other, they can't.

So either you need to put vmbr30 into the same subnet as vmbr0, or make sure it is possible to communicate between the 172.16.10 and 172.16.30 subnets...

That said, since it is clearly saying it is UP, and it is also getting an IP, my guess it is working fine here.
So the card and the module are ok to use with Linux (Proxmox at least). And it is likley only with pfsense (freebsd) that you will have an issue, which you will be able to solve when the DAC and/or new module arrives.

stephenw10 · Oct 31, 2024, 1:16 PM

You are seeing the replies from 172.16.30.11 which implies the pmox1 is using it. Which we know it is.

Host unreachable implies it cannot ARP for the address so a layer2 failure.

georgelza · Oct 31, 2024, 1:21 PM

@Gblenn Let me go check...
I know my MBP that sit on 172.16.20.29 has full access to everything on 172.16.10.0 ... need to confirm I have a rule that allows similar to 172.16.30.0
sure i did check, but lets recheck/verify ;)
G

Gblenn · Oct 31, 2024, 1:25 PM

@stephenw10 Hmm, I missed that it was actually two different IP's there. So I guess the ping was meant to have been to 172.168.30.11? Where the .1 belongs to pfsense VLAN where that .11 IP was actually handed out.

Anyway, I think the DELL module is working on Linux and the DAC and new Fiber module will fix it.

stephenw10 · Oct 31, 2024, 1:27 PM

Well it looks like 172.16.30.11 to 172.16.30.1 to me. And it's failing which implies it cannot ARP for it inside the same subnet.

georgelza · Oct 31, 2024, 1:35 PM

@stephenw10

login-to-view

that last rule is never used as as it already allowed by second.

login-to-view

georgelza · Oct 31, 2024, 1:37 PM

@Gblenn ye... but at the moment, going from the topton hosting pmox through fiber onto unifi sfp+ port 1, and then via my 2.5GbE Cat 6 uplink to pfSense is failing.

vLan30 with 172.16.30.1 lives on igb1.30 at the moment.

G

georgelza · Oct 31, 2024, 1:45 PM

login-to-view

Gblenn · Oct 31, 2024, 1:54 PM

@georgelza said in pfSense not enabling port:

As per suggestion, see below. Looks good, vmbr30 which sows up

root@pmox1:~# ip a
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN group default qlen 1000
link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff
10: vmbr30: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
inet 172.16.30.11/24 scope global vmbr30
valid_lft forever preferred_lft forever
inet6 fe80::aab8:e0ff:fe05:f091/64 scope link
valid_lft forever preferred_lft forever

Hmm, perhaps it isn't working after all. Just noticed that you have the NO-CARRIER notification = No cable connected?! As well as it is not enabled as in "noop state DOWN group".

Further on the vmbr30 there is no LOWER_UP as in the physical connection at the link layer is not there?? Compare vmbr0 to vmbr30:

vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP>
vmbr30: <BROADCAST,MULTICAST,UP>

What does it look like in the Proxmox UI, for the host and the listings under network?

georgelza · Oct 31, 2024, 2:01 PM

@Gblenn see above.

The UI implies it's up...

guess i need to say i wait... this is rightly not a netgate problem... well until i install the DAC cable from the Topton running pfSense into y core switch.

This atm is more unifi/Topton comm...

netgate/pfsense related though, at the moment it's not allowing me to specify a default gw or simply a gw to use for the 172.16.30.0 network.

i can ping from the pmox my 172.16.10.1 gw, but that's going via the 2.5GbE copper link, to switch and onwards to pfSense to the igc0 port

G

Gblenn · Oct 31, 2024, 2:19 PM

@georgelza said in pfSense not enabling port:

@Gblenn see above.

The UI implies it's up...
G

No, the UI only sais that it is administratively activated.

Here's what it looks like for me if I disable the switchport that my 10G link is connected to, same as you have: NO-CARRIER and no LOWER_UP.

3: enp10s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000

And this is what the UI is showing me...

login-to-view

georgelza · Oct 31, 2024, 2:25 PM

@Gblenn
ok, interesting... figured that implied it was working...
even though as you said ip a actually said otherwise.

so it's starting to look like the Pmox host is also not linked to the Unifi pro max...

wondering if this is caused by the switch... think i might need to "engage" some patience and wait for that 2nd SFP+ to arrive and then the DAC cable and then see from there.
Will first try them on the pmox host, as it's more compatible with anything and confirm they work, then if all good there then move them to the pfSense host.

G

Gblenn · Oct 31, 2024, 2:32 PM

@georgelza said in pfSense not enabling port:

login-to-view

login-to-view

Hmm, do you always assign IP from Proxmox? I'm not sure what Proxmox will do in this case... as I would imagine it is Proxmox handling the ICMP request within it's virtualization environment.

If you had a VM that you assigned vmbr30 to (leaving the ipv4 part empty), it would be assigned an IP from pfsense instead. You don't need to put it in any VLAN, as that is only complicating things when testing... But if you want to, it's just a matter of entering the VLAN tag in the field for the VM's interface instead.

But still, the fact that it shows the link is not UP at the interface level, makes me wonder...

georgelza · Oct 31, 2024, 2:31 PM

@Gblenn I figured i'd give the host a ip on the network the card lives.
i will then give the guest vm's their own ip's on that network also.

can easily remove that 30.11

G

stephenw10 · Oct 31, 2024, 2:36 PM

The status of the bridge device is not really important compared to that of the actual NIC. The bridge could appear up even if the NIC is not.

I would check the NIC stats and see if you see any incoming packets on it. ip -s link show enp4s0f0

Gblenn · Oct 31, 2024, 2:36 PM

@georgelza said in pfSense not enabling port:

@Gblenn I figured i'd give the host a ip on the network the card lives.
i will then give the guest vm's their own ip's on that network also.

can easily remove that 30.11

G

Leave all that to pfsense instead, that's where you want to control all those things, including all your static IP's. If you have set up the Unifi SFP+ port as VLAN 30 Untagged, then anything on the Proxmox side will of course get an IP from that range.

georgelza · Oct 31, 2024, 2:38 PM

@stephenw10

root@pmox1:~# ip -s link show enp4s0f0
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN mode DEFAULT group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
        236040    3934    282       0       0    3934 
    TX:  bytes packets errors dropped carrier collsns           
      11447434   67461      0       0       0       0 
root@pmox1:~#