pfSense not enabling port

[georgelza]

@Gblenn
know the 2.5GB ports are i226 based.
think need to figure out what chipset the 10GbE SFP+ cages are.

the Unifi's are taking the Dell/EMC SFP+'s happily... i would have expected it to have more of a problem than the Topton.

G

[Gblenn]

@georgelza Why ten? The Topton only has 2 SFP+ ports and it's only on the pfsense side that you are having the issue.

The Unifi Pro Max seems to recognize the modules but it too only has 2 SFP+ ports, right? All the rest are RJ45? If you plan on getting more switches, and stick to Unifi, I'm sure you can make use of all those Dell SFP+ modules you have.

But don't give up just yet, perhaps you can get pfsense to work with the Dell module as well?
So instead of messing with your LAN, create a "test bed" the way I explained earliers, by isolating the port on the Unifi. I believe they have something called port isolation as well, but not sure how that works vs your uplink port.

[georgelza]

@Gblenn ...

I have 6 top ton's. and 2 other machines.
first is the pfSense, going into core,
Core to go into 8 port aggregation, where the other topton's and 2 other machines are to go.

Their 10GbE SFP+ cards are still to be ordered,
<My storage network is going 10GbE, everything else is to be 2.5Gbe via the i226 ports>

G

[Gblenn]

@georgelza Ok, but will you run pfsense on all of the Toptons? My guess this is a driver issue and it will likely work with e.g. Linux, which I would suggest you test on one of the other machines then.

Another way could be to virtualize pfsense on Proxmox for example. If Proxmox works fine with the modules, you can assign the virtual interface instead. Not sure what performance degradation you would see but it's one way around the issue at least. And the i5-1335 seems powerful enough, as it beats the i5-11400 which I have in my machine running pfsense on Proxmox. At least comparing per thread performance.

[georgelza]

@Gblenn

pfSense is on first topton, others wil run Proxmox, already have one... need to order the balance and the aggregation switch which are the 8 additional ports.

trying to get this one working, onto the switch... implying the card/SFP+'s etc all works together.

G

[Gblenn]

@georgelza Thing is though, pfsense runs on freebsd, not Linux, and what you are seeing here is likely a driver issue related to freebsd.

Proxmox on the other hand is Debian which will have different drivers, and Unifi switches are also Linux based. So it may be so that you can use your Dell modules in all your other machines, not just the one running pfsense.

But the only way to find out is to test it... Plug a module into the Proxmox boxe, connect it to the Unifi switch and you will see immediately if it works or not...

[stephenw10]

@georgelza said in pfSense not enabling port:

you forgetting i'm re-assigning the main lan interface... i have one lan interface over which i run multiple vlans, this is the link between me... wifi into core switch or hard wired into core switch.

Can you not add one of the other igc NICs as a management interface at least temporarily?

[georgelza]

@Gblenn

Hehehe
Exactly what I started cabling this morning.
. Will advise ltr.

Curious, what dif would a DAC cable make as that’s just everything pre packaged, what I’m getting from our problem is pfSense not liking the Dell/EMC SFP+ itself.

G

[georgelza]

@stephenw10
I have a spare usw flex mini. Going to see if I can configure a management network using that directly into the pfSense. As a backup/backdoor
G

[Gblenn]

@georgelza said in pfSense not enabling port:

@Gblenn

Hehehe
Exactly what I started cabling this morning.
. Will advise ltr.

Curious, what dif would a DAC cable make as that’s just everything pre packaged, what I’m getting from our problem is pfSense not liking the Dell/EMC SFP+ itself.

G

There is actually a bit of software (firmware) in all those modules (transceivers). And it is this firmware may which is giving you the incompatibility issue here. So either you have to solve it in pfsense, with a driver that works with the DELL module. Or replace the module with something different.

And no it doesn't have to be a DAC cable... a fiber module from a different vendor may also work. It's just that DAC's are typically cheaper...

[Gblenn]

@georgelza said in pfSense not enabling port:

@stephenw10
I have a spare usw flex mini. Going to see if I can configure a management network using that directly into the pfSense. As a backup/backdoor
G

Or do what I suggested, use the ix0/1 as the testing connections towards your USW instead. Keep the working stuff as is and don't start reassigning interfaces until things are working with the 10G ports.

[georgelza]

@Gblenn

Need to figure out how to get this done... as there is allot of common bits here.
and sharing...

I got a 2nd topton with Proxmox on it... patched that into the Unifi Pro Max, SFP2.
configure Proxmox to use the fiber port ix0 as a 2nd bridge. the port is alive, i can ping the port from local. but i can't ping out, which tells me something is wrong more somewhere... the port is active as far as proxmox is concerned, if I can get this working then at least I know the hw is compatible on both sides... aka (problem sits inside pfSense).

I have ordered a DAC cable and a 2nd SFP+, different brand, allot cheaper than these enterprise level Dell/EMC's.

G

[Gblenn]

@georgelza said in pfSense not enabling port:

@Gblenn

Need to figure out how to get this done... as there is allot of common bits here.
and sharing...

I got a 2nd topton with Proxmox on it... patched that into the Unifi Pro Max, SFP2.
configure Proxmox to use the fiber port ix0 as a 2nd bridge. the port is alive, i can ping the port from local. but i can't ping out

So the bridge you created, how are you using it for thist testing / pinging?
Like, do you have a VM running on Proxmox that has the SFP assigned to it?

Here is what it looks like from a machine of mine, where enp9s0 is the motherboard NIC and enp10s0f0, f1 are the two ports on my X520 card, which I have assigned vmbr1 and 2 respectively.

login-to-view

So if you SSH in to Proxmox and do 'ip a', do you see the connected NIC reading something like this:
login-to-view

Key here is UP, meaning that my port has linked up with something, my switch in this case.

From the Proxmox host perspective I only have an IP assigned to vmbr0, which is where I access the Proxmox host interface (UI and SSH). When I start a VM which has vmbr1 assigned however, I will be able to see the IP from within the VM, and ping from it...

I have ordered a DAC cable and a 2nd SFP+, different brand, allot cheaper than these enterprise level Dell/EMC's.

G

Yes there are plenty available, although sometimes a good idea to check the compatibility list, or from someone who has tested already.

[georgelza]

@Gblenn said in pfSense not enabling port:

Hi
What I did was click on the pmox1 and click on shell

As per suggestion, see below. Looks good, vmbr30 which sows up

root@pmox1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:72 brd ff:ff:ff:ff:ff:ff
4: enp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:73 brd ff:ff:ff:ff:ff:ff
5: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:02:a3:74 brd ff:ff:ff:ff:ff:ff
6: enp4s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr30 state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:b8:e0:05:f0:92 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:02:a3:71 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.51/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe02:a371/64 scope link 
       valid_lft forever preferred_lft forever
10: vmbr30: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a8:b8:e0:05:f0:91 brd ff:ff:ff:ff:ff:ff
    inet 172.16.30.11/24 scope global vmbr30
       valid_lft forever preferred_lft forever
    inet6 fe80::aab8:e0ff:fe05:f091/64 scope link 
       valid_lft forever preferred_lft forever

If I ping 172.16.30.1 however. also if i ping from my laptop to the 172.16.30.11 ip which is suppose to be assigned on the pmox1 host it fails.

root@pmox1:~# ping 172.16.30.1
PING 172.16.30.1 (172.16.30.1) 56(84) bytes of data.
From 172.16.30.11 icmp_seq=4 Destination Host Unreachable
From 172.16.30.11 icmp_seq=5 Destination Host Unreachable
From 172.16.30.11 icmp_seq=6 Destination Host Unreachable
From 172.16.30.11 icmp_seq=9 Destination Host Unreachable
From 172.16.30.11 icmp_seq=10 Destination Host Unreachable
From 172.16.30.11 icmp_seq=11 Destination Host Unreachable
^C
--- 172.16.30.1 ping statistics ---

[Gblenn]

@georgelza Ok but the Proxmox host, vmbr0 and the vmbr30 are on different subnets. So unless you have rules set up to allow them to communicate with each other, they can't.

So either you need to put vmbr30 into the same subnet as vmbr0, or make sure it is possible to communicate between the 172.16.10 and 172.16.30 subnets...

That said, since it is clearly saying it is UP, and it is also getting an IP, my guess it is working fine here.
So the card and the module are ok to use with Linux (Proxmox at least). And it is likley only with pfsense (freebsd) that you will have an issue, which you will be able to solve when the DAC and/or new module arrives.

[stephenw10]

You are seeing the replies from 172.16.30.11 which implies the pmox1 is using it. Which we know it is.

Host unreachable implies it cannot ARP for the address so a layer2 failure.

[georgelza]

@Gblenn Let me go check...
I know my MBP that sit on 172.16.20.29 has full access to everything on 172.16.10.0 ... need to confirm I have a rule that allows similar to 172.16.30.0
sure i did check, but lets recheck/verify ;)
G

[Gblenn]

@stephenw10 Hmm, I missed that it was actually two different IP's there. So I guess the ping was meant to have been to 172.168.30.11? Where the .1 belongs to pfsense VLAN where that .11 IP was actually handed out.

Anyway, I think the DELL module is working on Linux and the DAC and new Fiber module will fix it.

[stephenw10]

Well it looks like 172.16.30.11 to 172.16.30.1 to me. And it's failing which implies it cannot ARP for it inside the same subnet.

[georgelza]

@stephenw10

login-to-view

login-to-view

that last rule is never used as as it already allowed by second.

login-to-view