XG-7100 & Unify console troubles: ix0, ix1, dhcp-kea & now sore head

pfS_noob_cust

Serious Noob here, but, it is desired to use the 7100 SFP+ as the WAN connect and the FW, DNS & DHCP server to a downstream Unify console and the other 7100 SFP+ as the LAN connection.

Had it working without the SFP+ connections and relying on the lagg0 switch; the Unify console Ethernet WAN connected via lagg0:3222 and a TRUNK via. lagg0:3333.

Now, the 7100 can't even provide DHCPd services to its own Ethernet switch.
port 1 is the WAN on lagg0:4090
port2 is the LAN on lagg0:4091

port4 & port8 (the only ones I've checked) do not provide DHCPd services. So, going forward seems futile.

Read lots of stuff on how easy it is but 3 days into this my easy button is highly bruised.

Thought I'd ask some smart folks what to do, cuz age is catching up.

Here's what I have...
*** Welcome to Netgate pfSense Plus 24.11-RELEASE (amd64) ***

Current Boot Environment: default
Next Boot Environment: default

NGP1_WAN (wan) -> lagg0.4090 -> v4/DHCP4: 138.....
NGP2_LAN (lan) -> lagg0.4091 -> v4: 10.33.33.1/24
NGIX1_WAN_EVENTUALLY (opt1) -> ix1.4000 ->
NGP3_BLDG (opt2) -> lagg0.612 -> v4: 172.16.12.1/24
NGP4_STORAGE (opt3) -> lagg0.622 -> v4: 172.16.22.1/24
NGP5_DOMAINS (opt4) -> lagg0.632 -> v4: 172.16.32.1/24
NGP6_VPNS (opt5) -> lagg0.662 -> v4: 172.16.62.1/24
NGP8_SEC (opt6) -> lagg0.682 -> v4: 172.16.82.1/24
UDMT_MOBILES (opt7) -> ix0.111 -> v4: 192.168.111.1/24
UDMT_VUZ (opt8) -> ix0.121 -> v4: 192.168.121.1/24
NGP7_SERVICES (opt9) -> ix0.131 -> v4: 192.168.131.1/24
UDMT_GUESTS (opt10) -> ix0.181 -> v4: 192.168.181.1/24
UDMT_THINGS (opt11) -> ix0.191 -> v4: 192.168.191.1/24
NGIX0_PFS_T_UDMWAN (opt12) -> ix0.2222 -> v4: 10.22.22.1/29

The Switch VLANs look like this...

Port 4 ruleset seems open:
FWrules.port4.jpg
Same with Port8:
FWrules.port8.jpg

None the less, the attached devices get little love from DHCPd as only the LAN device can connect...
dhcpd leases.2.jpg

Nothing in the system log about dhcp or within the dhcpd log, save the complaint about multithreading...
Jun 13 09:55:32 kea-dhcp4 11798 WARN [kea-dhcp4.dhcp4.0x17b76c12000] DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 4, queue size: 64
Jun 13 09:55:32 kea-dhcp4 11798 WARN [kea-dhcp4.dhcp4.0x17b76c12000] DHCP4_RESERVATIONS_LOOKUP_FIRST_ENABLED Multi-threading is enabled and host reservations lookup is always performed first.
Jun 13 09:55:32 kea-dhcp4 11798 WARN [kea-dhcp4.dhcpsrv.0x17b76c12000] DHCPSRV_MT_DISABLED_QUEUE_CONTROL disabling dhcp queue control when multi-threading is enabled.

I'm outta ideas and energy. So, I thought I'd ask.
At present, I have an 8 port box, where only two work!

Any guidance would be appreciated.

Also, which would be expected to have more traffic, the pfS-UDM Trunk or the UDM WAN? Cuz, that's the one that would seem to benefit from the SPF+ interface.

ix0 does have a SFP+ module plugged in attached to UDM & they look friendly.

[24.11-RELEASE]/root: ifconfig -vvvm ix0
ix0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
capabilities=4f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 00:08:a2:12:d3:f2
inet6 fe80::208:a2ff:fe12:d3f2%ix0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
supported media:
media autoselect
media 10Gbase-Twinax
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: ix0
plugged: SFP/SFP+/SFP28 10GBASE-T with SFI electrical interface (No separable connector)
vendor: Ubiquiti Inc. PN: DAC-SFP10-0.5M SN: AH24104309902 DATE: 2024-11-12

Guidance on the pfSense side of defining the Trunk & WAN vlan construct would be helpful.

Thanks in advance.

stephenw10

Have you enabled DHCPd on NGP4 and NGP8? It's only enabled on LAN by default.

Have you set the PVID in the switch ports tab to match the VLANs?

pfS_noob_cust

@stephenw10 Thanks for replying.
My short answer is yes, I did.

What's more scary is my actual implementation.

Here's what console shows:
LAN (lan) -> lagg0.4091 -> v4: 10.33.33.1/24
OPT1_IX0 (opt1) -> ix0 ->
OPT2_IX1 (opt2) -> ix1 ->
OPT3_NGP3_BLDG (opt3) -> lagg0.612 -> v4: 172.16.12.1/24
OPT4_NGP4_STORAGE (opt4) -> lagg0.622 -> v4: 172.16.22.1/24
OPT5_NGP5_DOMAINS (opt5) -> lagg0.632 -> v4: 172.16.32.1/24
OPT6_NGP6_VPNS (opt6) -> lagg0.662 -> v4: 172.16.62.1/24
OPT7_NGP8_SEC (opt7) -> lagg0.682 -> v4: 172.16.82.1/24
OPT8_PFS_UDM_TRUNK (opt8) -> lagg0.1221 -> v4: 10.12.21.1/29
OPT9_UDMWAN_IX0_PFS (opt9) -> ix0.2222 -> v4: 10.22.22.1/29
OPT10_UDMT_MOBILES (opt10) -> lagg0.111 -> v4: 192.168.111.1/24
OPT11_UDMT_VUZ (opt11) -> lagg0.121 -> v4: 192.168.121.1/24
OPT12_UDMT_SERVICES (opt12) -> lagg0.131 -> v4: 192.168.131.1/24
OPT13_UDMT_THINGS (opt13) -> lagg0.181 -> v4: 192.168.181.1/24
OPT14_UDMT_GUESTS (opt14) -> lagg0.191 -> v4: 192.168.191.1/24
OPT15 (opt15) -> lagg0 ->

and that reflects my intention.

And that is reflected in /var/db:
---------- 1 root wheel 0 Jun 12 00:38 dhclient.leases.ix1.2222
---------- 1 root wheel 0 Dec 27 2022 dhclient.leases.lagg0.1685
---------- 1 root wheel 2380 Jun 14 20:27 dhclient.leases.lagg0.4090
---------- 1 root wheel 1204 Jun 11 19:21 dhclient.leases.lagg0.4091
---------- 1 root wheel 0 Jun 11 22:52 dhclient.leases.lagg0.612
---------- 1 root wheel 0 Jun 11 22:54 dhclient.leases.lagg0.622
---------- 1 root wheel 0 Jun 12 01:00 dhclient.leases.lagg0.632
---------- 1 root wheel 0 Jun 11 22:57 dhclient.leases.lagg0.662
-rw-r--r-- 1 root wheel 16 Jun 22 2022 dhcp6c_duid
drwxr-x--- 2 _dhcp _dhcp 2 Dec 8 2023 dhcpcd
-rw-r--r-- 1 root wheel 13 Jun 13 09:50 ix0.111_cacheip
-rw-r--r-- 1 root wheel 13 Jun 13 09:49 ix0.111_ip
-rw-r--r-- 1 root wheel 13 Jun 13 09:50 ix0.121_cacheip
-rw-r--r-- 1 root wheel 13 Jun 13 09:49 ix0.121_ip
-rw-r--r-- 1 root wheel 13 Jun 13 09:49 ix0.131_cacheip
-rw-r--r-- 1 root wheel 13 Jun 13 09:49 ix0.131_ip
-rw-r--r-- 1 root wheel 13 Jun 13 09:49 ix0.181_cacheip
-rw-r--r-- 1 root wheel 13 Jun 13 09:49 ix0.181_ip
-rw-r--r-- 1 root wheel 13 Jun 13 09:50 ix0.191_cacheip
-rw-r--r-- 1 root wheel 13 Jun 13 09:49 ix0.191_ip
-rw-r--r-- 1 root wheel 10 Jun 12 22:02 ix0.2222_cacheip
-rw-r--r-- 1 root wheel 10 Jun 12 22:01 ix0.2222_ip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.111_cacheip
-rw-r--r-- 1 root wheel 13 Jun 12 09:13 ix1.111_ip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.121_cacheip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.121_ip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.131_cacheip
-rw-r--r-- 1 root wheel 13 Jun 12 09:13 ix1.131_ip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.181_cacheip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.181_ip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.191_cacheip
-rw-r--r-- 1 root wheel 13 Jun 12 09:14 ix1.191_ip
-rw-r--r-- 1 root wheel 10 Jun 12 09:14 ix1.2222_cacheip
-rw-r--r-- 1 root wheel 10 Jun 12 09:14 ix1.2222_ip
drwxr-xr-x 2 root wheel 3 May 27 04:18 kea
-rw-r--r-- 1 root wheel 13 Jun 8 17:35 lagg0.111_cacheip
-rw-r--r-- 1 root wheel 13 Jun 8 17:35 lagg0.111_ip
-rw-r--r-- 1 root wheel 13 Jun 11 06:44 lagg0.131_cacheip
-rw-r--r-- 1 root wheel 13 Jun 11 06:44 lagg0.131_ip
-rw-r--r-- 1 root wheel 11 Jun 11 15:40 lagg0.212_cacheip
-rw-r--r-- 1 root wheel 11 Jun 11 15:40 lagg0.212_ip
-rw-r--r-- 1 root wheel 11 Jun 11 12:21 lagg0.282_cacheip
-rw-r--r-- 1 root wheel 11 Jun 11 12:21 lagg0.282_ip
-rw-r--r-- 1 root wheel 10 May 30 05:47 lagg0.3222_cacheip
-rw-r--r-- 1 root wheel 10 May 30 05:47 lagg0.3222_ip
-rw-r--r-- 1 root wheel 13 Jun 14 08:27 lagg0.4090_ip
-rw-r--r-- 1 root wheel 10 Jun 13 09:49 lagg0.4091_cacheip
-rw-r--r-- 1 root wheel 10 Jun 13 09:49 lagg0.4091_ip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.612_cacheip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.612_ip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.622_cacheip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.622_ip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.632_ip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.662_cacheip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.662_ip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.682_cacheip
-rw-r--r-- 1 root wheel 11 Jun 13 09:49 lagg0.682_ip

I checked, and each and every one of those interfaces have DHCPd enabled.

At present each of these interfaces, all enabled; are DHCPd configured with "Allow all Clients". No matter which switch port I plug Annabelle into, she reports the LAN address - without fail.

Per the description, "When set to Allow all clients, any DHCP client will get an IP address within this scope/range on this interface. "

I could try other clients, OSes, etc, but the description is clear.

This behavior is consistent with my smaller test, which explored the various "Deny Unknown Clients" options. I did not do that here, but it is my intention to utilize that functionality.

My findings indicate that it doesn't work as intended.
And, THAT, is disconcerting, in that the entire intention of having a discriminating switch, is to ensure, that a device on its port, will have the offered address and VLAN id. That is NOT presently apparent.

I would like to explore how and why this is not working.
Your insights and experience are very welcomee.
Thanks.

patient0

@pfS_noob_cust what did you change from the working to the non working state: did you change the DHCP server from ISC to KEA? Did you change firewall rules?

In your last post "What's more scary is my actual implementation" you don't show the VLANs configuration so I have only the frist picture to go.

Let's narrow it down a bit (I'm a simple creature) and have a look at only one interface:

OPT4_NGP4_STORAGE (opt4) -> lagg0.622 -> v4: 172.16.22.1/24

How are the clients connected to that port, using a switch? On pfSense VLAN 622 is configured only on port 4, in access port mode (called "native VLAN" on Unifi). The traffic will leave port 4 untagged and the switch would have to be configured accordingly.

The firewall rule for OPT4_NGP4_STORAGE allows everything, that can't be the show stopper.

stephenw10

@stephenw10 said in XG-7100 & Unify console troubles: ix0, ix1, dhcp-kea & now sore head:

Have you set the PVID in the switch ports tab to match the VLANs?

Let's see that tab then.

Or the output of etherswitchcfg.

pfS_noob_cust

@stephenw10 said in XG-7100 & Unify console troubles: ix0, ix1, dhcp-kea & now sore head:

etherswitchcfg

Here's the latter...
etherswitchcfg

etherswitch0: VLAN mode: DOT1Q
port1:
        pvid: 4090
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
port2:
        pvid: 4091
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
port3:
        pvid: 4091
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (none)
        status: no carrier
port4:
        pvid: 4091
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
port5:
        pvid: 4091
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (none)
        status: no carrier
port6:
        pvid: 4091
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (none)
        status: no carrier
port7:
        pvid: 4091
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
port8:
        pvid: 4091
        state=8<FORWARDING>
        flags=0<>
        media: Ethernet autoselect (none)
        status: no carrier
port9:
        pvid: 1
        state=8<FORWARDING>
        flags=1<CPUPORT>
        media: Ethernet 2500Base-KX <full-duplex>
        status: active
port10:
        pvid: 1
        state=8<FORWARDING>
        flags=1<CPUPORT>
        media: Ethernet 2500Base-KX <full-duplex>
        status: active
laggroup0:
        members 9,10
vlangroup0:
        vlan: 1
        members none
vlangroup1:
        vlan: 4090
        members 1,9t,10t
vlangroup2:
        vlan: 4091
        members 2,9t,10t
vlangroup3:
        vlan: 30
        members 3,9t,10t
vlangroup4:
        vlan: 40
        members 4,9t,10t
vlangroup5:
        vlan: 50
        members 5,9t,10t
vlangroup6:
        vlan: 60
        members 6,9t,10t
vlangroup7:
        vlan: 2112
        members 7t,9t,10t
vlangroup8:
        vlan: 80
        members 8,9t,10t

What's odd to me, is the last half, pertaining to vlangroup is way outdated.

It's odd, but in exact agreement with the switches vlan page.
switch.vlans.jpg

In short, to answer your question, apparently I did not. oops.
I'll do that now.

pfS_noob_cust

in. addition, here are the interface vlan & assignments pages...
interface.vlans.jpg interface.assignments.jpg

now, nothing plugged into the 7100 switch works.
I did a reboot just I sent this, but after I "repaired" the Switch VLANs.

I'll now do a full power off & restart to ascertain the results.

patient0

@pfS_noob_cust said in XG-7100 & Unify console troubles: ix0, ix1, dhcp-kea & now sore head:

Here's the latter...

You can format that kind of output with by adding a line before and after that contains only three "`" (Markdown syntax)

etherswitchcfg
etherswitch0: VLAN mode: DOT1Q
port1:
pvid: 4090
state=8<FORWARDING>
flags=0<>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
...

As for the VLANs, now they are numbered different, are you trying to confuse me ;)

In "Interfaces / Switch / VLANs", all the VLANs you define on "Interfaces / VLANs" do have to exist too.

For example, the VLAN 622/ngp4_storage is now VLAN 60 on the Switch / VLAN page but still VLAN 622 on the Interfaces / VLAN page.

Do you mind me asking, how well do you understand VLANs, tagged and untagged, access ports?

stephenw10

Yeah you haven't set the PVID on the switch. All the ports are still set to 4091 so no inbound untagged traffic will work. Except on LAN where it's still using 4091.

You need to go to Interfaces > Switch > Ports and set the correct PVID for each port.

https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/configuring-the-switch-ports.html#ports

stephenw10

Ah also the VLAN tags on the switch no longer match the VLANs you have added on lagg0. Previously they did. You'll need to set those back to match.

pfS_noob_cust

@patient0 sorry for being so confusing - not intentional.
Stephenw10 pointed out that I my Switch VLANS and Assignment VLANS were out of sync. So "I repaired" that. I believe there remains some mismatches in this area.

To answer your last question on "my understanding" - pitiful little, apparently.

What I THINK (hoping?) I know is:
port 4 (allow all) devices should get 172.16.22.0/24
port 5 (allow known) devices should get 172.16.32.0/24
port 6 (allow all) devices should get 172.16.62.0/24

static mapped "annabelle" as 172.16.62.5 via the port6 dhcp server page.

when the 7100 LAN (port 2) has its members identified as:

2,9t,10t : dhcp leases reports statically mapped annabelle is up on both the LAN & port 6. But annabelle reports a self assigned IP address. The port 4 device reports not being not connected.
2,3t,4t,5t,6t,7t,8t,9t,10t :
dhcp leases reports statically mapped annabelle is up on both the LAN & port 6. But annabelle reports a self assigned IP address.
The device on port 4 reports a port 5 address, and not one in the pool.

Sure seems like the VLANid assignments did not get re-implemented without leaving some conflicting garbage.

patient0

@pfS_noob_cust I'm sorry but I don't know what to do with these information.

I don't think taking about DHCP is necessary right now, first we have to establish if the VLANs are setup correctly.

Screenshots of "Interfaces / VLANs", "Interfaces / Switch / VLANs" and "Interfaces / Switch / Ports" please.

And then pick one LAN or VLAN to look at. Don't change any VLANs after you posted the screenshots.

pfS_noob_cust

@stephenw10 thanks!
I failed there too! Didn't understand the switch ports page. fixed that so they match now.
switch.vlans.jpg switch.ports.jpg

pfS_noob_cust

@patient0 I think it's fixed now.

stephenw10

Yup looks good.

Just for reference the PVID on port 7 doesn't matter because the VLAN there is passed tagged. It usually doesn't hurt to set it there though.

pfS_noob_cust

@stephenw10 thank you.
Your statement is based on the fact that ~~port 7 is assigned with that VLANid on interface ix0, right? OOPS, Not true.~~
Switch VLANs Group 7 members are are all tagged (7t,9t,10t), right?

stephenw10

Yes exactly. Since all members of that VLAN are tagged it should never see untagged traffic so the PVID isn't used.

However worth baring in mind is that if the downstream device (a switch I assume) incorrectly sends untagged traffic to port 7 it will be tagged onto VLAN 1221. Now normally that's not a problem since the traffic could only ever be one way. Nothing on VLAN 1221 could ever respond. But if such rogue traffic turned out to be a broadcast flood for example that might start to affect something.

pfS_noob_cust

@stephenw10

Thank you. I'll try to absorb what you just said.

@patient0,stephenw10,

I do want to express my gratidude for kindly pointing out my failures of execution in utilizing the GUI properly. I fully appreciate the difficulties of writing words that describe the intent of a GUI.
Again, Thanks.