Connecting Cisco switch to pfsense difficulty.

stampeder · Sep 11, 2024, 6:22 PM

Opt 2 is configured in pfsense as a lan port.

I cannot ping from my Lan port to the devices on VLAN 100.

If I remove port 3 of the cisco switch and connect my second desktop ( with a vlan 100 address) to Opt2 the ping works successfully.

If I connect my 2nd desktop (with a VLAN 100 address) to the cisco port 3 I can ping the UUT devices without issue.

I have firewall rules in the FW-7541 firewall set as forward any protocol to any port.

Not sure why the desktop cannot ping the UUT devices from the LAN port.
I've set the Nx up as a layer 2 switch. The Eth1/3 port is configured as an access port to vlan 100 which is the inband management lan for the UUT devices.

I can connect a pc to port Eth1/3 and access either UUT device on vlan 100. I have the pfsense LAN port configured as a lan port with the same ip address and gateway as the UUT devices.

So the UUT's are on ip 172.16.95.120 and 121 on vlan 100. The Lan port is configured with the same 172.16.95.1/24 network as a lan port (so not gateway selected).

Thoughts?

johnpoz · Sep 12, 2024, 5:13 PM

@stampeder said in Connecting Cisco switch to pfsense difficulty.:

UUT devices

I take it you mean some device under test.. You mention trunk on the ports they are connected too.. But what is this vlan 1000,

What is the config on these ports 1 and 2? If you want vlan 100 to be untagged did you set that as native, ie untagged on the port. Is your pvid set to 100?

When you say you configured opt2 as a "lan" port I take that to mean you just put a native network on it, no tagging..

So for example here is a config for interface on my sg300 cisco where I set vlan 2 on the switch as the native, or untagged vlan

interface gigabitethernet14
 description "uap-ac-lite (guestroom)"
 switchport trunk allowed vlan add 4,6-7,110
 switchport trunk native vlan 2
!

stampeder · Sep 12, 2024, 3:43 PM

@johnpoz Thanks for the reply.
Yes, UUT (Unit Under Test).
Vlan 100 is an inband management vlan. Vlan 1000 is the customer traffic vlan between the two UUT's. The purpose of the 3048 is to emulate a network between the two devices.
Both ports 1&2 on the 3048 are configured as trunk portw with vlan add 100,1000.
Port 3 on the 3048 was configured as an access port and was added to vlan 100. This port is the "injection" point for the management of the two UUT devices. (It's how I can configure them remotely).
For opt2 I configured it with network 172.16.95.1/24 with no tagging.
Thanks for the example. What was the setup on the pfsense box to correspond to the sg300 config?
I going to change my running config on the 3048 to align with what you have.
Thanks again for your suggestions.

johnpoz · Sep 12, 2024, 4:24 PM

@stampeder the config on the pfsense is native vlan set on the interface, and then other vlans (tagged) are assigned to that interface, in my case igb2

🔒 Log in to view

You don't see vlan 7 on igb2 because it uses its own physical interface without any tag in pfsense, just like my igb2.. The switches vlan 7 is connected to that igb5 interface.

stampeder · Sep 12, 2024, 4:45 PM

@johnpoz Here are some screen shots of the interface setups on my Netgate FW-7541.
🔒 Log in to view
🔒 Log in to view
The ETX assignment is where my cisco 3048 connects.
Next here is a snapshot of part of the running config of the 3048
🔒 Log in to view
I added the port 4 as a test port configured as a trunk.
So, my thought is that I have the ETX port on the FW7541 configured correctly and the problem is the configuration on the Cisco 3048. I think I should be adding vlan 100 to the vlan access statement for both Eth 1/1 and Eth 1/2.
Thoughts?

johnpoz · Sep 12, 2024, 5:11 PM

@stampeder in your drawing you said you had both vlan 100 and 1000 on both 1 and 2, but your switch config only shows vlan 1000.. Yeah you should add vlan 100 as native, ie untagged.. If you send it tagged then both your test devices would need to be setup to understand the tag.

Also all of your ports are misconfigured - you can't be in mode trunk with access vlan like that.

For a trunk port you would have allowed vlans, like in my example - if you want one of those untagged then you would set that vlan as native.

stampeder · Sep 12, 2024, 7:29 PM

@johnpoz Doah....It has been a long time since I configured a Cisco switch.......So, traffic between the two UUT devices should just pass right through the switch on ports 1 and 2. So I really don't need to set port 100 as native on these two ports. What I do need to configure is for port eth1/1 and eth1/2 to pass all vlans.
As for eth1/3 I will set it as native vlan 100. Does this look correct:

🔒 Log in to view

Comments?

johnpoz · Sep 12, 2024, 7:33 PM

@stampeder if they were access then sure ok.. But if your going to trunk a port, you have to allow the vlan - but it switch would expect a tag.. If you want a vlan to be untagged, then you would need to set the native, unless its using the default vlan on the switch.

You wouldn't need to set native on port 3, its in access mode - you would just put it in the vlan on the switch you want that traffic to be on, ie 100 I take it.. And make sure the pvid is 100 as well. cisco would normally set that for you when you set it to access mode and put it in vlan X, etc.

If you want device to see untagged traffic on that trunk, that would be the native vlan for that port. For it to do anything with vlan 1000 then the device would have to be setup to understand the tag and send whatever traffic you want on 1000 with a tag.. That would be done on the device.

that example I gave above is an AP, it has a management IP that is my vlan 2 on the switch.. but I also send tagged traffic to it and get tagged traffic from it for its wifi networks it handles via different ssids..

So yeah the to talk to the AP management IP that traffic has to be untagged.

Your config for 1 and 2 would work if the devices are sending tags for the traffic you want on vlan 100.

stampeder · Sep 12, 2024, 7:43 PM

@johnpoz So the traffic between eth1/1 and eth1/2 has two vlans (100 and 1000). Hence I believe they should be trunk ports which allow both vlans.
The traffic coming out of port eth1/3 has only the vlan 100 traffic on it (this is my inband management) Vlan 100 uses the network ip of 172.16.95.0/24. This connects to the pfsense box on my ETX lan port. This is where I am a little confused.
Here is what I have so far:

🔒 Log in to view
According to what I am reading the ETX lan port expected native vlan (ie untagged traffic).
When I first configured my 3048 switch I had port 3 set to switchport access. Should I be reverting eth1/3 on the 3048 to an access port? How will I tell it to pass only vlan 100 traffic?
Sorry to be so thick on this. But I am getting closer. I appreciate your patience....

johnpoz · Sep 12, 2024, 7:47 PM

@stampeder if the only traffic on port 3 is going to be vlan 100, and untagged then yes that would be a access port in vlan 100.

If you want untagged traffic on trunk port, you will need to set vlan 100 as native.. on your trunk ports

stampeder · Sep 12, 2024, 8:10 PM

@johnpoz I set eth1/3 to mode access. I specified vlan 100. So with this connected to pfsense on my ETX port I should be able to go into Diagnostics and ping UUT#1 at 172.16.95.120:

🔒 Log in to view

However, my pings come back as 100% loss. Where am I going wrong in this? I have my firewall rules set to pass any to any.

BELAY THAT ORDER SAILOR....IT WORKS....
I had my lan plugged into the wrong port on the 3048. Once I plugged into the correct port the ping works like a charm on both UUT's.
You are my new hero.
THANKS SO MUCH FOR ALL YOUR HELP. Colour me one happy camper!!

johnpoz · Sep 12, 2024, 8:21 PM

@stampeder pinging what from where? Your trunks on 1 and 2 are still not right.. If the device is not sending a tag with its vlan 100 traffic its not going anywhere.. And if you ping from pfsense to one of the device, that traffic to the device would be tagged. And the device wouldn't see it, etc..

You need to set your ports 1 and 2 to be native vlan 100 so that traffic is not tagged, and any untagged traffic that enters the port from the device will be put on vlan 100.

edit: its not going to work like that.. unless your default vlan on the switch is 100? The default vlan on a trunk would be untagged.

Can you do say this command

sg300-28#sho interfaces switchport gi14 
Added by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, T-Guest VLAN, V-Voice VLAN
Port : gi14
Port Mode: Trunk
Gvrp Status: disabled
Ingress Filtering: true
Acceptable Frame Type: admitAll
Ingress UnTagged VLAN ( NATIVE ): 2
 
Port is member in: 
 
Vlan               Name               Egress rule     Added by     
---- -------------------------------- ----------- ---------------- 
 2                 Wlan                Untagged          S         
 4                W_PSK                 Tagged           S         
 6               W_Guest                Tagged           S         
 7                W_Roku                Tagged           S         
 9                  9                   Tagged           D         
110               W_Cam                 Tagged           S         

 
Forbidden VLANS: 
Vlan               Name               
---- --------------------------------

I not sure if I have access to a NX3048 in work network, and don't for sure have one at home.. We have 7k and 9k etc.., would have to look if any of those old switches are still in use, so not 100% sure that command is valid on them..

But notice that it lists vlan 9, that is the default vlan on my switch.. But I don't want untagged traffic to be on vlan 9, I want it on vlan 2 which is why I set native to 2.. Your config works if your default vlan is 100 on your switch.

stampeder · Sep 12, 2024, 8:19 PM

@johnpoz Crap. I spoke too soon. When I issued the ping it was from source autoselect which I believe was issued from the gateway of the ETX port.
Anyway when I used source as LAN it of course did not work as you pointed out.
I"ll go an add the vlan 100 as native on both eth1/1 and eth1/2. I'm assuming that all I have to add to eth1/1 and eth1/2 is"switchport trunk native vlan 100" I should also remove the vlan 100 from the "trunk allowed vlan" line?

johnpoz · Sep 12, 2024, 8:23 PM

@stampeder no you shouldn't have to remove it.. notice my command for allowed is an add, not a direct allowed list.. Your still going to want to allow the vlan..

stampeder · Sep 12, 2024, 8:56 PM

@johnpoz I tried to change the vlan allowed statement to add in eth1/1 and eth1/2. However, it doesn't appear the NX-OS 9.3 allows it as a config statement. Turns out one has to do the vlan add statement using the port-channel interface.
I'm off to do a bunch of reading on how to use the port-channel interface in NX-OS.
Isn't this fun.,....

johnpoz · Sep 12, 2024, 9:18 PM

@stampeder huh, those ports are not in a port channel.

conf t
interface ethernet1/1
switchport trunk native vlan 100

Keep in mind not sure what differences there might be with NX-OS 9.3

edit:

here
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/layer2/503_U1_1/b_Cisco_n3k_layer2_config_gd_503_U1_1/b_Cisco_n3k_layer2_config_gd_503_U1_1_chapter_0101.html

🔒 Log in to view

stampeder · Sep 13, 2024, 2:53 PM

@johnpoz Thanks for this and your patience. I have definitely been working on NetGear, TP-Link an Zyxel managed switches for too long and forgot all my Cisco commands stuff. Ignore my comment about port channel as I was reading about how to add a vlan to a trunk port.
My switch is doing some weird things. If I add a description of say port eth1/1 the running-config will display what I entered. If however I enter the command:
switchport trunk allowed vlan add 1000
the command does not show up on the running-config listing. If however I enter the command:
switchport trunk allowed vlan 100,1000
this command does NOT show up on the running-config listing even though according to the Cisco Nexus 3000 Series NX-OS Layer 2 Switching Configuration Guide, Release 9.2(x) it is an allowed command:
Configuring the Allowed VLANs for Trunking Ports
You can specify the IDs for the VLANs that are allowed on the specific trunk port.

Before you configure the allowed VLANs for the specified trunk ports, ensure that you are configuring the correct interfaces and that the interfaces are trunks.

SUMMARY STEPS
switch# configure terminal
switch(config)# interface {type slot/port | port-channel number}
switch(config-if)# switchport trunk allowed vlan {vlan-list all | none [add |except | none | remove {vlan-list}]}
DETAILED STEPS
Command or Action Purpose
Step 1
switch# configure terminal

Enters global configuration mode.

Step 2
switch(config)# interface {type slot/port | port-channel number}

Specifies an interface to configure, and enters interface configuration mode.

Step 3
switch(config-if)# switchport trunk allowed vlan {vlan-list all | none [add |except | none | remove {vlan-list}]}

Sets allowed VLANs for the trunk interface. The default is to allow all VLANs on the trunk interface: 1 to 3967 and 4048 to 4094. VLANs 3968 to 4047 are the default VLANs reserved for internal use by default; this group of VLANs is configurable. By default, all VLANs are allowed on all trunk interfaces.

Note
You cannot add internally allocated VLANs as allowed VLANs on trunk ports. The system returns a message if you attempt to list an internally allocated VLAN as an allowed VLAN.

Example
This example shows how to add VLANs to the list of allowed VLANs on an Ethernet trunk port:

switch# configure terminal
switch(config)# interface ethernet 1/3
switch(config-if)# switchport trunk allow vlan 15-20

Do you think that as it says "The default is to add all VLANS" that it doesn't store it because it is already allowed?

I'm going to go back to the start of our conversation and reread it all. I think I am missing something.
By the way my name is Glenn....

johnpoz · Sep 13, 2024, 3:08 PM

@stampeder Not sure what your going on to be honest.. You have gone down some rabbit hole of your own making... I have told you multiple times now how to configure your ports.. you need to set 100 as native vlans on those ports.

I even linked to the cisco docs that show you how to set it as native.