Tagged VLAN Setup on Single Switch

gmarler · Dec 30, 2021, 5:55 PM

I've got a pfSense setup that currently works just fine, and want to expand into using VLANs to segregate certain types of traffic, since all of my networking equipment seems to support this.

I'm going to be adding tagged VLANs 50, 60, 70, and 99 in addition to the default VLAN of 1, which my switch won't let me eliminate.

But I'm going to start with just VLAN 70 to keep things simple until I know this works.

Here's the current setup I'm testing:

+-----------------------------------------------------------------+
|                                                                 |
|   +---------+  +---------+   +---------+  +---------+           |
|   |         |  |         |   |         |  |         |           |
|   |  OPT2   |  |  OPT1   |   |   LAN   |  |  WAN    |           |
|   |  igb3   |  |  igb2   |   |   igb1  |  |  igb0   |           |
|   +---------+  +----+----+   +---------+  +---------+           |
|                     |                        Protectli VP2410   |
+---------------------+-------------------------------------------+
                      |OPT5 created on
                      |igb1.70
                      |All other VLAN interfaces
                      |on this pfsense port disabled at present
                      |until VLAN 70 confirmed working
                      |OPT5 VLAN 70 IP address: 172.30.70.1/24
                      |
                      |
                      |
                      |
+---------------------+-----------------------------------------------+
|                     |Tagged                                         |
|                                                                     |
|  Indicated ports all                             VLAN Capable Switch|
|  members of VLAN 1 (default)                     Netgear GS748T     |
|  and 50, 60, 70, 99                                                 |
|                                                                     |
|                              This port a member of                  |
|                              VLANs 1 and 70 only                    |
|   |Untagged                        Untagged|                        |
+---+----------------------------------------+------------------------+
    |                                        |
    |                                        |
    |                                        |
    |                                        |
    |                                        |
+---+--------------------------+   +---------+------------------------+
| Solaris DHCP Server          |   |Linux Packet Capture System       |
|                              |   |  Doing packet capture here to    |
| Host with virtual interfaces |   |  confirm activity on VLAN 70     |
| on VLANs 50, 60, 70, 99      |   |                                  |
|                              |   |                                  |
| VLAN 70 IP address is        |   |                                  |
| 172.30.70.64/24              |   |                                  |
|                              |   |                                  |
+------------------------------+   +----------------------------------+

Here's the assignment of the OPT5 interface:
login-to-view

The VLAN config:
login-to-view

The Firewall Rules for OPT5:
login-to-view
The Interface Configuration for OPT5:
login-to-view

The Interface Status for OPT5 (which does seem to indicate OUT errors for some reason I can't determine):
login-to-view

The Routing Gateways:
login-to-view

And the actual IPv4 Routes:
login-to-view

With this setup, here's what I'm seeing:

If I'm on the Solaris server, trying to ping the OPT5 interface (172.30.70.64/24 to 172.30.70.1/24), I don't get any response, but on the Linux Packet Capture System, I do see the VLAN 70 tagged packets being emitted from the Solaris host, just no responses coming back from the pfSense device.
Now I go to the pfSense device and try to ping 172.30.70.64 from the OPT5 interface. I get 100% packet loss, and don't see any packets coming from the OPT5 interface on the Linux Packet Capture System.
If I do the packet capture from the pfSense itself, I see packets coming from 172.30.70.1, but they don't appear to be VLAN tagged, which seems odd.

My questions at present are:

Should the switch port OPT2/OPT5 is connected to be in Tagged or Untagged mode for the VLANs?
*Have I missed a step that would explain either the incorrect routing or non tagging of the traffic coming from the VLAN 70 interface on the pfSense?

bingo600 · Dec 30, 2021, 6:12 PM

@gmarler

1:
Did you add that Vlan70 (OPT5) Gateway your self ?
Gateways are not needed for (on) any local lan interfaces.

2:
Since the pfSense IGB2 IF will transport multi vlans via vlan tags, the switchport you connect IGB2 to , must also be setup to use the same vlan tags (means a tagged switchport). - Here i see 50,60,70,99 , so you must config the switchport for those as tagged.

3:
Then you can make an untagged vlan70 port on the switch , and connect the Sun (or any other untagged ethernet device) there.

The Vlan1 your "less god" (crappy) switch ...
Can't disable , would probably be the "IGB2" switchport's "untagged or native vlan" or PVID.

"crappy" is IMHO allowed to use for "older tp-links" or their "cousins" , that enforces vlan1 on all ports.

/Bingo

johnpoz · Dec 30, 2021, 6:12 PM

@gmarler said in Tagged VLAN Setup on Single Switch:

addition to the default VLAN of 1, which my switch won't let me eliminate.

Huh.. if your switch will not allow you to remove vlan 1 from ports.. Then you really need to get another switch.. That is completely borked and breaks the way vlans are to work.

There is a big difference from not able to remove it from the switch itself, but you need to be able to remove it from ports you want in different vlan.

gmarler · Dec 30, 2021, 7:05 PM

@bingo600 said in Tagged VLAN Setup on Single Switch:

@gmarler

1:
Did you add that Vlan70 (OPT5) Gateway your self ?
Gateways are not needed for (on) any local lan interfaces.

Yes, I added that one manually - I've deleted it now, but doesn't seem to make a difference in the results.

2:
Since the pfSense IGB2 IF will transport multi vlans via vlan tags, the switchport you connect IGB2 to , must also be setup to use the same vlan tags (means a tagged switchport). - Here i see 50,60,70,99 , so you must config the switchport for those as tagged.

Yes, I thought my diagram indicated that I had configured the switch port that igb2 attaches to as being a member of VLANs 1 (untagged), 50, 60, 70 and 99 (tagged). But thanks for answering the question about whether that port should be tagged in those VLANs or not.

3:
Then you can make an untagged vlan70 port on the switch , and connect the Sun (or any other untagged ethernet device) there.

Which I did, and the Sun machine seems to work fine on its (untagged) VLAN 70 switch port.

The Vlan1 your "less god" (crappy) switch ...
Can't disable , would probably be the "IGB2" switchport's "untagged or native vlan" or PVID.

"crappy" is IMHO allowed to use for "older tp-links" or their "cousins" , that enforces vlan1 on all ports.

Ah, I see a potential problem. All of the switch ports have a default PVID of 1 - if I change it to any other PVID, then I can delete VLAN 1 from that switch port, which I've now done. I changed the PVID of that switch port to 70, but not really sure what I ought to set it to in the end. However, I still don't see any difference in the ultimate behavior - still can't ping the IP on the igb2.70 interface for whatever reason - and that interface is showing output errors as I noted above.

/Bingo

gmarler · Dec 30, 2021, 7:07 PM

@johnpoz said in Tagged VLAN Setup on Single Switch:

@gmarler said in Tagged VLAN Setup on Single Switch:

addition to the default VLAN of 1, which my switch won't let me eliminate.

Huh.. if your switch will not allow you to remove vlan 1 from ports.. Then you really need to get another switch.. That is completely borked and breaks the way vlans are to work.

There is a big difference from not able to remove it from the switch itself, but you need to be able to remove it from ports you want in different vlan.

You're right - I just didn't understand how to eliminate VLAN 1 from any given switch port - you first have to change the PVID of that port to anything but 1, then you can delete it from that port.

bingo600 · Dec 30, 2021, 7:24 PM

@gmarler said in Tagged VLAN Setup on Single Switch:

I changed the PVID of that switch port to 70, but not really sure what I ought to set it to in the end.

You can't have a tagged port having bot tagged VlanX and PVID X.
I usually use PVID 999 for all my "Trunk/Tagged Ports" , unless a specific untagged port is needed. And I never use Vlan999 for anything, but to "Blackhole" untagged traffic on a "Trunk/Tagged ports".

johnpoz · Dec 30, 2021, 7:37 PM

@bingo600 said in Tagged VLAN Setup on Single Switch:

I changed the PVID of that switch port to 70, but not really sure what I ought to set it to in the end.

If the switch port is access, ie just 1 device connecting to it that you want in vlan 70, which would be untagged to and from the device then yes setting the pvid to the vlan is correct. Many switches will do this for you automatically when you set a port to be untagged in specific vlan. But if your does not, then yes if setting to vlan 70 access port (cisco term) where only untagged traffic in vlan 70 enters and leaves this port, the the pvid should be 70

@bingo600 statement about setting an odd ball pvid would be when the port is going to carry only tagged traffic.. so on the off chance some untagged traffic is seen on this port, its just black holed to some vlan that goes nowhere..

bingo600 · Dec 30, 2021, 7:40 PM

@johnpoz said in Tagged VLAN Setup on Single Switch:

@bingo600 statement about setting an odd ball pvid would be when the port is going to carry only tagged traffic.. so on the off chance some untagged traffic is seen on this port, its just black holed to some vlan that goes nowhere..

Correct , 999 is only used on my "pure tagged" links.

@gmarler
Have you tried to set the solaris switchport to both untagged 70 & pvid 70
Does those output errors increase when you ping the Solaris ?

Are you sure the Solaris is using untagged ethernet frames ?
You haven't configured any vlan stuff on that one ??

gmarler · Dec 30, 2021, 8:03 PM

@bingo600 said in Tagged VLAN Setup on Single Switch:

@johnpoz said in Tagged VLAN Setup on Single Switch:

@bingo600 statement about setting an odd ball pvid would be when the port is going to carry only tagged traffic.. so on the off chance some untagged traffic is seen on this port, its just black holed to some vlan that goes nowhere..

Correct , 999 is only used on my "pure tagged" links.

@gmarler
Have you tried to set the solaris switchport to both untagged 70 & pvid 70
Does those output errors increase when you ping the Solaris ?

Are you sure the Solaris is using untagged ethernet frames ?
You haven't configured any vlan stuff on that one ??

I tried setting the Solaris switchport to both untagged 70 & PVID 70, with no difference in behavior, and the output error count on OPT2 of the pfSense is not increasing.

However, the Solaris system started by only being on a subnet that I'm going to retire (192.168.1.0/24), and I'm now going to put it on VLANs 50, 60, 70, and 99, to serve as the DHCP server for all of them, then retire 192.168.1.0/24.

So I think the issue might be that I've only got the default Solaris interface on 192.168.1.0/24 (no VLAN tagging), so it's going over the default VLAN. And I've got interfaces for VLANs 50, 60, 70, and 99 configured, with only VLAN 70's interface up. I can see the packets from that interface coming out tagged as VLAN 70 as they leave the Solaris system. BUT... I suspect this means I need to change the single switch port for this connection from Solaris to tagged for those VLANs, eventually. And when I do, untagged traffic from the default (non-VLANed) interface will simply stop. Do I have that right?

Separately from that, and after the changes so far, I now see, from the packet capture on the pfSense, the packets from the Solaris system showing up, but they're no longer tagged as being VLAN 70 - so I suspect the fact that the Solaris attached switch port being untagged is simply stripping the VLAN 70 tags as they enter the switch.

To me, that means that I have to configure the Solaris box to be purely VLAN'ed, with no interface being on the default VLAN - AND mark it's switch port as tagged for at least VLANS 50, 60, 70, and 99 - but not sure about that.

But I also see that packets coming out of the pfSense igb2.70, before they enter the switch, are also not tagged as being VLAN 70. That seems concerning.

gmarler · Dec 30, 2021, 8:14 PM

Ah, yes, that did it.

I set the Solaris attached switch port to tag VLAN 70, and suddenly the pings from Solaris to pfSense worked just fine.

But this switch won't let you tag some VLANs and not others, so after a few seconds, all of that port's other VLANs switch from untagged to tagged, and the default VLAN (which is all other traffic from that host) can't pass, while the VLAN 70 traffic passes just fine.

I wonder if just explicitly setting the original/default interface on Solaris to be in tagged VLAN 1 for the time being would clear this up.

bingo600 · Dec 30, 2021, 8:19 PM

@gmarler said in Tagged VLAN Setup on Single Switch:

I tried setting the Solaris switchport to both untagged 70 & PVID 70, with no difference in behavior, and the output error count on OPT2 of the pfSense is not increasing.

OK - Have you tried a different pds cable to the switch ?

However, the Solaris system started by only being on a subnet that I'm going to retire (192.168.1.0/24), and I'm now going to put it on VLANs 50, 60, 70, and 99, to serve as the DHCP server for all of them, then retire 192.168.1.0/24.

So I think the issue might be that I've only got the default Solaris interface on 192.168.1.0/24 (no VLAN tagging), so it's going over the default VLAN. And I've got interfaces for VLANs 50, 60, 70, and 99 configured, with only VLAN 70's interface up. I can see the packets from that interface coming out tagged as VLAN 70 as they leave the Solaris system. BUT... I suspect this means I need to change the single switch port for this connection from Solaris to tagged for those VLANs, eventually.

Yes if your Solaris is tagging , you's need the Switchport to accept those tags.
AKA configure it for tagging on the vlans of interest.

And when I do, untagged traffic from the default (non-VLANed) interface will simply stop. Do I have that right?

Depends ...
If your Solaris would send an untagged packet , it would by the switch be put into the PVID vlan.
That said i can't see why the Solaris should send any untagged packages , if you have just configured tagged interfaces in that box.

Separately from that, and after the changes so far, I now see, from the packet capture on the pfSense, the packets from the Solaris system showing up, but they're no longer tagged as being VLAN 70 - so I suspect the fact that the Solaris attached switch port being untagged is simply stripping the VLAN 70 tags as they enter the switch.

If you don't see any tags on the pfSense capture , then those packages would be sent untagged from the solaris.

To me, that means that I have to configure the Solaris box to be purely VLAN'ed, with no interface being on the default VLAN - AND mark it's switch port as tagged for at least VLANS 50, 60, 70, and 99 - but not sure about that.

That would be correct, and the same goes for the switchport the Solaris is connected to.

But I also see that packets coming out of the pfSense igb2.70, before they enter the switch, are also not tagged as being VLAN 70. That seems concerning.

I'll have to agree .. concerning.

bingo600 · Dec 30, 2021, 8:25 PM

@gmarler said in Tagged VLAN Setup on Single Switch:

Ah, yes, that did it.

I set the Solaris attached switch port to tag VLAN 70, and suddenly the pings from Solaris to pfSense worked just fine.

But this switch won't let you tag some VLANs and not others, so after a few seconds, all of that port's other VLANs switch from untagged to tagged, and the default VLAN (which is all other traffic from that host) can't pass, while the VLAN 70 traffic passes just fine.

Get a new switch , that functions correct.
Life is to short for bad switches.

I wonder if just explicitly setting the original/default interface on Solaris to be in tagged VLAN 1 for the time being would clear this up.

Why would you do that ?

If you want to keep that switch.
Why not let pfSense be your DHCP server , and just run the Solaris as untagged (normal) ethernet , and put the solaris switchport in "Untagged 70 + PVID 70"

johnpoz · Dec 30, 2021, 8:27 PM

@bingo600 said in Tagged VLAN Setup on Single Switch:

are also not tagged as being VLAN 70. That seems concerning.

how exactly are you seeing these? sniffing on with say pfsense for vlan 70 in packet capture would not show the tag.

You would need to use tcpdump or on the parent in promiscuous mode in the gui..

So for example I have a igb2 with native untagged network and vlan 4 and vlan 6 riding on it.. If I sniff on the parent in packet capture, set to full details. I can see untagged traffic and vlan 4 and vlan 6 traffic.

Here is small snip

14:22:18.938762 00:08:a2:0c:e6:20 > a8:1b:6a:24:ec:26, ethertype 802.1Q (0x8100), length 70: vlan 4, p 0, ethertype IPv4, (tos 0x0, ttl 237, id 10371, offset 0, flags [DF], proto TCP (6), length 52)
    18.232.219.54.8883 > 192.168.4.81.39146: Flags [.], cksum 0x2717 (correct), seq 1, ack 1, win 425, options [nop,nop,TS val 4065030919 ecr 133836713], length 0
14:22:18.985750 0c:51:01:8c:19:ae > 00:08:a2:0c:e6:20, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.253 tell 192.168.2.200, length 46
14:22:19.168749 00:08:a2:0c:e6:20 > 18:db:f2:3e:44:ce, ethertype 802.1Q (0x8100), length 182: vlan 6, p 0, ethertype IPv4, (tos 0x0, ttl 54, id 19259, offset 0, flags [DF], proto UDP (17), length 164)
    64.43.241.202.4500 > 192.168.6.142.52559: [udp sum ok] UDP-encap: ESP(spi=0x214629e8,seq=0x2947), length 136
14:22:19.187563 d4:a6:51:d1:a2:8a > 00:08:a2:0c:e6:20, ethertype 802.1Q (0x8100), length 127: vlan 4, p 0, ethertype IPv4, (tos 0x0, ttl 255, id 29915, offset 0, flags [none], proto TCP (6), length 109)
    192.168.4.58.61343 > 52.12.196.80.8886: Flags [P.], cksum 0xca2b (correct), seq 484189:484258, ack 2631772778, win 3828, length 69

where you can see untagged the 192.168.2 stuff, and then vlan 4 and 6 192.168.4 and 192.168.6 traffic.

bingo600 · Dec 30, 2021, 8:40 PM

@gmarler

Sorry i first noticed now , that i could scroll in the top post network diagram.
Missed that.

You have a Netgear GS748T.

Even though i'm not a fan of Netgear , i find it hard to beleive that it can't do basic tagging correct.

For the Linux sniffer port , you'd want to use the mirror port function in the switch. And decide if it should mirror the Solaris switchport or the pfSense switchport. The mirror function would copy every frame from the source port to the mirror port. And then Wireshark won't miss a single bit.

Edit: Seems like you might be able to use multiple source ports

https://www.downloads.netgear.com/files/GS748T_UM_30Oct07.pdf
Pg. 53
login-to-view

johnpoz · Dec 30, 2021, 8:46 PM

@bingo600 said in Tagged VLAN Setup on Single Switch:

i find it hard to beleive that it can't do basic tagging correct

Same here, I have used netgear now and then over the years, and have never seen any problems with tagging. I don't have any experience with that specific model. But it sure isn't an entry level model ;) Not at 48 ports..