The 1100 is one switch with three ports. Very wild guess but the last paragraph of that doc caught my eye: "With both the LAN and OPT switch ports using the same VLAN on the switch (4091), the firewall will receive traffic from either port on its mvneta0.4091 interface, which is assigned as LAN by default." It sounds like LAN is detected as down and that prevents access from OPT.
What happens if you swap them and put your PC on OPT?
Wanted to give an update:
I did receive my 16-port Unifi Switch Lite 16, swapped it in and moved some switches around. Doing so, I was able to take out two Ubiquiti Edge Router X's (in VLAN Switch mode) and a simply unmanaged switch. Now the only brand of switch I am using is Unifi switches.
After taking out the Edgerouters, things started to work as expected. I'm very familiar with the Unifi switches, but a little less so with the Edgerouters. Despite my best attempts to set them up properly with the correct VID's and PVID's for my different VLANS, ports, and trunks downstream from my primary switch, I must have still gotten something wrong and been creating some sort of STP issue.
As I said, now that I am using only Unifi switches, things are working as expected, so we seem to be all clear! Thanks again for all of your help and input!
@creationguy You don't typically untag more than one vlan on a port. While vlan 20 'should' work on that port, the others definitely won't as the device plugged in wouldn't be tagged so all egress traffic would go out on vlan 20 (pvid).
Just think it through, Trunk ports carry vlans to where you need them.
Access Ports let you use those vlans.
Have to assume port 24 goes to pfSense, then just untag the ports as you need them with just the vlan needed.
If you need to carry the vlans to another device, use a trunk and tag the vlans needed on it, then untag ports that will use each specific vlan.
Switchports that connect to a device should be untagged.
Tag the interface in switch one going to pfSense.
make sure both vlans in pfSense are on the same interface.
Then tag port one in both switches with both vlans.
all other ports are untagged.
pfSense to switch one, tagged with both vlans.
sw1 port1, tagged with both vlans.
sw2 port 1, tagged with both vlans.
All others untagged with appropriate vlans as needed.
Thank you so much that has work, I really appreciate your advise and taking the time to help me.
What is the best way to move Proxmox to VLAN 50, it is still on my existing DHCP range of 192.168.1.x? Would it be best to VLAN aware linux bridge VMBR0 or should it be done via Shell?
@miracullix So just bridge the ports together and give it a try. You can always "undo" what you setup - go in reverse order to tear it all apart.
Under Interfaces, select the Bridge button. In there, click the Add button. In there add the 2 ports you want together (use the shift key on the keyboard to select multiple ports) and then click the Save button. Keep in mind, the only interfaces you can add to a bridge are "enabled" interfaces. In other words, they have to be active. I think the 4100 comes with all interface ports enabled.
So, now that you have a bridge added, you have to enable it and set it up. Be careful here, I think you could inadvertently lose your LAN connection and the IP address range you already had on it.
Long story short, I don't believe you can simply click a couple of buttons and add another available interface to a bridge. There's a little bit of setup, and some pretty good setting tweaks. And, obviously the performance hit. So, that's why it's said to just add a switch to keep it simple.
if I want to block the traffic from VLAN1 to VLAN2, shall I write a block rule from VLAN1?
Correct the rule would go on the vlan1 interface, you block traffic before it enters pfsense.. Why would you allow traffic into your house and then stop them from leaving your house.
Here is an analogy I like... Someone knocks at your front door and says hey can I go to your back yard.. Do you let them into the house and let them walk through your living room with their dirty shoes and then stop them as they try and leave out your backdoor. Or do you just not let them in the front door in the first place ;)
I could prove that L2 isolation from Ubiquity works only for certain devices
No idea what that is suppose to mean, I have unifi AP and had no issues with L2 isolation. So not sure what your doing or asking about.
Your talking about devices connected to the same SSID/vlan not able to talk to each other? Are they both wireless, is one wireless and other wired?
Are you trying to stop a wireless client from talking to wired client on the same L2? Or a wired client on the same L2 from talking to the wireless client on that L2.
Pfsense has zero to do with devices on the same L2 network from talking to each other - there is no way pfsense could do that, unless you setup device A on one side of bridge and client B was on the other side of the bridge.
I mean that by default, L2 isolation is not behaving in the way I expect, it should block the traffic between the 2 SSIDs I have, but it doesn't completely, some device is able to ping the other one, some not.
Plus, the same story is between the SSID VLAN and another "wired" VLAN, which comes from the switch, nothing to do with the AP.
Anyway, I've been testing a lot and I can see that nothing major changed.
These below are the most remarkable scenarios:
I create a rule to stop the traffic from one VLAN to all the others.
How did I do it? I created an alias with the correct IP range of all the VLANs, obviously is much more extensive but it's fine.
It doesn't work unfortunately, or it's intermittent, it's not the first time that I end up in this situation in Pfsense, not even a system reboot helped. On the fly, I also tried to clear the DHCP leases, reset the filter table, reset the state table etc, no positive result.
If I enable the rule above only, it goes in that intermittent behaviour, but after it blocks the traffic (almost entirely), there is no way back...
If I also add up a rule to stop the firewall access from that VLAN, it locks that too.
It doesn't make much sense this behaviour because the filter rules are reloaded almost on the fly, I could have considered my IP range as an issue if it didn't happen in real time...
There is one problem here, the gateway IP is in that range obviously, so, technically speaking, it makes sense to be locked out...
There is one funny thing though, the ping to 220.127.116.11 works :D , and this is the second inconsistency that I find with Pfsense...
The same ping to the modem/router works, which is out of range in that rule alias, but still, the Gateway IP is the first entry point and it should be blocked, so why it doesn't block everything...
I mean, how did I reach the modem/router without passing from the default VLAN gateway, which should be blocked...
Below I list more details:
Gateway IP: 10.70.70.1 (this should be blocked)
Machine IP: 10.70.70.2
Gateway IP: 10.70.71.1 (this should be blocked)
Machine IP: 10.70.71.2
The funny story is always the same, if I create a rule to allow the traffic from the VLAN 1 network to ANY, everything works, but I need to block the traffic to the firewall IPs (usually set automatically as every VLAN gateway IP), plus I need to block the traffic to other VLANs. NOTE that what you said it's not valid in my box, Pfsense doesn't allow the traffic automatically everywhere, it blocks the traffic but not ping, so I need to create a dedicated rule to allow the traffic.
I can do the extensive manual test to lock these things one by one, but there is already some weird behaviour, I don't think it will change much...
I'll also try to act from other VLANs where I have other devices, in this VLAN used for testing, I smell Microsoft issues...
@swemattias while I would agree with you on the mini its a bit off.
Switches are switches are switch, vlans are vlans are vlans.
The vlan is either tagged or it isn't.. When you connect a end user device to a port on a switch is is almost always untagged.. Unless you have gone into the OS of that device and specifically set it to understand a tag..
Almost always a end user device will be untagged on a switch port, with the port in that vlan, and the pvid set on the port that you want the traffic to be in.
I've finally managed to get this fixed, thanks to a kind soul found on the Internet. I basically got schooled(again!) on layer 2 traffic and having an extra pair of eyes go through the firewall config, I found out what the problem was. I was basically trying to shoehorn VLAN traffic through the switch and causing a loop(even with loop prevention turned off). However, this was not affecting my regular traffic which made me continue to troubleshoot and assume that my configuration was correct.
Considering my requirement has been that VMs talk to each and gets update over the internet and nothing outside of these VLANs, I added another interface to pfsense(trunk port) and in pfsense, changed the VLANs to be going through the new interface, rather than still pushing it through the physical LAN which I was trying to do. I now get DHCP AND the machines are able to reach out to the internet.
Once I added the trunk network interface as an additional NIC, it showed up as a 3rd interface on pfsense which showed as vmx2
I used the third NIC to pass my VLAN traffic
Earlier, I had configured VLAN to be going vmx1, by letting the traffic go out through the LAN/Trust interface and then trying to get it back through the same port (since I didn't have another NIC free on ESXi). Now, all my VMs are getting the correct IP address range
@jarhead Are you actually seeing a problem or just a few counters and only when you reboot? Lots of things happen when a system is rebooted and not just on the pfSense side. Assuming you're connected to a switch which also will run through some link up/link down procedures. If I was only seeing a few error counters on reboot and then no further incrementing or problems, I would personally move on to something else.
You'll likely need to send screen shots of the interface assignment page and detail exactly what you are doing to attempt to reassign the VLAN to the physical if.
@martywise have you tried to reconfigure the native lan for the port connected to your pfsense box. You could make the native lan of the switch trunk port different than the rest of the switch so it doesn't pass data other ports having same native vlan