Enabling access between LAN and other non-WAN interface

granroth · Mar 16, 2024, 8:31 PM

I find myself unable to access my hosts on another interface (direct or VLAN) from within my LAN through ssh or http. Only ping works. I have read the pfSense docs and very many tutorials but following those guides hasn't worked for me.

I have a Protectli box running pfSense CE 2.7.2-RELEASE. The network interfaces that matter for this are LAN on igb1, IotVLAN on igb5.20, and a basic OPT5 to enable the igb5 port itself.

The IotVLAN (igb5.20) is enabled and has static IPv4 Address 10.100.20.1. OPT5 (igb5) is also enabled with static IPv4 Address 10.100.1.1

Both of igb5 interfaces have DHCP enabled and both will assign a valid dynamic IP to any request client plugged into igb5, depending if they have the VLAN ID set or not. The hosts assigned to either OPT5 or IotVLAN can access the internet through my WAN gateways.

I am able to ping hosts on LAN from OPT5 or IotVLAN hosts and vice versa. I am unable to ssh or http between these hosts in either direction.

Putting a packet trace on the LAN interface and attempting to ssh to igb5 hosts, I can see entries on port 22 as the request passes through LAN to igb5 with no response. Packet traces on either interface on igb5 show nothing at all.

I don't have very many firewall rules. The rules that seem to come into play:

LAN:

pass in quick on igb1 inet from <LAN__NETWORK> to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule" label "id:0100000101" ridentifier 100000101

IotVLAN:

pass in quick on igb5.20 inet from <OPT3__NETWORK> to any flags S/SA keep state label "USER_RULE: Allow IotVLAN to any rule" label "id:1708405871" ridentifier 1708405871

OPT5:

pass in quick on igb5 inet from <OPT5__NETWORK> to any flags S/SA keep state label "USER_RULE: Allow OPT5 to any rule" label "id:1710031352" ridentifier 1710031352

If any screenshots of the web configurator or raw CLI output would help, let me know and they will be forthcoming.

I'm sure I'm missing something very simple but I'm also missing what that is.

viragomann · Mar 16, 2024, 8:57 PM

@granroth
Ensure that the devices allow access from other subnets. By default most devices block access from outside of their own subnet.

Also this issue could be due asymmetric routing, maybe a layer 2 leak on a switch, where both subnets are connected.

granroth · Mar 16, 2024, 9:52 PM

@viragomann
Okay, let's talk about all the devices in scope. I had a bear of a time getting VLANs to work at all and so this is as stripped down as I can get such that any host gets a VLAN IP.

LAN is connected to Protectli igb1 from a TP-Link TL-SG116E Smart Switch (managed and supports VLANs in practice, but I couldn't get them to work at all through this switch so assume unmanaged)

OPT5/IotVLAN is connected to Protectli igb5 from port 1 on a TP-Link SG2218 "Omada" managed switch. It is assigned 10.100.1.10 since it is on OPT5 by default. Omada abstracts the tagged/untagged/pvid nomenclature and so I have port 1 set to essentially "untagged all" to pass VLAN IDs and port 2 set to VLAN 20 ("IotVLAN").

I have a Raspberry Pi is my IotVLAN Host. It is running latest Debian; configured with DHCP; and is plugged into port 2 on the SG2218. I can see the DHCPDISCOVER/OFFER/REQUEST with the VLAN 20 ID when packet tracing IotVLAN in pfSense and, indeed, the Raspberry Pi gets an IotVLAN IP of 10.100.20.10.

The LAN Host for this testing is a Mac Mini going through various unmanaged switches but ultimately going through the (assume unmanaged) TL-SG116E switch into the LAN port in pfSense. It has a LAN IP.

When running a packet trace in pfSense (Diagnostics -> Packet Capture) on the LAN interface, I can see SSH requests on port 22 going from my Mac Mini to the Rasberry Pi on 10.100.20.10. There is no response in the packet feed.

When running a packet trace in pfSense on either OPT5 or IotVLAN, I see absolutely nothing at all on port 22.

viragomann · Mar 17, 2024, 6:40 PM

@granroth
Since the SSH packets enter the LAN interface, I'd expect that you can also see them going out on the VLAN interface, if it is configured properly and the LAN rule allows any.

However, if you capture the pings you can see the either packets, request and reply, on both interfaces?

NightlyShark · Mar 17, 2024, 7:37 PM

@granroth I've had problems when I tried creating a non-VLAN interface on a NIC that already has VLANS on it. Botch the OPT5. You don't need it to have VLANs on igb5. Or, if you really need the subnet, create a VLAN ID 1, eg igb5.1, and create the subnet interface there. OR, better yet, make the switch port a trunk port.

NightlyShark · Mar 17, 2024, 7:35 PM

@granroth VLAN ID 1 is usually the default VLAN for all switches.

granroth · Mar 18, 2024, 1:32 AM

@NightlyShark
Yeah, I'd prefer to just use the LAN (igb1) interface, as well. Alas, I've been beating my head against the wall trying to get VLANs to work for the past couple of weeks with very minimal success. Tutorials online (pfSense and all others) pretend that implementing VLANs in trivial when, in reality, they are incredibly fragile constructs where every part of an often multi-layered solution needs to deal with VLANs exactly right -- if even a single component in the chain falters, then the entire concept of VLANs falls to pieces.

So the big question for me was: which part of my setup was the root fault of my VLANs not working?

Working with OPT5 was my attempt to narrow down my debugging to just pfSense as much as I could. If I could prove that pfSense itself was working perfectly, then I could work down the chain, one suspect switch at a time, to figure out where things were failing.

But as you say, working with a non-standard interface creates its own problems, so it found myself having to work around those problems first.

granroth · Mar 18, 2024, 1:49 AM

I followed @viragomann's line of thinking and focused very intently on the devices in question. Specifically, I started by concentrating multiple hosts on the single proven-working Omada managed switch and experimented with them being part of OPT5 or part of IotVLAN or both and then in each case, connecting and receiving connections from/to LAN, OPT5, and IotVLAN.

When the dust settled and I collated all of the successful and failed attempts, it was pretty obvious that the root problem was my original SG116E "Smart Switch" since it was only connections to and from that switch that potentially failed. As long as the hosts were all on the Omada switch -- regardless of what LAN or VLAN they were on -- they would work with each other. Furthermore, any hosts on the Omada switch could typically connect to any host on the LAN network, even though that went through the suspect "Smart Switch".

My plan, now, is the replace the "Smart Switch" with my new Omada switch and see if all of my problems just disappear.

NightlyShark · Mar 18, 2024, 1:54 AM

@granroth said in Enabling access between LAN and other non-WAN interface:

@NightlyShark
Yeah, I'd prefer to just use the LAN (igb1) interface, as well. Alas, I've been beating my head against the wall trying to get VLANs to work for the past couple of weeks with very minimal success. Tutorials online (pfSense and all others) pretend that implementing VLANs in trivial when, in reality, they are incredibly fragile constructs where every part of an often multi-layered solution needs to deal with VLANs exactly right -- if even a single component in the chain falters, then the entire concept of VLANs falls to pieces.

So the big question for me was: which part of my setup was the root fault of my VLANs not working?

Working with OPT5 was my attempt to narrow down my debugging to just pfSense as much as I could. If I could prove that pfSense itself was working perfectly, then I could work down the chain, one suspect switch at a time, to figure out where things were failing.

But as you say, working with a non-standard interface creates its own problems, so it found myself having to work around those problems first.

Look. The best way to handle VLANs (if you do not have a Layer 3 switch) is:

Forget VLANs exist for a second, and decide how many LANs you need. Lets say 2 LANs, X and Y.
Remember again that VLANs exist
Select two random numbers from 2 to 4096
One number is for X, the other for Y
Mark the port number of the port that PfSense goes to on the switch
Mark the port numbers of the ports that the devices of X go to on the switch
Mark the port numbers of the ports that the devices of Y go to on the switch
Get a laptop with ethernet and connect it to a port on the switch
Access the switch's GUI through the laptop
Do a config reset and reboot the switch
Go to switching -> VLANS
For every VLAN,
- enter the VLAN ID (the random number)
- enter the numbers of the ports that the devices of the LAN in question connect on, where it says "Member ports". If you select them graphically (like my Netgear switch), select all those ports as U, "Untagged". If not, enter those numbers where it says "Untagged ports".
- enter the number of the port PfSense connects on where it says "Tagged ports". Or, select it graphically as T, "Tagged".
- for all ports of the VLAN except PfSense, give them a PVID = current VLAN ID
Select a third random number. Mark it on your notebook as Management VLAN
Select the port you PC goes to, create the management VLAN with PfSense port tagged and PC port untagged
Find the switch option for Management VLAN
Change it to the Management VLAN ID
Decide which IPv4 subnet the Management VLAN will have and change the IP of the switch WebGUI to an IP in that subnet
Make a note of that IP
Go to PfSense
Delete all interfaces already existing on the device (port) that will connect to the switch
Delete any firewall rules referencing those deleted interfaces
Create the VLANs in the device (port) that goes to the switch
Create an interface for every VLAN
DO NOT CREATE A NON-VLAN INTERFACE ON THE PfSense PORT THAT GOES TO THE SWITCH
Set IPs
Set FW rules
Set DNS Resolver, DHCP Server (for every VLAN), NTP
Enjoy

Example:

Here you create the VLANs

Here you can select the ports and Tagged/Untagged

Here you can set everything

Where it says "VLAN Tag" it means VLANs that are tagged on that port (the device on the port gets frames with VLAN tags for those VLANs).

Where it says "Untagged VLANs", the switch simply subtracts the tagged VLANs from all VLANs the port is a member, and those left are the untagged VLANs.

Port PVID is the VLAN ID that is assigned to all Untagged frames that go from the cable to the switch port (inbound). PVID of 1 means no VLAN tag, but is a VLAN tag, just ignore it.

Acceptable frame means to configure if you want the port to accept incoming frames (from the connected device) with a VLAN tag, without a VLAN tag, or both.

Forbidden VLANs are VLANs that, based on the config, this port can never talk to, because there is no path.

NightlyShark · Mar 18, 2024, 1:51 AM

@granroth That smells like bad port quality + cheap cable. Won't create problems in TCP, but UDP suffers quietly, and ends up wrecking your nerves, by breaking DNS. Start wireshark and you will see the the tale-telling "Spurious TCP retransmission".

@granroth said in Enabling access between LAN and other non-WAN interface:

I followed @viragomann's line of thinking and focused very intently on the devices in question. Specifically, I started by concentrating multiple hosts on the single proven-working Omada managed switch and experimented with them being part of OPT5 or part of IotVLAN or both and then in each case, connecting and receiving connections from/to LAN, OPT5, and IotVLAN.

When the dust settled and I collated all of the successful and failed attempts, it was pretty obvious that the root problem was my original SG116E "Smart Switch" since it was only connections to and from that switch that potentially failed. As long as the hosts were all on the Omada switch -- regardless of what LAN or VLAN they were on -- they would work with each other. Furthermore, any hosts on the Omada switch could typically connect to any host on the LAN network, even though that went through the suspect "Smart Switch".

My plan, now, is the replace the "Smart Switch" with my new Omada switch and see if all of my problems just disappear.