pfSense with OpenWRT Guest logon with VLAN

[Ramosel]

@stephenw10
OK, back in the saddle. I know I needed a break (not that dealing with family matters was a good break) and I felt I owed you one too.
I got the June 6 Snapshot... did they re-break it?

Model	Linksys WRT3200ACM
Architecture	ARMv7 Processor rev 1 (v7l)
Target Platform	mvebu/cortexa9
Firmware Version	OpenWrt SNAPSHOT r19756-d5e48a1e8e / LuCI Master git-22.137.71281-d6dbedd

If I rebuild the network on the br-lan with or without the "Bridge VLAN Filtering"
I get this:

root@Testbed_OpenWrt:~# bridge v
port              vlan-id
lan4              1 PVID Egress Untagged
lan3              1 PVID Egress Untagged
lan2              1 PVID Egress Untagged
lan1              1 PVID Egress Untagged
wan               1 PVID Egress Untagged
wlan0             1 PVID Egress Untagged
wlan1             1 PVID Egress Untagged
br-lan            1 PVID Egress Untagged
root@Testbed_OpenWrt:~# uname -a
Linux Testbed_OpenWrt 5.10.115 #0 SMP Mon Jun 6 08:19:20 2022 armv7l GNU/Linux

At the time I collected the above, the VLAN filtering was invoked and working.

I have not upgraded WPAD as of yet. Just added LuCI and ip-bridge.

[stephenw10]

Hard to imagine it would have broken again but....it's always a possibility!

So you moved LAN to br-lan.1?

You need to set VLAN 10 as local in the bridge filtering so it creates a br-lan.10 device and configures the internal port to include it as tagged so the router can use it.

So for reference:

Steve

[Ramosel]

@stephenw10
Thanks, I'll try again.
The good news is, I now know this will work... we may not be there yet, but it will work.
The bad news is:
Yesterday, I could not get the VLAN filtering to create the .1 and .10 devices. I would just get locked out on the save and had to revert. I had to manually create them and then set the VLAN filtering as I posted yesterday. I couldn't get the bridge vlan command to show the .10 device in any manner.
I'm rural and can barely see neighbors but my security conscience tells me to shut the box off and leave this for the next day. This morning I cannot reach LuCI or the SSH. Box looks up, but I can't get to it. I was just about to reset it and had a text to come through on my old iPhone I use for initial testing. A friend sends me a link and when I open it, it's full of ads... wait, pfBlockerNG usually kills all that stuff. I guess I left that phone pinwheeling on a failed logon to GusetTest and it connected. I checked and sure enough, the Guest network is working... and assigning addresses on the correct network. So at least I'm 100% sure pfSense is correctly configured for what we are working on.

The rest of the bad news is neither of the OpenWRT SSIDs are allowing a logon at this time. At this point I don't know if that is the either/or problem on the wireless of the inability to access the primary network. What I do know is I turned off the wireless on the toughbook and plugged it into one of the LAN ports on the router and it gets a good connection and address on the 192.168.1.0 network.

[stephenw10]

Did you change the LAN interface to use the br-lan.1 device? With DSA doing what it's supposed to you will be locked out without that change.

It's still possible there is a bug in the WRT3200 build. The hardware details need to be correct for DSA to do the right thing with it.

I wish I could find a way to read the actual switch config. Or maybe see what DSA thinks is happening. This is a 5min task with swconfig.

Steve

[Ramosel]

@stephenw10 said in pfSense with OpenWRT Guest logon with VLAN:

Did you change the LAN interface to use the br-lan.1 device? With DSA doing what it's supposed to you will be locked out without that change.

Thanks for posting that. That is what I suspected I failed to do. I may take the box apart and use a TTL connection and see if I can edit the network file to .1 device. I've got some things to do but I'll get on this later for sure this afternoon.

[Ramosel]

@ramosel said in pfSense with OpenWRT Guest logon with VLAN:

@stephenw10 said in pfSense with OpenWRT Guest logon with VLAN:

Did you change the LAN interface to use the br-lan.1 device? With DSA doing what it's supposed to you will be locked out without that change.

Thank you, Steve!!
It IS working. The Guest network logs on and gets an address on the prescribed network.

root@Testbed_OpenWrt:~# bridge vlan
port              vlan-id
lan4              1 PVID Egress Untagged
lan3              1 PVID Egress Untagged
lan2              1 PVID Egress Untagged
lan1              1 PVID Egress Untagged
wan               1 PVID Egress Untagged
                  10
br-lan            1
                  10
wlan1             1 PVID Egress Untagged
wlan0             1 PVID Egress Untagged
wlan1-1           10 PVID Egress Untagged

I've put some feelers out to see if/when they expect to release this fix in a 21.02.4 or a 22.xx?? I have to do some strange configuration of my physical network to make the SNAPSHOT work so if the release is imminent, I may wait to push this. My lord, what a slog.

Steve, my offer still stands on the WRT3200ACM hardware. If you'd like to have one in your test library I'll gladly ship one your way. My buddy just sent me another one so now I have a 2 backups and a spare. Just get an address to me.

John, thank you for your help as well. You helped get this ball rolling. I'm not getting old, I'm already there and this may be my last fight. I certainly have learned over the years that the more I know about something, the less I value it's quality. I'm certainly going to start gathering some of the Unify toys and learning the systems. I swore when I retired, I wasn't going to jump into anything else... but I've already broken that promise.

It just goes to show the value of the knowledge and the people in the pfSense community. I reached out on this problem to OpenWRT, Reddit and OneMarcFifty's discord. I got help to a point then it exhausted. The solution was found here.

Rick

Edit to fix the version numbers for OpenWRT. Thanks to the poster who mentioned my mistake to me. Fingers were faster than my brain.

[stephenw10]

Ah, great result!

Yeah, somewhat more involved that anticipated but at least we learned some things along the way. My opinion of the DSA is only slightly dulled, I'm unconvinced of it's advantages. But if everything is going that way I guess we will need to be on board.
I could certainly use a WTR3200ACM if you have spares. I'll PM you.

Steve

[HarrisonMarlay]

This post is deleted!

[stephenw10]

For anyone following I'm not sure how I didn't see this earlier:
https://forum.openwrt.org/t/what-happened-to-eth1-now-that-we-are-using-dsa/89807
That's exactly what I was querying in this thread previously. The take-away being:

Yes, the DSA framework currently only supports a single CPU port; eth1 -while present- is ignored by the kernel for the time being.

So you can only use a single NIC queue. That's pretty much the difference between our own 1100 and 2100 and the performance difference there is huge.

At least at some point it will probably be restored. Still no idea how the user is expected to configure it though.

Steve

[Ramosel]

@stephenw10 Thanks for posting that here Stephen. Everything is still working smoothly on this end. The "stable" release of v21 still doesn't support what we found/fixed, but it still works fine in the snaphost version we used. I'm testing the rc6 of v22 and will be rolling my configs to whatever the release of v22 will be. Keep your ears on, I'm stirring up a new esoteric stew on the ethernet side this time.

[stephenw10]

Yes, it seems like a solid device. I've been updating snapshots periodically. Bit of a PITA having to reinstall Luci every time but....
The only thing that doesn't work are the LEDs. A number of the defined LEDs do not work at all or do something unexpected. Are you seeing that? Not really a huge deal.

Steve

[Ramosel]

@stephenw10 I've not experienced any issue with LEDs (edit see PS)... and I used that device for about 6 rotations of bench test vs. live network test mules. hmmm. I do use the System/LED Configuation drop down to change the WAN connection to an amber, but otherwise leave it alone. That's only because the power LED does weaken but works substantially better as amber.

Yes, after changing snapshots 4 or5 times, reloading LuCI got to be a pain - especially since I don't like putting unconfigured machines on the network. I'd been following the 22.03 development so when they put out rc4, I just switched over to 22.03. Fortunately those builds do come with LuCI pre-installed.

I do take a config backup, reset to defaults, then update from a squashfs-factory image each time. Then all I have to reload are WPAD and the ip-bridge diagnostic.

PS: on these 3200s the LEDs for the two radios seem much, MUCH more durable than the older 1900s and 1200s. The constant flicker of the 2 radio band LEDs weakened them to a point you could only see them in a pitch black room. I have a hand full of surface mount LEDs left from replacing mine on the older boxes. Let me know if you need a couple.

[Ramosel]

@stephenw10 Thanks again for all your help, I really didn't feel like putting the Ver 21 snapshots into full use on my Dumb-APs then they stopped development on V21 and V22 RCs started coming out. I did testing with rc 4, 5, and 6 and they all looked promising so I just sat on rolling out the VLAN guest network on the WRT3200ACMs I have. Finally, Ver 22.03.0 (stable) was released so I have upgraded all my Dumb-APs and implemented the Guest network delivery via VLAN. I also had a need to have an ethernet IOT network pushed out over the backhaul connections to the remote routers. Again, your help made that easy to provision as well. I now have Guest network running on a separate network IPv4 with client isolation and a IOT network hard wired through another VLAN to run IP cameras at the out buildings on my property. All is well.... well, not quite.

The new provisioning is getting IP addresses via the VLANs and DHCP servers on pfSense. The configured OpenWRT Network on the WRT3200ACMs seem functional. Web pages load just fine. But, I came across something early this week I can't get to work. Whether the Guest network WiFi or the IOT ethernet, neither will allow a Speedtest (ookla) to run. The devices, WiFi or ethernet on the primary networks do. Whether using the Speedtest.net website or their App, once you get it up, when you hit start it says connecting and just pinwheels until it stops and says "Test failed to complete. Check your internet connection and try again". Any Ideas??

[stephenw10]

Hmm, that's curious. Any other sites that fail?

Do you have captive portal running on either?

Speedtest doesn't require anything special usually.

[johnpoz]

@stephenw10 If I had to guess maybe 8080 is blocked.. If can not talk on 8080, it can not find a server to test to, and would fail to start..

If I block 8080 I get this.

[stephenw10]

Oh, nice! Yeah, almost certainly that. Learned something today.

[Bob.Dig]

I learned that lesson earlier myself.

And just to let you know, maybe take a look at FreshTomato and if your device is supported. It is much easier to set up.

Spoiler

[Ramosel]

@stephenw10 said in pfSense with OpenWRT Guest logon with VLAN:

Hmm, that's curious. Any other sites that fail?

Do you have captive portal running on either?

Fail, not that I am aware of. Just Speedtest so far. Oddly, devices (even iOS) on the WiFi connections are unaffected.

No, don't run captive portal. I'm really rural and my guest count is single digit.

[Ramosel]

@johnpoz Yep, that's the exact indication I get. I do not have any rules blocking 8080. My rules on the VLANs block all of them from accessing LAN and accessing each other. LAN has access to all the VLANs. I don't run any rules, routing or dhcp on the blue box routers. They are all just Dumb-APs on a physical backhaul,

Observation: on an iOS wifi client on the routers, I can see this same result. But, if I play around with the test server I can get it to work.

So....
I know there is some scuttlebutt around the net about some (especially streaming) services playing games with Starlink clients since they often have out of state POPs. It would have to be unfortunate timing on my part that this problem cropped up just as I implemented the VLAN isolation to my routers. I'm very rural and did a lot of the early Beta testing for Starlink and it is a game changer for rural so I won't be switching. But there are factions out there with disdain for Musk so I'm not saying it isn't possible. If I use the speedtest util built into the Starlink software, it works every time.

I don't think this is a pfSense problem but that is just a gut feeling. I can't seem to find the real reason.

And yes the little "John Poz" voice in my head keeps whispering "the solution starts with U"

[johnpoz]

@ramosel said in pfSense with OpenWRT Guest logon with VLAN:

I do not have any rules blocking 8080.

But do you have rules limiting to only specific things? Simple thing to do would be to sniff the traffic and then fire up the speedtest web or app, and see where it tries to go..

If it works with specific servers - some of their servers don't use 8080, etc..

Are you routing traffic on these vlans out a specific gateway, like a vpn or something?