Replacing VLAN switch in router on a stick configuration

airlab · Jan 25, 2020, 1:27 AM

I am currently in the middle of implementing pfsense in my home network. As an MVP I set up an Intel NUC in a router on a stick configuration with a 5 port TP link managed vlan switch

I set up the pfsense box with vlan 999 for my WAN and vlan 40 for LAN. I configured vlan 999 with DHCP and vlan 40 with a static IP and DHCP server.

on the switch:
Port 1 on the switch is the trunk connected to the NUC
Port 2 is untagged vlan 999 for WAN -> connected to my cable modem
Ports 3-5 are untagged vlan 40 for LAN -> connected to an AP and a desktop

Everything is working great. I got everything set up to replace my ubiquiti equipment and it was running fine for the past week.

I decided to change out the switch for a 16 port version of the same series and configured it the same except made ports 3-16 with vlan 40 untagged.

The problem I'm running into is that I can't get the WAN port to get an IP with the new switch. I tried releasing the IP and renewing the lease multiple times. I swapped the original switch in and it works fine. I also tried moving the WAN port to a different physical port with no luck. When I connect to the modem directly with a PC it gets an IP fine from the ISP.

I have not changed anything in my pfsense config yet since I am still learning the interface. I am thinking if I should try reinstalling pfsense and setting it up fresh with the new switch or is there a way to do this for the WAN interface without reinstalling? Is there some sort of dependancy that the first switch caused in my config? Maybe the MAC address of the first switch is a factor here?

Any ideas on what I may be missing?

JKnott · Jan 25, 2020, 2:27 AM

@airlab said in Replacing VLAN switch in router on a stick configuration:

5 port TP link

Bad choice. Some TP-Link gear doesn't handle VLANs properly.

airlab · Jan 25, 2020, 3:32 AM

@JKnott the 5 port TP link is the one that is working... and has been working for over a week now.

JKnott · Jan 25, 2020, 11:43 AM

@airlab

The problem is that it leaks multicasts between VLANs There is also an issue where VLAN1 appears on all ports. @johnpoz can tell you about that one.

johnpoz · Jan 25, 2020, 3:18 PM

What specific tplink switch do you have, and what firmware is on it?

This is a long running issue where their cheap entry level smart switches (maybe even others made by them) do not allow you to remove vlan 1 from port. So yeah you would get leakage between your vlans you setup. Now they supposedly fixed it with firmware for the v3 of hardware. But they never released the firmware for v1 or 2..

But I found out recently that you could install the v3 firmware on the v2 hardware. And it did allow you to remove vlan 1 from ports you wanted in a different vlan. This works - but I have yet to validate that it not leaking info... I just validated that the web gui does allow you to remove vlan 1.. There is a thread around here somewhere where I posted that you can indeed install the v3 firmware on v2 (the one I have)..

At some point I will look to see if actually working correctly.. But personally I would just stay away from that brand for switches and or AP that are suppose to do vlans. Since I just don't think they get how vlans are suppose to actually work ;) it took them well over a year of people complaining for them to come out with a "fix" And then they never bothered to back port the firmware to their previous models. There are posts on their forums where they stated that it was normal and on purpose that you couldn't remove vlan1 ;)

edit: Here is the thread where v3 firmware works on v2 hardware
https://forum.netgate.com/topic/149308/can-t-reach-my-switch-s-management-interface-from-my-vlans

airlab · Jan 25, 2020, 3:18 PM

thanks for the info @johnpoz and @JKnott . I am going to look into a different switch solution. I had the 5 port laying around and after I proved the concept to work I bought the 16 port one so I'm going to be returning that asap.

The 5 port model is the TL-SG105E v4 and the 16 port model is the TL-SG1016PE v1. My APs are unifi AP ac pros.

So am I just "lucky" that my current set up is working with the 5 port switch? I am just using this as a stop gap until I get my T620 plus set up with a dual/quad NIC next week. Not sure if I will be sticking with the router on a stick config once I have more interfaces available.

johnpoz · Jan 25, 2020, 3:21 PM

Does your 105eV4 allow you to remove vlan 1 from the ports you have put in other vlans? You set the pvid on the ports?

So its still leaking vlan traffic across, be it multicast or broadcast?

airlab · Jan 25, 2020, 3:39 PM

@johnpoz It allows me to remove ports from being a member on vlan 1. I also have the PVID set for each port.
How can I check if it is leaking?

JKnott · Jan 25, 2020, 3:40 PM

@airlab said in Replacing VLAN switch in router on a stick configuration:

The 5 port model is the TL-SG105E v4 and the 16 port model is the TL-SG1016PE v1

If that 16 port switch uses the same software as the V1 5 port, it has the problem. Hopefully they got it fixed with V4.

airlab · Jan 25, 2020, 3:44 PM

@JKnott Ah good to know. It's going back to the store today. any recommendations for a similar capacity switch that does VLAN properly? I need at least 9 ports and 4 of them POE. I also don't want a rack mount type switch due to space. How are the netgear managed switches?

johnpoz · Jan 25, 2020, 3:49 PM

poe or poe+ how many watts?

JKnott · Jan 25, 2020, 3:51 PM

@airlab

A few months ago, I bought a Cisco SG200-08, but it's only 8 ports.

Essentially, you can buy whatever switch you want, as I've only heard of this problem with TP-Link.

johnpoz · Jan 25, 2020, 4:01 PM

Yeah I have only ever seen this problem on the tplink stuff.. I have some lowend netgear and dlinks - and they handle vlans correctly. So really anything should work - seems your requirement of poe would be the limiting factor in your choices.

There are a lot of different things involved with poe.. is it af or at, or the new bt.. Or are your devices using some odd passive setting with specific voltage requirements. And then what is the total wattage your poe devices are going to require... its possible that switch might have your needed 4 poe ports, but not be able to supply the total wattage across these ports that your devices want/need.

What are the specific device(s) your wanting to use poe for?

airlab · Jan 25, 2020, 4:09 PM

@johnpoz @JKnott

Ah okay so its more that tp link stuff is bad not that in general cheaper switches are bad.
I have 2 unifi AP AC pros which draw 9W max each, a unifi cloud key, and one unifi AP AC lite which maxes out at 6.5W. they use poe af.

johnpoz · Jan 25, 2020, 4:16 PM

@airlab said in Replacing VLAN switch in router on a stick configuration:

AP AC lite

How old is that model, have you validated it what poe it does, they use to sell those with only 24v passive poe support..

Here is some info
https://www.reddit.com/r/Ubiquiti/comments/76l762/8023af_support_for_uap_ac_lite/

airlab · Jan 25, 2020, 4:22 PM

@johnpoz yep it’s the revision with both the weird UniFi Poe and poe af.

johnpoz · Jan 25, 2020, 4:36 PM

Whats the budget for this switch... The 9 port requirement pretty much forces you to a 16 port model.. Which drives the cost up..

All unifi gear for poe, have you looked at the unifi switches?

Can 1 of the ports be via sfp, where you could add a copper sfp or sfp+ if you need copper connection. The UniFi Switch 8 (150W) would meet your needs if 1 of the ports can be sfp. You would have to buy the sfp for what you want the 9th port for - 200$

Same sort of thing but dlink DGS-1100-10MP, but again the 9th port would be sfp. Same sort of 200$ price range.

cheaper option the netgear JGS516PE, less total poe power but more ports. The GS108PE would be cheaper, but only 8 ports total... And only 53 watts, which should be enough but its 1 port short..

airlab · Jan 25, 2020, 4:40 PM

@johnpoz $200 is the max budget. I actually have a UniFi switch 8 150W that I was using in my USG setup but I want to get away from UniFi as much as possible. I have had increasingly bad experience in the last year with unstable firmware on the UniFi gateway and switches. I was running year old FW in them since they keep locking up with the newer versions.

The APs have been solid.

I’m going to look into the other ones you mentioned.

johnpoz · Jan 25, 2020, 4:43 PM

Yeah I have heard some horror stories with their switches, shame too since their AP are so freaking solid ;) I had a usg while my sg4860 was on back order when I upped my isp speed and needed something with more umph.. Couldn't get it off my network fast enough... While it could push the packet for my speed, it was not even close to ease of use compared to pfsense..

Now that usgp3 just sits on my self as an emergency spare, and a toy to play with run into some usg question I get interested with helping with ;)

airlab · Jan 25, 2020, 5:33 PM

@johnpoz yea the APs were how I got sucked into the ecosystem. I should have stopped at the APs. I will admit UniFi is great for managing my parents network from across the country so I’ll give them that.

@JKnott & @johnpoz Thanks for the help and discussion!