I can't get VLANs to work / No DHCP
-
@johnpoz I was just thinking that. I'm going to delete EM2 and put it on EM1. I was following the official documentation from pfSense as best I could, and they had me setting up a second interface - a virtual one. So give me a minute and I'll reconfigure.
-
@robh-0 See my edit with showing tags... If your sending stuff tagged that whatever interface is on the 4095 then your vlans are working... If your not seeing any tags, then you have a problem..
You can also see there is untagged traffic being seen as well.
-
@johnpoz I switched it all over, got rid of em2, put the VLAN on em1, ensured that it is on 4095 and I'm not seeing those tags. I have to go to some conference calls (dang work getting in the way of my fun) but once I can look over my configs on my switches and such, I'll get back. I did try setting up a port group on the switches so that all VLANs were tagged, but that made no difference either.
-
@robh-0 well if your on a vlan id 4095 on yoru port group, and your not seeing any tagged traffic, then there is something wrong before pfsense. Pfsense has no control over what gets sent to its interface..
You should be seeing arps in your vlan for example.. You should be seeing pings to pfsense vlan with the tag, etc.. Does your device in the vlan see the mac of the interface in pfsense vlan interface? when you try and ping it? If not then no it would never send traffic - but you should see the arp requests even if pfsense didn't answer, etc.
If pfsense never sees the arp, then it couldn't answer, if can not answer with its mac - then no you couldn't have any conversations. Or other traffic. If your sniffing on full verbose, and not seeing the tagged traffic since 4095 shouldn't strip any tags, then traffic is not going to pfsense to do anything with.
With vlan ID set at 4095 you are in VGT mode
-
@johnpoz I can't figure this out. I fired up Wireshark on my laptop and things are really weird. If I connect to the default LAN, and ping the pfSense box, that works just fine. If I put my laptop on the VLAN and then try to ping the VLAN interface on pfSense, not only do I not see the pings, but in Wireshark I never see the pings go out from the laptop when I try and ping the pfSense VLAN interface! I have no idea why this is happening, you'd think I'd see the attempts. If I plug another device into the VLAN and ping it from my laptop, it works.
I'm digging in even deeper to see if I can get this figured out over on the Unifi forum, but I'm pretty confused how this could happen. All the devices are on the same switch, so it's not like I have an issue between switches.
I'm kind of bummed because I need these VLANs to work to be able to migrate off of the USG.
-
@robh-0 said in I can't get VLANs to work / No DHCP:
never see the pings go out from the laptop when I try and ping the pfSense VLAN interface
As I stated if you do not get a mac back, then no you can not actual send a ping - you need a mac to be able to send traffic.
If you sniff on your pfsense vm parent vlan interface - and you don't even see the arp, then the traffic is not getting to pfsense - so it can not answer even arp, etc.
Look in your arp table on your machine - do you see a mac for this IP...
You have something blocking it, either your switch config is wrong, or something not setup correctly in esxi.. But if pfsense never sees the traffic - then nothing it can do.. If your switch was sending tagged packets to esxi, and you have the vswitch set to 4095 then you should see the tags at pfsense, even if you don't have anything setup for that tag on the virtual nic in pfsense. Take your switch out of the equation, and connect your pc or laptop direct to this this physical interface you have on the esxi box.. Now set your nic on your interface on this PC/laptop to send tags.. Should be able to do that in the advanced settings on the interface..
-
@johnpoz Good point! My brain was fried at the point I posted that, thank you for being the voice of reason. :) I'll dig into it more today if I can get around to it.
-
@robh-0 @johnpoz I think I'm on to something. Since FreeBSD is not one of the choices for my install, I selected Other Linux 12.4 64 Bit. Now, what this does when you use "other Linux" is it only will allow you to use E2000 NICs, I can't select VMX3. I am going to delete this build yet again and go back and select FreeBSD 12 64 bit, which will then allow me to choose the VMX3 NICs. I think there might be something in the E1000 config that's keeping me from using VLANs. Also, I found some instructions for doing this in ESX 7, and I was missing one critical step. I have to assign the VLANs as port groups, but I do not add those to the VM, and that's why on your example you don't see the added virtual NICs as something you can assign. I'm going out to eat right now, but I'll report back later. With any luck, I may be able to document this and help someone else in the future.
-
@robh-0 If you look at the compatibility you see freebsd and e1000 vs e1000e, etc. and yeah the vmx3 drivers.
I have to assign the VLANs as port groups, but I do not add those to the VM
No not really - went over this real early in the thread, you have 2 options - you can have esxi handle the vlans with port groups on your vswitches. Or you could use vlan ID 4095 which would pass the tags to your VM virtual nic. And the VM has to handle the tags, this is VGT mode, and even linked to it, etc.
I ran pfsense on esxi for many years.. Started with like way back when with vmserver 2 or something.. Before esxi was even a thing. ;)
I loved many of the advantages that running it on vm brings - but there also issues with it, like my whole internet went down any time I had to reboot my VM host.. I don't think I could ever go back to not running pfsense on its own hardware - other than testing/lab sort of thing.. I do run pfsense now on VM under VMM on my synology nas.. But only for playing with..
But yeah part of the reason you have to update your VM host software is full support for your guest OSes as they change..
-
@johnpoz I quit running my DNS servers in VM for those same reasons, if I had to reboot the host, it killed my connectivity. But, I've split out so much of my network now that it really isn't a problem anymore. I also use a Synology NAS and have quite a bit of stuff on it, but I'm no longer fully dependent on any one thing. Right now I have two major drivers to get this moved over, one is that my USG is suddenly going down and needing to be rebooted on a daily basis, the second is because I have more internet connectivity coming and it's simply too big a pain in the butt to do any policy based routing on the Unifi gear.
I'm about to the point that if this does not work today I'm going to bail on it and go dedicated hardware. I ordered another Intel 4 port NIC yesterday and I have a PC I believe will run pfSense well enough for me, so it is currently an option. The PC I have is my old HTPC, based on (here it comes) an old Sandy Bridge i3-2100. Yeah, it is old. I also have a homemade NAS that runs on a 2100T, mostly really just for file storage.
I did take note of your mention of the VGT mode and all of that. I just couldn't put 2 and 2 together.
Here is the video I found. The reason I did not find it before is because I did not search for ESXi 7 use cases. Now that I'm on 7, when I did the search, this came up. He has nice chapter stops in the video, so you can find your way around it pretty easily.
https://www.youtube.com/watch?v=SsaGeXx2qh0
Anyway, here we go, about to wipe it and try once more before I give up and go dedicated hardware.
-
@robh-0 said in I can't get VLANs to work / No DHCP:
too big a pain in the butt to do any policy based routing on the Unifi gear.
To be honest its a PITA to do really most things on unifi stuff, at least compared to how simple stuff is to do on pfsense.
I ran a usgp3 for very short time on my network. While my sg4860 was on back order, and just got new 500/50 internet and the VM pfsense I was running couldn't handle that.. I couldn't get that thing off my network fast enough to be honest. Doing stuff that was simple click in pfsense was like pulling teeth to get working.. And couldn't use their eye candy ids either if wanted to route at full 500..
My son is currently using the usg with a flexHD AP that is managed out of my controller.. He is just basic 1 flat network and only 100/5 so usg can handle such a simple network.. He is uping his network to 300/300 here soon... I might have to get him a little netgate box when that usg dies, I know it can handle the 300/300 if I turn off the ids.. So unless he wants to get fancier with his network (he wont) its 2 tvs and phones and latops, prob just run that usg until it dies.
Let us know how it turns out..
-
@johnpoz I tried to get a Netgate box. They are all on backorder just like most electronics. Heck, I've been trying to get a new refrigerator for 1.5 years and still waiting. Yeah, you can get the little tiny Netgate boxes, but I need multiple WAN ports. I'd rather just build one than buy a box honestly, I need the flexibility.
Because I live way out in the middle of nowhere, we only have a WISP. I'm paying $125 a month for 25/10, and a lot of the time I don't even get that because their sectors are over-subscribed. AT&T recently did some upgrades in the area, and I can get 100/25 over LTE, so I have ordered an unlimited and uncapped SIM that will run me $135 a month, including a static IP so I don't have to bother with CG-NAT. I'm going to keep the WISP, but I'm going to throttle it down to 5/3 for $60 a month. Ah the cost of country living and peace and quiet. :)
-
@robh-0 said in I can't get VLANs to work / No DHCP:
I've been trying to get a new refrigerator for 1.5 years
Yeah - it is crazy times we are living in.. I had both a refrig and stove go out on me few months apart - LUCK have it - they did have stock in our local bestbuy. We didn't have much choice, without having to wait for what could of been forever.. But lucky the models they did have in stock worked for us..
I ordered a new PC back a few months ago, and yeah that took ages - what normally would of been a few days turned into a month something.
As to your internet connection - have you looked into starlink? I would think you could prob get better speeds for less than $135 from them.. I believe its $99 a month and can get speeds of up to 250.. But I think there is a equipment cost upfront, which could take some time to get back if only $35 a month difference in cost.
-
@johnpoz I give up. It still doesn't work. I'm just going to wait for my new NIC and build a physical. It would have been nice to use my VMWare host, but it just won't cooperate. As you said, it's probably better to set it on its own anyway so I'm OK with that.
I'm on the Starlink wait list, but I'll be OK on LTE. I won't have to deal with CG-NAT on the LTE setup.
-
@robh-0 did you take your switch out of the equation? And just connect a box that you know you can send out a tag on - you can view via sniff on the machine that your actually sending a tag..
-
@robh-0 well trying to get esxi 7 running on my old HP N40L ;) I got it up and running after using allowLegacyCPU=true
all was looking good, I was going to be able to run through using vlans.. But ran into a snag where my VM would boot, but whenever I would try to connect to it via console or VMRC it would crash.. So can not get a vm running to test anything with.. Let alone pfsense..
So installing 6.7 now - same no longer cpu supported. But hoping I can get vm running, networking shouldn't be that different, and same sort of settings for vswitches and port groups.. And 6.7 supports freebsd 12..
I won't have any hardware to play with esxi that is somewhat current until I get my new work laptop, then my old laptop I get to keep and can run esxi on that..
-
@robh-0 Ok I had to go out - just got back... So I booted esxi 6.7 update3, and set my allowLegacyCPU=true. This seems seems to be working, and remote console working - so got a ubuntu vm installed. I will have to do some digging into why on version 7, whenever I would try and launch a console it would crash the vm..
Anyhoo - so got ubuntu 20.04.4 server install, min install no gui just console. So I could do some testing real quick
So simple test... I set on my switch port that connects to my esxi host to pass my untagged vlan 9, which is 192.168.9/24 and also pass tagged vlan 2 and vlan 3, which are 192.168.2/24 and 192.168.3/24
Now on the VM that I connected to a port group that is set to vlan 4095, did simple sniff with -e to show the vlans.. Then forced some arps on boxes on those vlans to something that doesn't exist so I could see the arps.. As you see the vm sees the untagged arps on 192.168.9/24 and also when I arp for something on tagged network 2 and 3, it sees those arps as well. So esxi is passing tagged traffic through this port group without stripping it.
Now if I change the port group to be vlan 3, and then have the vm get a new dhcp address via bring the interface down and then back up - you see it gets an address from my vlan 3 dhcpd.
Then I can do the same thing for vlan 2
-
@johnpoz Hey thanks for all of this!
Unfortunately, my father's health took a nosedive over the weekend, so I haven't had time to do anything fun. We are heading up to his place tonight to be there for several days. I'm keeping a link to this so that I can get back to it eventually, but for now it's probably going to be at least a week until I get back to it.
-
@robh-0 no problem - hope your father is ok.. Thanks for chiming in though.. Was wondering where you had gone.
-
@johnpoz The new 4 port Intel NIC just arrived, so when I get back home I'm going to try going physical on that i3-2100 system I told you about. Should be a breeze. If nothing else, I can get that going to cover for my failing USG while I figure out the VM.
