I can't get VLANs to work / No DHCP
-
@johnpoz
I hope you have an Intel NIC in the VMvare Box ....
My little "NUC lookalike" (Acer) has a realtek , and i slipstreamed the "old" realtek (linux subsystem) driver in.Guess what subsystem has been deprecated in 7.x ...
On 6.7 forever ....
/Bingo
-
@bingo600 Its an old Gen 8 HP microserver.. I do believe I had a 2 port HP intel nic in it.. low profile ;)
Have to pull it off the shelf and fire it up.. I had pulled all the disks out of it, but think I have 128GB ssd I could use to test it out with.. enough to get a pfsense up and running on it, from looking real quick, looks like freebsd 12.x was added in 6.7 - so its possible something odd with the old 6.5 and freebsd.. You could quite often get OS that were not officially supported to work.. But always better to be an actual supported guest OS.
-
@robh-0 OK here's the new install:
VLAN assigned to VLAN Tag 108, I still can't ping it from 192.168.108.20. Like I was saying, I know I'm missing something simple here, just can't figure out what. I've also tried sending it a trunk with VLAN108 tagged, and I've tried sending it just 108, untagged
@bingo600 Twin, real Intel Quad Port cards. Everything is working great. I did notice though when I did the upgrade that they said my poor old Xeon E3-1246 v3 may not be supported in future releases.
-
Oh, and I did add the firewall rule back, just forgot to share that.
-
@robh-0 ok if your setting it up on pfsense as tagged, then it needs to be connected to a vswitch/port group that has vlan 4095 ID set so it doesn't strip the tags.
Did you assign anything on this em2? If you have your lan interface on a vswitch/port group with vlan ID 4095 set, then put this vlan on that interface on pfsense. looks like em1
-
@johnpoz This is what I have.
I tried it assigned as shown in the first screenshot in my last page, and a trunk port going to that. Doesn't work. So then I tried native 108 to that port and just using EM2, that doesn't work. And I tried every combination of those I could think of, and saw no change. I'm starting to get to the point that I want to build a non-VM machine for this, but that seems like a waste of my closet space and would add more heat, I really want to use the VM host.
-
@robh-0 and what interface of your virtual pfsense is connected to thse port groups. They have the same mac..
So what specific interface is connected to these. If your going to connect em2 to vlan 108, then there should be a different mac. And this em2 should just be native, no vlan setup on it.
If your going to put your vlan on the interface connected to this 4095 port group. Then your vlan should be on that - em1? since I take that is your lan interface and working..
if your sending tagged traffic then the interface that is in the 4095 should see it and the tag..
Example.. If I sniff on my igb2, I can see tagged traffic.
Set to full verbose level
-
@johnpoz I was just thinking that. I'm going to delete EM2 and put it on EM1. I was following the official documentation from pfSense as best I could, and they had me setting up a second interface - a virtual one. So give me a minute and I'll reconfigure.
-
@robh-0 See my edit with showing tags... If your sending stuff tagged that whatever interface is on the 4095 then your vlans are working... If your not seeing any tags, then you have a problem..
You can also see there is untagged traffic being seen as well.
-
@johnpoz I switched it all over, got rid of em2, put the VLAN on em1, ensured that it is on 4095 and I'm not seeing those tags. I have to go to some conference calls (dang work getting in the way of my fun) but once I can look over my configs on my switches and such, I'll get back. I did try setting up a port group on the switches so that all VLANs were tagged, but that made no difference either.
-
@robh-0 well if your on a vlan id 4095 on yoru port group, and your not seeing any tagged traffic, then there is something wrong before pfsense. Pfsense has no control over what gets sent to its interface..
You should be seeing arps in your vlan for example.. You should be seeing pings to pfsense vlan with the tag, etc.. Does your device in the vlan see the mac of the interface in pfsense vlan interface? when you try and ping it? If not then no it would never send traffic - but you should see the arp requests even if pfsense didn't answer, etc.
If pfsense never sees the arp, then it couldn't answer, if can not answer with its mac - then no you couldn't have any conversations. Or other traffic. If your sniffing on full verbose, and not seeing the tagged traffic since 4095 shouldn't strip any tags, then traffic is not going to pfsense to do anything with.
With vlan ID set at 4095 you are in VGT mode
-
@johnpoz I can't figure this out. I fired up Wireshark on my laptop and things are really weird. If I connect to the default LAN, and ping the pfSense box, that works just fine. If I put my laptop on the VLAN and then try to ping the VLAN interface on pfSense, not only do I not see the pings, but in Wireshark I never see the pings go out from the laptop when I try and ping the pfSense VLAN interface! I have no idea why this is happening, you'd think I'd see the attempts. If I plug another device into the VLAN and ping it from my laptop, it works.
I'm digging in even deeper to see if I can get this figured out over on the Unifi forum, but I'm pretty confused how this could happen. All the devices are on the same switch, so it's not like I have an issue between switches.
I'm kind of bummed because I need these VLANs to work to be able to migrate off of the USG.
-
@robh-0 said in I can't get VLANs to work / No DHCP:
never see the pings go out from the laptop when I try and ping the pfSense VLAN interface
As I stated if you do not get a mac back, then no you can not actual send a ping - you need a mac to be able to send traffic.
If you sniff on your pfsense vm parent vlan interface - and you don't even see the arp, then the traffic is not getting to pfsense - so it can not answer even arp, etc.
Look in your arp table on your machine - do you see a mac for this IP...
You have something blocking it, either your switch config is wrong, or something not setup correctly in esxi.. But if pfsense never sees the traffic - then nothing it can do.. If your switch was sending tagged packets to esxi, and you have the vswitch set to 4095 then you should see the tags at pfsense, even if you don't have anything setup for that tag on the virtual nic in pfsense. Take your switch out of the equation, and connect your pc or laptop direct to this this physical interface you have on the esxi box.. Now set your nic on your interface on this PC/laptop to send tags.. Should be able to do that in the advanced settings on the interface..
-
@johnpoz Good point! My brain was fried at the point I posted that, thank you for being the voice of reason. :) I'll dig into it more today if I can get around to it.
-
@robh-0 @johnpoz I think I'm on to something. Since FreeBSD is not one of the choices for my install, I selected Other Linux 12.4 64 Bit. Now, what this does when you use "other Linux" is it only will allow you to use E2000 NICs, I can't select VMX3. I am going to delete this build yet again and go back and select FreeBSD 12 64 bit, which will then allow me to choose the VMX3 NICs. I think there might be something in the E1000 config that's keeping me from using VLANs. Also, I found some instructions for doing this in ESX 7, and I was missing one critical step. I have to assign the VLANs as port groups, but I do not add those to the VM, and that's why on your example you don't see the added virtual NICs as something you can assign. I'm going out to eat right now, but I'll report back later. With any luck, I may be able to document this and help someone else in the future.
-
@robh-0 If you look at the compatibility you see freebsd and e1000 vs e1000e, etc. and yeah the vmx3 drivers.
I have to assign the VLANs as port groups, but I do not add those to the VM
No not really - went over this real early in the thread, you have 2 options - you can have esxi handle the vlans with port groups on your vswitches. Or you could use vlan ID 4095 which would pass the tags to your VM virtual nic. And the VM has to handle the tags, this is VGT mode, and even linked to it, etc.
I ran pfsense on esxi for many years.. Started with like way back when with vmserver 2 or something.. Before esxi was even a thing. ;)
I loved many of the advantages that running it on vm brings - but there also issues with it, like my whole internet went down any time I had to reboot my VM host.. I don't think I could ever go back to not running pfsense on its own hardware - other than testing/lab sort of thing.. I do run pfsense now on VM under VMM on my synology nas.. But only for playing with..
But yeah part of the reason you have to update your VM host software is full support for your guest OSes as they change..
-
@johnpoz I quit running my DNS servers in VM for those same reasons, if I had to reboot the host, it killed my connectivity. But, I've split out so much of my network now that it really isn't a problem anymore. I also use a Synology NAS and have quite a bit of stuff on it, but I'm no longer fully dependent on any one thing. Right now I have two major drivers to get this moved over, one is that my USG is suddenly going down and needing to be rebooted on a daily basis, the second is because I have more internet connectivity coming and it's simply too big a pain in the butt to do any policy based routing on the Unifi gear.
I'm about to the point that if this does not work today I'm going to bail on it and go dedicated hardware. I ordered another Intel 4 port NIC yesterday and I have a PC I believe will run pfSense well enough for me, so it is currently an option. The PC I have is my old HTPC, based on (here it comes) an old Sandy Bridge i3-2100. Yeah, it is old. I also have a homemade NAS that runs on a 2100T, mostly really just for file storage.
I did take note of your mention of the VGT mode and all of that. I just couldn't put 2 and 2 together.
Here is the video I found. The reason I did not find it before is because I did not search for ESXi 7 use cases. Now that I'm on 7, when I did the search, this came up. He has nice chapter stops in the video, so you can find your way around it pretty easily.
https://www.youtube.com/watch?v=SsaGeXx2qh0
Anyway, here we go, about to wipe it and try once more before I give up and go dedicated hardware.
-
@robh-0 said in I can't get VLANs to work / No DHCP:
too big a pain in the butt to do any policy based routing on the Unifi gear.
To be honest its a PITA to do really most things on unifi stuff, at least compared to how simple stuff is to do on pfsense.
I ran a usgp3 for very short time on my network. While my sg4860 was on back order, and just got new 500/50 internet and the VM pfsense I was running couldn't handle that.. I couldn't get that thing off my network fast enough to be honest. Doing stuff that was simple click in pfsense was like pulling teeth to get working.. And couldn't use their eye candy ids either if wanted to route at full 500..
My son is currently using the usg with a flexHD AP that is managed out of my controller.. He is just basic 1 flat network and only 100/5 so usg can handle such a simple network.. He is uping his network to 300/300 here soon... I might have to get him a little netgate box when that usg dies, I know it can handle the 300/300 if I turn off the ids.. So unless he wants to get fancier with his network (he wont) its 2 tvs and phones and latops, prob just run that usg until it dies.
Let us know how it turns out..
-
@johnpoz I tried to get a Netgate box. They are all on backorder just like most electronics. Heck, I've been trying to get a new refrigerator for 1.5 years and still waiting. Yeah, you can get the little tiny Netgate boxes, but I need multiple WAN ports. I'd rather just build one than buy a box honestly, I need the flexibility.
Because I live way out in the middle of nowhere, we only have a WISP. I'm paying $125 a month for 25/10, and a lot of the time I don't even get that because their sectors are over-subscribed. AT&T recently did some upgrades in the area, and I can get 100/25 over LTE, so I have ordered an unlimited and uncapped SIM that will run me $135 a month, including a static IP so I don't have to bother with CG-NAT. I'm going to keep the WISP, but I'm going to throttle it down to 5/3 for $60 a month. Ah the cost of country living and peace and quiet. :)
-
@robh-0 said in I can't get VLANs to work / No DHCP:
I've been trying to get a new refrigerator for 1.5 years
Yeah - it is crazy times we are living in.. I had both a refrig and stove go out on me few months apart - LUCK have it - they did have stock in our local bestbuy. We didn't have much choice, without having to wait for what could of been forever.. But lucky the models they did have in stock worked for us..
I ordered a new PC back a few months ago, and yeah that took ages - what normally would of been a few days turned into a month something.
As to your internet connection - have you looked into starlink? I would think you could prob get better speeds for less than $135 from them.. I believe its $99 a month and can get speeds of up to 250.. But I think there is a equipment cost upfront, which could take some time to get back if only $35 a month difference in cost.
