PFS & cisco & esxi with vlans
I got a little over my head with this one so I hope someone might help me understand what I should be doing. I've never used VLANs before and either than I know they exist and that they use tags… that's about it. But I'd like to make the network a bit more secure than it is now since everything is on the same network and everybody has got access to everything else. Not the most optimal solution I guess?
I've got an VMW ESXi with a guest PFS 1.2.2 the machine has two network cards one for the WAN and one for the LAN.
Anyway I get my internet connection from a Cisco Catalyst 3560 (and I've got 6 IPs to use, but this one is not he problem) on the LAN side the server is connected to a cisco srw248g4p.
Now I've setup PFS with a few vlans (vlan1:office,vlan2:wifi,vlan3:classrooms,vlan4:tel) and I've done the same on the cisco assigned a vlan to a group of ports (for office pcs, classroom pcs, wifi and telephony). The ports on the cisco get grouped fine. If I plug my pc into one of the grouped ports I don't get an IP from the dhcp of the LAN but I don't get an IP from the vlan dhcp (dhcp is setup). It's like there were no vlans configured on the PFS.
Would somebody be so kind to walk mi through how to set up a vlan on PFS so that it will work with the one on the cisco?
Otherwise the only solution I can think of is to get another pc with a bunch of network cards and set PFS there and have a cable going to each one of the groups on the cisco. But I think it can be done without drastic measures like this. Right? And I hope I don't get to use this one.
Thank you very much for your help
Well I've tried to re configure the vlans again but there was no effect.
I'm now going to install a PFS on a separate machine and try to use it as a gateway. Setup the vlans and try it that way.
Hope it works.
Anyone have any ideas?
I don't have any Cisco or VMWARE experience, so at the very least I probably don't have the correct terminology.
The Cisco port connected to the pfSense LAN NIC is configured as a "trunk" port or has VLAN tagging enabled?
pfSense has exclusive access to the two NICs? If not, how does the VMWARE hypervisor know what traffic on the LAN NIC is to go to the pfSense guest?
I've just tried the same configuration but eliminated the VMware using a separate PC for the PFS.
I setup a PFS with one NIC going to WAN and the LAN going to the Cisco. I setup 2 test VLANs on the LAN NIC and configured connected the LAN cable to a trunked port on the switch. I configured some ports on the switch to be on one VLAN and other ports to the other VLAN. I also configured the firewall and dhcp for each VLAN. The ports seem to group together on the switch fine all tagged ports seem to see each other and are on the VLANs that their supposed to be. But the switch just isn't seeing the VLANs set on the PFS.
I don't know is there something else I should setup. I'll go over it again-
1. I setup PFS LAN and VLAN1(tag:7), VLAN2(tag:5), with dhcp(lan:192.168.0.100-200, vlan1:192.168.2.100-200, vlan2:192.168.3.100-200) set for VLANs,
2. Connected the LAN cable to the switch,
3. setup switch
a. set port that LAN is connected to to TRUNK
b. set ports 10-20 to VLAN tag 5 and ports 1-5 to VLAN tag 7 and apply settings
4. connect my PC to any port between prot 10-20 and I should get an IP somthing like 192.168.2.* and if I connect to ports 1-5 I should get 192.168.3.*.
But I don't all that happens is I get an DHCP error witch means that there is no DHCP on that network and if I setup the IP manually I still can't reach the PFS.
Currently I have no more ideas accept to get a bunch of network cards for the PFS and connect each one to the VLAN on the switch.
wallabybob -> the WAN nic is exclusiv to PFS but the LAN is shared with other virtual systems running on the VM so it is a virtual switch.
I don't know I've tried to play around with the setting but keep getting stuck with the same result as above.
Am I setting something up wrong?
Thanks for the help.
When your PC is issuing DHCP requests and with pfSense running on your standalone PC
do you see any traffic on the LAN interface?
Do you see any traffic on the VLAN interfaces?
Is there anything "interesting" reported in the pfSense logs (Firewall, DHCP, system)?
no traffic whatsoever. The only thing that happens is the PC tries to reach the DHCP on the VLANed ports. On the not VLANed ports the PC gets an Ip form the LAN DHCP server.
And I can't find any strange logs on the PFS as if everything was working fine and no one has connected to the VLANs.
I mean if the ports and the PFS interfaces have the same VALN ID tag they should see each other if they are connected via lan cable right?
Sounds like you haven't configured the Cisco correctly, in particular the port to the pfSense LAN interface. Does it belong to VLANs 5 and 7? Is it configured to send VLAN tags?
Do you need to restart the Cisco after changing the configuration?
I've set the Cisco ports a few to tag 7 and some others to tag 5. The port that is connected to the PFS si not one of these ports but a separate one. It is set to trunk but if I set it to general I can only set ti to one VLAN.
I haven't restarted the Cisco I can give it a try but I'm reluctant to restart the thing too often because it acts strange for a few minutes after it has been restarted.
I'll give it a try and see what happens. I'll post the result.
Thanks & bye
Nope no luck.
Same thing, on the LAN i get an IP on the VLANs nothing.
And I did everything justr as I described it a few posts above.
Any other ideas?
On the information you have given me it looks as if the Cisco is not forwarding traffic with VLAN tags to your pfSense LAN interface. Suggest you call Cisco support or read the manual (may be available online) or look for Cisco online VLAN troublshooting or …
If you search the pfSense forums you might some help with the configuration of VLANs. Advanced search for "cisco vlan configuration" might turn up something.
I consulted a college that has a lot of experience with Ciscos and he told me that I'm on the right track.
So I did everything again. The only thing I changed was that I put the VLANs on a different NIC than LAN and didn't set a non VLAN on that NIC (apparently this way I would not mix tagged and non-tagged traffic). I set the switch up so that the VLAN port was trunk and then tagged all the ports I wanted to one VLAN or another. Rebooted anything that could be rebooted.
Now this is where everything is supposed to fall into place and start working but it didn't same result as last few times I tried. Nothing like no VLANs were configured.
I don't know the only thing my college could think of is maybe something on PFS was restricting the VLANs or Cisco and PFS don't communicate with the same protocol for VLANs.
Anyone have any more ideas? (Just as a contingency I ordered another server with a few NICs so that if I can't figure out these VLANs I can still use physical NICs)
Thanks for the help and bye.
you need to make sure you have VLANs working in ESX/ESXi first
add a VLAN id/tag to your current vswitches with VLAN 4095 (this VLAN tag will allow any and all VLAN tag to talk to the Virtual Machine located on that vswitch)
yes I've tried that too. I set the VLAN on the virtual switch to 4095 but no effect. So now I'm trying on a non VM machine.
I just took a PC and put 3 NICs in it and am now trying on that one but it doesn't work either way.
I keep setting up VLANs on both the PFS and the Cisco but neither way any VLANs still work.
I don't know I'm a little confused if there is something else that I should be aware off. I read that if there are VLANs configured on an interface that it is automatically set to trunk and that if I've got VLANs on that NIC no physical network should be configured on that NIC. So I've got one NIC only for VLANs.
The switch is configured as far as I know correctly and it still isn't working.
Thanks for the help and any input is welcome.
I don't use Cisco switches, I have HP ProCurves, but I can tell you how I did it with my network.
In ProCurves, there are two different ways to assign ports to VLANs. One is called "untagged." Basically you create a VLAN and any device that plugs into a port that's assigned untagged to that VLAN will be able to communicate with others on the same port group. The devices themselves don't need to be VLAN aware (i.e. no VLAN IDs sent in the ethernet frame). It just basically carves up certain ports into a mini switch.
The other way is to assign ports to tagged VLANs. This is what you are attempting I believe. As someone mentioned earlier, I think this is called "VLAN trunking" in Cisco terminology, but I have no experience with it so I'm not certain.
Say I have two VLANs that I want one interface on pfSense and one interface on my VMware server to share.
I create two VLANs:
ID: 10 NAME: TEST1
ID: 20 NAME: TEST2
On pfSense, I would create a VLAN with the correct ID in Interfaces->Assign->VLANs. Adding the VLAN is self explanatory. I choose the parent interface "bce2" and the ID "10" then I name it "TEST1". It's very important that the parent interface (bce2) isn't assigned to any other networks already by pfSense. The parent interface to any VLAN should remain unassigned. After the VLANs are created, assign the interface in Interfaces->Assign and specify the network information you want like IP/subnet etc.
On the switch, I would assign "Tagged" to ports 1 and 2 for both VLAN IDs 10 and 20. This is where you should look for "trunking" on Cisco.
On VMware, I create a Virtual Machine network bound to the interface that will plug into the switch port 1. This virtual switch I will assign to the correct ID. So I create two virtual switches bound to that interface, one with VLAN ID 10 and one with VLAN ID 20.
So VMware is plugged into switch port 1, and pfSense is plugged into switch port 2. Both ports are assigned on the switch to use VLAN IDs 10 and 20.
That should be pretty much it. Finally, create a VM, attach it to the corresponding switch with the VLAN ID you want in VMware, and assign the VM an IP address in the same subnet as the IP you assigned pfSense for that VLAN.
FWIW this took me about a week to work out on my network since I had little experience with VLANs, but ultimately it seems to work quite well. Good luck.
after some playing around I finally got the thing working but not on the VMWare but on a standalone pc.
Here's how I did it.
I set the VLANs up on PFS(vlan2,vlan3,vlan4,vlan5) on a separate adapter(sis2) from LAN(sis1) and didn't set any network on sis2 accept for the VLANs.
I connected the sis2 port to a port on the switch and set the port to trunk and tagged the port in every VLAN I wanted to configure to switch ports. Then I set all the ports I wanted in one VLAN on the switch to access mode and untagged on a specific VLAN for example vlan2 ports 1-5 and vlan3 ports 6-10, vlan4 ports 11-15.
Then connected the other two switches via trunk port with the VLANs tagged on the mutual ports setup the same VLANs as on the first switch and configured some ports to one VLAN or another as on the first switch.
When I plugged my PC into ports on any switch that were configured for example on vlan2 I got an IP address from the vlan2 DHCP server. If I plugged it into a vlan5 port I got the corresponding IP address.
So everything seems to be working as i should accept for one thing.
The moment I enabled the VLAN interfaces on the PFS I set them an IP address and DHCP server in the appropriate ranges (vlan2-192.168.2.0/24,vlan3-192.168.3.0/24,vlan4-192.168.4.0/24,ect) and went to the Rules menu and duplicated the LAN rule for all the other interfaces. So everything is setup just as on LAN.
But I can't seem to connect to the internet on any other network either than LAN. I get the DHCP IP and I can see all the other devices on my network but I can't access the internet.
I can't ping an address or IP in both cases I get a time out.
Any ideas on this wired problem?
Thanks all for the help and jwbrown77 thanks you got me thinking in the right direction and connecting the VMWare comes next.
pfSense is a firewall. It doesn't know what access you want enabled for the OPTx networks so it blocks everything until you tell it otherwise. What LAN rule did you duplicate? Did you need to change some addresses in the LAN rules to make them appropriate to the OPTx networks?
yes I changed the interface on witch it was residing from LAN to Tel(the vlan5 interface) and source from LAN Subnet to Tel Subnet. But it didn't work for a while. I tried rebooting and now it's working fine.
So thanks again for the clarification.
Now the only thing I've got to tackle is how to connect the other WAN connections but I'll leave that for tomorrow and think about it a little more.
Oh before I forget if I set the VLAN ID on the ESXi switch to 4095 that means it's set to trunk. Right?
Thanks for the help.
I have this setup in my lab. Can you please post a network diagram (visio, etc) of what you are trying to accomplish so I can compare your configuration with mine? I am using Cisco 3750E switches connected to a pair of ESX servers and a pair of pfSense firewalls.
I haven't restarted the Cisco I can give it a try but I'm reluctant to restart the thing too often
well I finally got everything working (regarding the VLANs) and I was also able to determine what went wrong.
I'm now running a dedicated machine for PFS and ESX is on its own.
My first mistake was that I wasn't sure what access or general port was on the switch and my second was that first I didn't set the port that contained the VLANs as trunk. So after I created VLANs on PFS and attached them to the OPT1 interface (interface is used only for vlans) and set the port this cable was connected to on the switch as trunk. I precoded to tag this port on every VLAN I needed on the switch and added access ports to the appropriate VLAN. It started working right away without any restarts or reboots of PFS or the switches.
Now the ESX is a bit of a different story. For the VMs on the ESX I created a virtual switch, each with corresponding VLAN tags and connected it via trunk to the switch. Then I added the VMs to the appropriate virtual switch and changed the IPs on them and everything started to work as it should. I'm still not sure if I could have set the virtual switch to 4095 and setup VLANs on each VM separately, but since it's working it doesn't make much sense to start medaling with it now. :)
Anyway thank you all for your help and I hope that anyone with similar problems might benefit from this information here. I'm also attaching a diagram of my network topology for reference (sorry but it's not very good, but I think it illustrates the network).
By the way for example if I have setup OPT2 as a second LAN and it is working what happens if I attach a few VLANs to the same interface as OPT2 and then set the port on the switch as trunk. Will OPT2 still work and fall into the default vlan on the switch 1 and all other VLANs to tagged the appropriate VLANs. Would this work? It works the ESX any non tagged traffic falls into vlan1 on the switch. Or is it a better idea to leave only VLANs on the PFS nic without the non VLAN traffic? And when you attach VLANs to a nic in PFS is that nic automatically marked as trunk?