Custom VLAN setup - help needed!

kcleveland

Hey All-

This is my first post in this forum, so please go easy on me ;)

I'll try to keep this as short and to the point as I can.

I recently loaded up PFsense on VMware ESXi 5 - I used it to replace my current router - I gave it two interfaces:

PPPoE on the WAN side (em0) that connects to my DSL modem
192.168.0.0/24 on the LAN side (em1) (using 192.168.0.1 as the LAN IP of the pfsense, so that is the default gateway address for all of the clients on my network).

Piece of cake - after ripping out the old router and switching over to the pfsense box everything works great!

But that was only the first step of many plans I have for the pfsense box.

Now here is where I am starting to confuse myself - I want to setup VLAN's (my first several attempts resulted in complete failure). The core switch of my network is a Nortel ERS4548GT-PWR. I have a lot of experience with these switches, so setting up VLAN's and Q-trunks is very easy for me. I also have a solid understanding of VLAN's - at least I though so until I tried to implement them on pfsense (much different than anything I'm used to!).

I want to separate my network out into 3 VLAN's (maybe 4 - I'll get to that).

I want to create VLAN10 for wireless clients (iPhone, Kindle, wireless laptop, etc.). VLAN10 will also be where my access point resides (a Cisco Aironet 1142N).

Then I want to create VLAN20 for "media stuff" (both of my Apple Tv's, the iTunes server, NAS, etc.)

Lastly, I want VLAN30 for the rest of the LAN (wired PC's, etc.)

They will be addressed as follows:
VLAN10: 192.168.10.0/24
VLAN20: 192.168.20.0/24
VLAN30: 192.168.30.0/24

This brings me to my first question: I read that I may need to create a fourth VLAN to share the same DSL connection between these three VLAN's? Is this true? If so, what would the steps be?

Next question: maybe I don't need a fourth VLAN, because I also read that by default pfsense will NAT all outbound traffic to the public WAN interface IP. Is this true? This seems like a better way than creating a fourth VLAN.

So from here, I know that I need to create VLAN's 10, 20, and 30 on my Nortel switch, and create a 802.1q trunk from the switch to the pfsense box that will allow these three VLAN's to travel across it. Easy enough.

I also (think I) know that I need to create VLAN's 10, 20, and 30 on em1 in pfsense, but actually assign them each to their own OPT interface instead of em1. Still easy.

Now here's is where I am utlimately stuck: my LAN interface (em1) has an IP of 192.168.0.1 - it is capable of serving the 192.168.0.0/24 network. Obviously, this is not the network that the VLAN's are in. How do I setup my LAN interface to support these 3 VLAN'd subnets? Obviously I need pfsense to route between the 3 VLAN's for me (after all, going between VLAN's is the router part of things), but I can't figure out how to set it up in pfsense properly.

In addition, I have set the VLAN ID of the NIC's in ESXi to 4095 (ALL) - I was told that this will allow a tagged packet to flow from the pfsense vm, to the vmware virtual switch, and out the physical NIC onto the network (and vice-versa) without stripping the tag from the packet. Can someone tell me if I am thinking correctly here or if I should do this another way?

Here is a diagram of what I am trying to do:

Thank you for any help!

marcelloc

As you are using vmware, you need to configure virtual switch to allow these trunks or create on virtual switch for each vlan you wan to apply.

For each virtual switch you create you add an network interface on pfsense virtual machine. This way you will not need to configure vlans on pfsense.

The idea of plug dsl/cable modem on switch is to assing a vlan for it and for example configure two pfsense with carp sharing the same modem.

kcleveland

I believe by setting the ESXi virtual switch to VLAN ID 4095 (All) I have allowed these trunks into the PFsense VM - are you saying that I need to a separate NIC port for each VLAN? Doesn't that defeat the purpose of an 802.1q trunk?

You also say "this way you don't have to create VLAN's in pfsense" - but I DO want to create VLAN's in pfsense, and need guidance as to how.

Also, this is only one part of at least 5 questions I asked above - hopefully one of the gurus can step in and provide some help here.

marcelloc

Of course you can setup vlans if you want, I just asked the first point that is between your switch and pfsense.

The best way to setup vlans(on my opinion) is to assign it via console(in your case vmware console too), so you will not loose access to interface you are configuring.
Follow instructions from 1) Assign Interfaces to assign what vlans id you will need for selected interface.
the next step is to configure basic access to gui using assigned vlans to WAN/LAN

After you got vlans assigned to wan/lan, go on web interface to setup the other vlans on interface -> assign

the result will be something like this:

LAN (lan) -> bce1_vlan100 -> 172.16.x.10
WAN (wan) -> bce0_vlan202 -> 201.x.x.x
opt1 (opt1) -> bce0_vlan203 -> 10.255.X.10
opt2 (opt2) -> bce0_vlan204 -> 201.X.X.X
opt3 (opt3) -> bce0_vlan205 -> 192.168.X.100
opt4 (opt4) -> bce0_vlan207 -> 172.X.10.76
opt5 (opt5) -> bce0_vlan208 -> 172.X.5.2
opt6 (opt6) -> bce0_vlan209 -> 10.10.X.X

I read that I may need to create a fourth VLAN to share the same DSL connection between these three VLAN's? Is this true? If so, what would the steps be?

You do not need to do this, if you want you can do but it is not mandatory.
The share will be using firewall as gateway instead of vlan routing.

maybe I don't need a fourth VLAN, because I also read that by default pfsense will NAT all outbound traffic to the public WAN interface IP. Is this true? This seems like a better way than creating a fourth VLAN.

while using vlans, it's recommended to disable default automatic nat (firewall -> nat -> outbound) and add rules just for outgoing traffic that you need to nat(for example wan). In default nat setup, every communication between vlans will match nat rules.

So from here, I know that I need to create VLAN's 10, 20, and 30 on my Nortel switch, and create a 802.1q trunk from the switch to the pfsense box that will allow these three VLAN's to travel across it. Easy enough.

If your vmware virtual switch is able to forward these tags to pfsense it will be enough. If not, you will need to create these vlans on vmware and assing an interface for each virutal switch

Now here's is where I am utlimately stuck: my LAN interface (em1) has an IP of 192.168.0.1 - it is capable of serving the 192.168.0.0/24 network. Obviously, this is not the network that the VLAN's are in. How do I setup my LAN interface to support these 3 VLAN'd subnets? Obviously I need pfsense to route between the 3 VLAN's for me (after all, going between VLAN's is the router part of things), but I can't figure out how to set it up in pfsense properly.

After you setup vlans and interfaces at pfsense, each vlan will have pfsense as gateway and firewall rules on each tab(firewall -> rules) will define what access you want to allow between vlans and/or internet.
In this case, em1 will need to be configured with all netwoks tagged in vlans. Untagged vlans sharing the same tagged interface will freak out your pfsense(and many other SO).
lan will be for example em1_vlan1

In addition, I have set the VLAN ID of the NIC's in ESXi to 4095 (ALL) - I was told that this will allow a tagged packet to flow from the pfsense vm, to the vmware virtual switch, and out the physical NIC onto the network (and vice-versa) without stripping the tag from the packet. Can someone tell me if I am thinking correctly here or if I should do this another way?

that's what this article from http://kb.vmware.com tells, so it should work.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004074

kcleveland

Thanks marcelloc, let me digest all of this and post back :)

kcleveland

So my first question about your post is:

LAN (lan) -> bce1_vlan100 -> 172.16.x.10
WAN (wan) -> bce0_vlan202 -> 201.x.x.x
opt1 (opt1) -> bce0_vlan203 -> 10.255.X.10
opt2 (opt2) -> bce0_vlan204 -> 201.X.X.X
opt3 (opt3) -> bce0_vlan205 -> 192.168.X.100
opt4 (opt4) -> bce0_vlan207 -> 172.X.10.76
opt5 (opt5) -> bce0_vlan208 -> 172.X.5.2
opt6 (opt6) -> bce0_vlan209 -> 10.10.X.X

It looks like you have the VLAN's on the same physical NIC as the WAN interface - is this correct? I was tying my VLAN's to the LAN interface's NIC, like such:

WAN -> em0 -> PPPoE
LAN -> em1 -> 192.168.0.1
VLAN10(opt1) -> em1_vlan10 -> 192.168.10.1
VLAN20(opt2) -> em1_vlan20 -> 192.168.20.1
VLAN30(opt3) -> em1_vlan30 -> 192.168.30.1

Should I be doing it the other way?

Okay, second question: you say "each vlan will have pfsense as a gateway" - which IP on pfsense are you referring to? Won't each VLAN have a different default gateway? i.e. VLAN10's subnet is 192.168.10.0/24 - wouldn't it's gateway be 192.168.10.1? And 192.168.20.1 for VLAN20?

Third question: how do I setup the routing between the 3 VLAN'd subnets? I'm not sure how my LAN interface should be setup to accomodate these VLAN'd subnets?

marcelloc

@kcleveland:

It looks like you have the VLAN's on the same physical NIC as the WAN interface - is this correct? I was tying my VLAN's to the LAN interface's NIC, like such:

WAN -> em0 -> PPPoE
LAN -> em1 -> 192.168.0.1
VLAN10(opt1) -> em1_vlan10 -> 192.168.10.1
VLAN20(opt2) -> em1_vlan20 -> 192.168.20.1
VLAN30(opt3) -> em1_vlan30 -> 192.168.30.1

Should I be doing it the other way?

It will not work with em1 having tagged and untagged networks.
You will need to configure lan as a vlan too
LAN -> em1_vlan1 -> 192.168.0.1 (if lan is on default switch vlan)
You can use any interface to tag vlans, when you tag wan will not be em0 but em0_vlan40 for example. In some clients I use only one interface to get a backup hardware easier.
To tag wan, connect modem to a port on your switch with vlan40 untagged and then get it tagged on pfsense through trunk port that will have all vlans.
WAN -> em1_vlan40 -> PPPoE

@kcleveland:

Okay, second question: you say "each vlan will have pfsense as a gateway" - which IP on pfsense are you referring to? Won't each VLAN have a different default gateway?

you will have to setup clients gateway to pfsense ip. For example, vlan20 has 192.168.20.1 assigned to pfsense so any host on 192.168.20.x will have 192.168.20.1 as the default gateway.

@kcleveland:

i.e. VLAN10's subnet is 192.168.10.0/24 - wouldn't it's gateway be 192.168.10.1? And 192.168.20.1 for VLAN20?

Yes. Each vlan will have pfSense corresponding ip as a gateway.

@kcleveland:

Third question: how do I setup the routing between the 3 VLAN'd subnets?

The routing will happen if you change outbound nat to manual and create rules to allow traffic between vlans.
Rules are created where traffic begins.
If you want to allow traffic from vlan20 to lan you create a rule on vlan20 firewall rule tab(opt2)
To deny lan access to vlan20, you create a rule on lan that denies access to vlan20(opt2) network range and place it before any other rule that allow traffic to any ip for example.

@kcleveland:

I'm not sure how my LAN interface should be setup to accomodate these VLAN'd subnets?

The same way i told you on first answer. You are tagging all vlans on this port, so LAN will be on default vlan with id 1(check it on your switch)
LAN -> em1_vlan1 -> 192.168.0.1

stephenw10

Reading through this there seems to be quite a bit of confusion. ;)

You shouldn't need to switch to manual outbound NAT in my opinion.

It's important to make the distinction between Network Ports which are either physical ethernet interfaces (em0, em1 etc) or virtual interfaces (vlan1, pppoe1 etc) and pfSense Interfaces (WAN, LAN, OPT1 etc).

There's nothing special about any of pfSenses interfaces, they are just names, and they can be assigned any of the network ports. pfSense distinguishes only between interfaces that have a gateway (usually external) and those that don't (usually internal).

Hmm, not sure I'm reducing the confusion. ::)

Steve

marcelloc

Hi steve,

Without disabling outbound nat, all traffic between vlans will have pfsense ip as source.
This may not be a good choice in my opinion.

cmb

@marcelloc:

Without disabling outbound nat, all traffic between vlans will have pfsense ip as source.

That's not true, only if you have a gateway chosen under Interfaces>(VLAN interface name) page will the source be NATed. That should only be done on Internet connections, so it will never happen on VLANs unless they're configured as an Internet connection.

marcelloc

This maybe also something started on 2.0 too because on 1.2.3 I had always to disable automatic nat when using vlans.

One more note for me.

Sorry for that.

cmb

1.2.3 was exactly the same there, if you don't have a gateway specified on an interface, it does not NAT out of that interface. If you do have a gateway, or it's set to a dynamic interface type (DHCP, PPPoE, etc.) it does NAT out of that interface since almost always that's going to be an Internet connection, where NAT is required.

kcleveland

Aah, ok! I have had some success now thanks to you marcelloc (but I'm not finished with you yet ;)

I also see others have jumped in - thank you so much for sharing!

I think the whole em1 -> vlan1 192.168.0.1 was key here - thank you for explaining that in a different way for me.

I went ahead and moved everything into its respective VLAN's, setup the trunk, and added the rules and this time I've got it working :)

My next question is: Now that everything is working, I want to change the IP address of the LAN interface on pfsense to be 192.168.5.1 instead of 192.168.0.1 - this way I can start to setup openVPN without worrying about duplicating IP's. Can I just go into the interface, change the IP and be good to? Or will it totally trash my setup if I try to change it? I'm not using the default VLAN on the switch for anything anymore (I changed the mgmt VLAN to VLAN 30 so only one of my wired mgmt PC's can get to it), so I think that changing pfsense's LAN IP to 192.168.5.1 should work, as pfsense is literally the only thing left with an IP in 192.168.0.x subnet, a subnet which really doesn't even exist anymore - thoughts?

Another question: It looks like this may still be up for debate, but I left the outbound NAT at default, and everything is working. You had suggested disabling it and creating rules instead- I just have rules right now that allow everything, but outbound NAT is still default - can I just turn NAT to manual and everything will still work (and I can slowly lock everything down with better rules)? Or will I need to re-create specific rules that would mimic what the default outbound NAT does?
From what I see below, I did not specify a gateway under the interface, so I'm guessing that because my WAN connection is PPPoE any traffic that hits it (i.e. any outbound traffic) will get NAT'ed to the WAN IP - thoughts?

Thanks again for all of your help so far, you guys rock.

marcelloc

@kcleveland:

My next question is: Now that everything is working, I want to change the IP address of the LAN interface on pfsense to be 192.168.5.1 instead of 192.168.0.1 - this way I can start to setup openVPN without worrying about duplicating IP's. Can I just go into the interface, change the IP and be good to? Or will it totally trash my setup if I try to change it? I'm not using the default VLAN on the switch for anything anymore (I changed the mgmt VLAN to VLAN 30 so only one of my wired mgmt PC's can get to it), so I think that changing pfsense's LAN IP to 192.168.5.1 should work, as pfsense is literally the only thing left with an IP in 192.168.0.x subnet, a subnet which really doesn't even exist anymore - thoughts?

If you are on the right vlan, you can use any network range you want. :)

@kcleveland:

Another question: It looks like this may still be up for debate, but I left the outbound NAT at default, and everything is working. You had suggested disabling it and creating rules instead- I just have rules right now that allow everything, but outbound NAT is still default - can I just turn NAT to manual and everything will still work (and I can slowly lock everything down with better rules)? Or will I need to re-create specific rules that would mimic what the default outbound NAT does?
From what I see below, I did not specify a gateway under the interface, so I'm guessing that because my WAN connection is PPPoE any traffic that hits it (i.e. any outbound traffic) will get NAT'ed to the WAN IP - thoughts?

Keep nat on automatic just like cmb said. It will work.

kcleveland

Thanks again marcelloc.

Ok, I've accomplished everything I've set out to with pfsense so far, but I have one last question/issue (it changes gears a bit).

Everything on my network is now good to go, except that I separated my wireless into a vlan and my media into a vlan - the one issue that caused is that my wife has an app on her iPhone that streams movies via upnp/dnla from my NAS to her iPhone - the NAS is now in a separate VLAN from her iPhone (NAS in media vlan and iPhone in wireless VLAN), so the app cannot discover the NAS anymore. I need to get this one last thing working for her before I can declare this deployment complete - from a starting phase at least :)

What would be the best way to get the iPhone to discover the NAS like it is in the same L2 network (even though it isn't)? Could I setup a port-forwarding rule that forwards any traffic coming in on the wireless vlan on the upnp/dnla ports to the media vlan? Thoughts?

Thanks!

marcelloc

This last answer I do not have. :(

Take a look on services->UPnP & NAT-PMP Settings to see if it helps.

kcleveland

So while waiting I tried a couple of things - below is a screenshot of the port forwarding rule I created, but of course it does not work (and I'm sure there is an obvious reason why, as this is my first time):

Can you take a look an help me out? I'm trying to go from "iphone in vlan10 –> NAS in vlan20" trying to port forward the upnp ports (1900 UDP, 2869 TCP - is this even right?) from the iphone to the NAS, so that when the iphone app does a discovery the upnp traffic gets forwarded to the media vlan (specifically the NAS box - 192.168.20.54) and the iphone can see the NAS box and stream upnp from it. Thoughts?

marcelloc

try to change source port to any on this nat rule.

kcleveland

changed the source port to any - still no luck.

I tried a couple of other things with no luck as well.

Google isn't being very helpful in the matter either (everything is about how to hook up an xbox to pfsense, and it is setup differently than this would be).

Any other ideas? Should I start another thread on this topic of upnp forwarding?

Thanks!

stephenw10

This is an interesting problem. :)
If you imagine this working as a traditional WAN-LAN port forward you should have the destination address as 'VLAN10 address'. However if you do that with so many ports a lot of stuff might stop working.
You could perhaps add a virtual IP address to the VLAN10 interface and forward ports with that as destination.
Hmmm.

Can you not simply set the address of the server in the iphone app.?

Steve

Edit: I think you have selected a range of ports 1900 to 2869 where as upnp uses only two ports, 1900 and 2869.
It's hard to know how this might work. You would usually have NAT running between WAN and LAN but that isn't the case in your situation.
Port forwarding 1900 and 2869 on the VLAN10 interface would prevent upnp working to open NAT holes to WAN for other services. Maybe not a problem.