Inter VLAN Routing - Internet Access



  • I have a Netgear GSM7248v2 L2 switch. I was searching for a L3 switch as I needed to stop all inter VLAN traffic from going through pfSense. 5TB backups and file transfers were killing the network. After some research I found that my switch infact can do VLAN routing. Found a netgear manual online and followed the steps and now have a good internal routed vlan environment.

    My problem now is that only the first VLAN is able to browse the internet and the rest 3 VLANs can only browse if the gateway is set to the respective pfSense NIC instead of the switch IP. With this config the data is still routed through pfSense which kinda beats the purpose of routed VLANs.

    Here is my setup.

    Switch
    LAN (VLAN 10) - 10.1.1.1
    VoIP (VLAN 20) - 10.2.1.1
    Video (VLAN 30) - 10.3.1.1
    .. and so on

    The above are IP configured with virtual ports on the switch. Devices on vlan 10 can ping and connect to vlan 20/30 and vice versa. They can all ping the virtual interfaces 10.1.1/1.2/1.3

    on pfSense I have similar network with dedicated (no vlans) NICs for each network

    LAN - 10.1.1.2
    VoIP - 10.2.1.2
    Video - 10.3.1.2
    .. and so on

    Now with normal setup the DHCP on each network assigns the pfSense network interface as the default gateway. This is causing all the devices to route through pfSense to hop on to the other VLAN for file transfers. Tested it multiple times and still see traffic going through pfSense.
    Online netgear documentation says to assign the routing vlan IP as the default gateway, which makes sense. But the moment I set the default gateway in DHCP to 10.1.1.1 or 10.2.1.1 for each network, I am unable to browse or route to the internet. I can't even ping the pfSense IP 10.1.1.2 from the other networks as data is not flowing through the VLAN to pfSense.

    I believe its a simple routing fix I need to do in pfSense to show it the downstream router/switch IP (10.1.1.1/10.2.1.1….) for directing WAN traffic to the specific network since I can ping from pfSense to the devices in all the subnets as well. I need to change the default gateway for each subnet/network their respective switch IP 10.1.1.1/10.2.1.1....

    Could someone please provide step by step direction on how to get this done in pfSense?

    Example..

    Video network.
    IP            10.3.1.xx
    Subnet    255.255.255.0
    Gateway  10.3.1.1 (switch IP)  (internet works when this is set to 10.3.1.2)
    DNS        10.3.1.2 (pfSense DNS)


  • LAYER 8 Global Moderator

    What is your transit network?  If you going to connect pfsense to a downstream router, ie your L3 switch doing routing.. Then pfsense would be connected via a transit network..  Its firewall rules for internet on this transit interface would have to allow your downstream networks, and your outbound nat has to be adjusted to outbound nat your downstream networks.

    I am confused about your vlan IPs given.. Why are you changing the second octet if your using a /24 mask.. Why would you networks not be 10.1.1, 10.1.2, 10.1.3 ??

    Here is how you would do pfsense with a downstream router.

    " rest 3 VLANs can only browse if the gateway is set to the respective pfSense NIC instead of the switch IP. "

    Huh??  Are you sure you have actually created vlans on your switch and routing them, vs just creating 1 big layer 3 network and using different IP ranges??




  • Thanks for the detailed diagram. Much appreciated. I guess I am a bit confused on the implementation. The  network addresses have been this way for many years and I suppose I need to change them now.

    At the moment I think my problem is that I am using netgear document which does not use a transit ip but uses the first subnet as the primary and piggy backs the rest. But the implementation is flawed as it used just netgear gateway for static routes within the same 192.168.x.x network.

    Now my question is, I have 4 separate NICs  (for lan, voip, video and home network) on pfsense for which pfsense did all the routing.if I move all the 4 networks to the L3 switch, should the pfsense now have just one transit network to connect to the switch? And I can delete the networks from pfsense?

    Secondly the transit ip on the netgear switch is an ip of the physical switch..correct? And not a vlan routed ip?


  • LAYER 8 Global Moderator

    Yes you would only have 1 transit network..  Pfsense does not need interfaces in network its not routing/firewalling for.  You should delete these networks off pfsense since you are moving them to your downstream router.

    No this transit network does not need to be vlan tagged, unless your connecting this transit across multiple switches??  Which I doubt.

    "Secondly the transit ip on the netgear switch is an ip of the physical switch..correct? And not a vlan routed ip?"

    As to physical IP?  You mean management IP, or the IP you have assigned to vlan 1?  What IP you use for management has little bearing on anything, normally you do not use the transit IP but sure you could use this to access the switch if you wanted.  It does not need to be tagged, sure it could be your management IP.  The only thing the switch needs to know is that IP of pfsense on this transit network is its gateway..

    Are you using vlan 1 on your switch?  For management or any other ports?  This could be your transit, or you could create another vlan on the switch and use that vlan as your transit.  Just no reason to tag it, unless you want to create a vlan on pfsense, etc.

    What are you looking at from netgear for doing intervlan routing?  Can you provide link and will take a look..



  • Yes with physical I meant the management ip. Typically I set it at a 192.168.x.x address which is on vlan 1. I do not use the management vlan.

    For lan I have vlan 10, voip I have vlan 20 and so on.

    So to start, I would first create a 172.16.x.x single vlan for transit which will be the only physical connection to pfsense. Since this is a vlan I would need to untag it on the switch port connecting it to pfsense transit port  on the same vlan network.

    Next I will re-create my lan, voip..etc "routing vlans" with 10.1.1.x, 10.1.2.x etc. And for the default route next hop I will put the single pfsense 172.16.x.x ip.

    Should the transit vlan be in the routing vlan or single vlan is fine?

    I will send the netgear doc link shortly.


  • LAYER 8 Global Moderator

    "Should the transit vlan be in the routing vlan or single vlan is fine?"

    I think your not grasping the concept of what a transit network is actually?  Routing vlans?  You mean your vlans off your downstream switch - why are you calling them routing vlans?  Been in the biz for 30+ years, that is not a actual term ;)  You mean the downstream networks?



  • Ha! Netgear has them as routing vlans. Just using their terms.  :D There is a separate tab in the switch to create routing vlans and has a "van routing wizard". Though in the end you can make changes to all vlans in the main vlan configuration page.
    Check this …

    http://kb.netgear.com/24709/How-do-I-use-the-VLAN-Routing-Wizard-on-a-smart-switch?cid=wmt_netgear_organic

    And this is the routing vlan doc from netgear I referred to earlier.

    http://kb.netgear.com/30818/How-to-configure-routing-VLANs-on-a-NETGEAR-managed-switch-with-shared-internet-access?cid=wmt_netgear_organic

    There is no mention of a transit ip. And in the end they use a netgear gateway for static routing.

    OK so I will do the below.

    Vlan 10 - 10.1.1.1 - ip on virtual port for lan (it's usually 4/1 on my switch)
    Vlan 20 - 10.1.2.1 - ip on virtual port for voip (4/2)
    Vlan 30 - 10.1.3.1 - ip on virtual port for video (4/3)
    Vlan 40 - 10.1.4.1 - ip on virtual port for home (4/4)

    Vlan 100 - 172.16.1.1 - transit ip to pfsense (4/5)
    172.16.1.2 - transit ip (single lan ip) on pfsense

    Let me know if this is right.


  • LAYER 8 Global Moderator

    Wow – what a shitty document..

    So they create asymmetrical routing condition…  Since you have hosts on what is your transit network ;)

    So they show to set the gateway of clients to be the switches IP in their "routing vlan" 192.168.10.254..  So I have a client that is on this 192.168.10 say .100, where does it send its traffic to get to the internet.. its gateway is .254 the switch, who then sends it on in the same network to the router at internet gateway at .1.  Now when that traffic gets back, where does it send it?  To the client directly, why would it send it to .254?  So now you have asymmetrical condition..

    Yeah wouldn't in a million years set it up like that.. If your going to put hosts on your transit, then you have to host route on them or you end up with hairpins and asymmetrical conditions.  And with a firewall in play as well that leads to trouble with out of state traffic..



  • This is the very reason I opened this thread.

    OK so now did go ahead and wiped all vlans and created a new set of vlans as per my post above.

    The devices within vlan 10 to 40 and 100 can ping each other and I can ping the virtual ips of each vlan.. like 10.1.1.1 and 10.1.2.1..

    Now I have a new issue. Transit ip on switch 172.16.0.1 can't ping  back to pfsense ip 172.16.0.2. From pfsense I can ping the switch transit ip 172.16.0.1 successfully.


  • LAYER 8 Netgate

    So look at your firewall rules.

    Where traffic needs to exit the firewall there has to be a route telling it where to go. (Policy route, connected interface, static route, default route.)

    Where traffic needs to enter the firewall there has to be a rule allowing it to enter.



  • Tried that already.

    Here is my pfsense LAN (transit) firewall rule

    IPv4 * 172.16.0.1 * 172.160.0.2 * * none Switch to Firewall
    IPv4 * LAN net * * * * none Default allow LAN to any rule

    IPv6 * LAN net * * * * none Default allow LAN IPv6 to any rule

    I have the LAN (transit) cable coming from pfSense on port 44 on the swtich.

    On the switch, I have vlan 100 with port 44 untagged. If I tag it I can't ping from pfSense. No other port is enabled on that vlan. The switch transit ip is 172.16.0.1 and pfSense is 172.16.0.2.

    I even tried assigning an IP directly to the port in IP Routing tab on the switch. Same results, can ping from pfSense but not from the clients inside the vlans.


  • LAYER 8 Netgate

    Looks like a typo here:

    IPv4 *  172.16.0.1  *  172.160.0.2  *  *  none      Switch to Firewall

    16 != 160



  • Just a typo .. it was 30 past midnight  8)

    It's 172.16 on pfsense.


  • LAYER 8 Global Moderator

    So all those rules allow is switch to talk to pfsense.. Doesn't allow any of your other traffic..

    You have all your downstream networks, your transit interface rule has to allow for that. I went over that in 1st post..  Your switch is not NATTING to its IP in the tranist..

    Also its much easier to post screenshot of your rules..



  • OK.. finally some progress. I connected to the transit network just to get my head around it and saw that I could connect to both 172.16.0.1 (switch) and .0.2 (pfsense) and the internet was working. But I of course couldn't ping 10.1.1.0/24 network on the switch. So I added a new gateway on pfSense for the LAN 172.16.0.1 and in static routing I pointed the network 10.1.1.0/24 to it. That got me reach to 10.1.1.0/24 netowork on the swtich. I then added a static route on pfSense for destination 10.1.1.0/24 use the gateway 172.16.0.1. That got my 10.1.1.0/24 network to start pinging the pfSense and the outside world.. 8.8.8.8



  • @johnpoz:

    your outbound nat has to be adjusted to outbound nat your downstream networks.

    I am on the default "Automatic outbound NAT rule generation." and my internal switch network is pointed to pfSense LAN (transit ip) 172.16.0.2 for DNS and its working. Am I doing this right? If not, then could you please guide me on what the setting need to be?


  • LAYER 8 Global Moderator

    Dude post up your outbound nat.. How is psense going to know to nat these downstream networks when it doesn't have them directly connected..

    You don't need to create multiple routes for all your /24's on pfsense - just use a summary route.  They are all in the 10 space, so 1 route to 10/8 gets you to your switch.  On your switch pfsense is internet, so that is the default route..



  • I didn't even touch the outbound nat. I think it took the static routes and updated the mappings. All I did is create static route for each subnet and pointed the transit ip as the gateway.

    Here is my automatic outbound nat.

    Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description
    WAN 127.0.0.0/8 10.1.1.0/24 10.1.2.0/24 10.1.3.0/24 10.1.4.0/24 172.16.0.0/30 * * 500 WAN address * Auto created rule for ISAKMP
    WAN 127.0.0.0/8 10.1.1.0/24 10.1.2.0/24 10.1.3.0/24 10.1.4.0/24 172.16.0.0/30 * * * WAN address * Auto created rule


  • LAYER 8 Netgate

    I then added a static route on pfSense for destination 10.1.1.0/24 use the gateway 172.16.0.1. That got my 10.1.1.0/24 network to start pinging the pfSense and the outside world.. 8.8.8.8

    So what's still not working?



  • The comment by johnpoz "your outbound nat has to be adjusted to outbound nat your downstream networks." got me thinking if I may be missing something that needs to be in place not to break the routes and if its not working the way it should.

    Hey as long as I don't have to manually change the settings I am a happy camper ;D

    Now if there was a way just to get pfSense DHCP relayed to the downstream networks it would had been icing on cake. I think I will move pfSense back to vmware and install a dhcp vm on the same hardware. With all my vlan traffic now handled by the switch I barely see any CPU usage on pfSense.


  • LAYER 8 Netgate

    You could also probably just DHCP in your switch.



  • @Derelict:

    You could also probably just DHCP in your switch.

    Yup that's what I am using currently but its not full fledged as I need it. Have squid proxy dhcp options plus a ton of dhcp static ip assignments which are a pain to manage or configure on the switch.

    I may need to start a new thread (or just use this one) as I have to do the same inter vlan routing for ipv6 to the outside world which is tunneling through HE. At the moment its completely broken in the new network. Was working fine when pfSense handled this.



  • @johnpoz:

    You don't need to create multiple routes for all your /24's on pfsense - just use a summary route.  They are all in the 10 space, so 1 route to 10/8 gets you to your switch.  On your switch pfsense is internet, so that is the default route..

    Thanks I will remove the routes and use the summary route. Makes managing the routes a bit simpler :)


  • LAYER 8 Netgate

    @Asterix:

    @Derelict:

    You could also probably just DHCP in your switch.

    Yup that's what I am using currently but its not full fledged as I need it. Have squid proxy dhcp options plus a ton of dhcp static ip assignments which are a pain to manage or configure on the switch.

    I may need to start a new thread (or just use this one) as I have to do the same inter vlan routing for ipv6 to the outside world which is tunneling through HE. At the moment its completely broken in the new network. Was working fine when pfSense handled this.

    Dual-stack the transit interface using a /64 out of the /48 (you might be able to get more clever here, but I wouldn't for now.)

    Probably just route a /56 out of the /48 to the switch and assign interfaces out of it there. You will have 256 /64s to play with.

    Default IPv6 route back to pfSense transit.

    Then just pass source traffic from that /56 on the pfsense transit interface.


  • LAYER 8 Global Moderator

    Yeah if you want ipv6, you just need to do the same thing via ipv6 transit which could just be link-local.  Doesn't have to be one of your /48 from HE.. But with a /48 you have plenty of /64s to play with so yeah you could use one as your transit network.  This would make for fully functional traceroutes, etc.

    https://tools.ietf.org/html/rfc7404
    Using Only Link-Local Addressing inside an IPv6 Network

    As to your dhcp relay problem - its been brought up quite a bit leveraging pfsense dhcp for pools that pfsense does not have an interface in that network.. I am not sure if that is going to be a future feature or not.  But it has come up quite a bit for long time.

    Personally I could care less.. If you have come to the point where you network is larger and more complex (ie using downstream routers) you prob should have a dedicated dhcp system with failover, centralized,etc etc.  While I agree with you the dhcp server feature list on switches prob going to be limited and interface and or config prob clunky compared to simple gui pfsense has put in place.

    If you have vm infrastructure already in place just fire up VM for your dedicated dhcp, etc. Prob want to match that up with your local dns so you can resolve your dhcp clients or reservations, etc.



  • Yup, doing that exactly on VM. Just that I have a dependency on a vm for DNS/DHCP which is not what I am a fan of. Anyways I installed a 2.4 snapshot last night on VM and got the network back up. Now getting the DNS/DHCP in place.

    I will resurrect this thread in a day or two as I would need some hand holding on the IPv6 part as I am not comfortable with IPv6 yet as much I am with IPv4 which I did on a regular basis back in 1998. (Windows NT… good old days)


  • LAYER 8 Global Moderator

    If your using a /48 tunnel from HE, then you can use any of those /64 behind pfsense - since that whole /48 is routed down your tunnel.

    So just use say the first 1 or the last one of the /64 as your transit.. then put your other /64 on your other segments on your downstream router.  Just create your routes for your /64 or summarize them with a /cidr that includes all the /64 your using but does not include your transit network.. Say a /56 on the other end of what your not using as your transit and there you go ;)



  • I bet I was not doing it correctly before… so would need you to point me where to put which IP.  ;D

    Ok. So I have the below info from HE.

    Routed /64:2001:470:xxxx:1010::/64
    Routed /48:2001:470:yyyy::/48

    Before going the L3 switch internal lan route I was using the /48 to /64 in this manner

    LAN: 2001:470:yyyy:1::1  (DHCP assigning lan clients 2001:470:yyyy:1::11 through 2001:470:yyyy:1::99)
    VoIP: 2001:470:yyyy:2::1 (DHCP assigning voip clients 2001:470:yyyy:2::11 through 2001:470:yyyy:2::99)
    Video: 2001:470:yyyy:3::1 (DHCP assigning video clients 2001:470:yyyy:3::11 through 2001:470:yyyy:3::99)

    Always wondered what happens to the 2001:470:xxxx:1010::/64 allocated by HE.

    So when you say I have 64 of these (2001:470:yyyy:: ) how would I write the 64 subnets.. if you can please provide an example of first 2 subnets and the last 2 subnets it would be really helpful.

    Here is my network.. I have also added a Microsfot DNS & DHCP on the internal vlan that is serving the clients on the L3 switch. The switch has DHCP relay which is helping relay IPs to all 4 intra lan subnets.

    pfSense
    WAN: some WAN IP
    Transit: 172.16.0.1  (what IPv6 goes here..  the xxx or the yyy one?) should it be like 2001:470:xxxx:1010::1 OR 2001:470:yyyy:172::1

    Switch
    DNS/DHCP: 10.1.1.2 (what IPv6 goes here?) (DHCP has IPv4 and IPv6 scope options)
    Transit: 172.16.0.2 (does this need an IPv6? If so how can I configure one as this virtual IP)
    LAN: 10.1.1.1 (same here, since this is virtual as well)
    VoIP: 10.1.2.1 (same for all below)
    Video: 10.1.3.1
    Home: 10.1.4.1


  • LAYER 8 Netgate

    One example:

    Route this to the L3 switch:

    2001:470:yyyy:ff00:/56

    You can then use 2001:470:yyyy:ff00:/64 through 2001:470:yyyy:ffff:/64 on interfaces there. 256 total.


  • LAYER 8 Global Moderator

    yeah that works.. your network expanded is this

    2001:0470:yyyy:0000:0000:0000:0000:0000/48

    So your first yeah the /64 subnets would be

    2001:0470:yyyy:subnet:0000:0000:0000:0000



  • @Derelict:

    Route this to the L3 switch:

    2001:470:yyyy:ff00:/56

    How do I route it to the L3 switch? That was one of my question earlier. I can add an IPv6 address to the transit interface on pfSense but where do I assign it on the switch? Should I just add a static IPv6 on the DNS/DHCP server and then add the /64 scopes for each subnet in the DHCP scopes section?

    What about the virtual routed vlan ips 10.1.1.1, 10.1.2.1.. etc


  • LAYER 8 Global Moderator

    you route it to your switch same way you route your 10 networks to your swtich.. over your transit ipv6 network which could be link-local or a /64 global address.

    Your pfsense and switch would have to have your global ipv6 transit IP on them..



  • @johnpoz:

    Your pfsense and switch would have to have your global ipv6 transit IP on them..

    Now what's a global ipv6? Is it one of the diffent 64 subnets that I can use? I understand on the pfsense it can be easily done by adding an IPv6 interface but how do I assign the same on the switch? especially when the transit ip on the switch itself is virtual.

    EDIT: Looks like I don't have Routing IPv6 configuration option in the switch. Just on the management port. Checked online docs and it shows higher end managed switches have the IPv6 routing tab under routing vlans.  >:(


  • LAYER 8 Netgate

    how do I assign the same on the switch?

    Probably need a different forum for that.

    Checked online docs and it shows higher end managed switches have the IPv6 routing tab under routing vlans.  >:(

    Yeah you'll need a real IPv6-ready Layer 3 (or maybe Layer 2+) switch to make that work.



  • Yup. My switch needs an IPv6 license to unlock the functionality. Darn netgear.


  • LAYER 8 Global Moderator

    Does your switch not allow for a management ipv6 address?

    Yes your global IPv6 address is one that falls in 2000::/3, this is the current global unicast IPv6 space.. There is PLENTY more that can be assigned.. but that is what is current..



  • So if I am reading that correctly, they are using a Layer 2 switch as a Layer 3 NAT router to bridge additional vlans into a single VLAN??


  • LAYER 8 Global Moderator

    If you have grown to the point that you need to do downstream routing, then its time to move to full time router or switch that actually supports full L3 to be honest..

    Its a common problem to be honest.. There is really no way to use your firewall as your router and not have a hit to the speed at which packets can move.. When network is small, or you do not do a lot of intervlan traffic that needs full wire speed it very convenient to just use the one device to handle the routing between your segments and the firewalling, etc.

    You need to make a decision.. If you need full wire speed between devices and can not put them on the same network then you can up your hardware to allow for the speed you want running through pfsense.  You can move the routing decision downstream which normally comes at a loss of firewall control between segments.  Depending the router/l3 switch you use may still have some ability to ACL but prob not going to be as easy as with the pfsense gui ;)

    Seems your wanting to do more than your current switch can provide - time to update to something better.  Port density with full L3 support comes at cost..  Depending on the number of devices and number of networks and room you have for hardware, etc.  You could get a smaller density L3 switch or true full blow router and use access switches for the port density you need.

    This really just comes down to a typical 3 layer model of access, distribution layer and core..

    How many devices total do you have, how many devices in each segment - which segments need the fastest intervlan?  You can not collapse the segments that really need to talk to each other at switching speed to the same layer 2?



  • I guess you are right, I may have to move to a managed L3 switch. Could you recommend a good solid L3 switch or a router?

    I have well over 70+ devices on my network. SmartTVs, mediaplayers, PS4s, Xboxs, iPads, Tabs, laptops, gaming desktops, home automation devices all pretty much running at the same time. Hard wired all devices that support it with CAT6 cables. With 11 kids (8 of my brother..  ;D) in the house especially on weekends, my initial network in fact ran like a 10Mbps hub (remember those things back in the 90's). Plus there is a ton of data that needs to flow for nightly backups. Kids have way too much digital stuff they just can't let go. I have my own test network consisting of servers and workstations, which I didn't mention in my previous posts.

    pfSense had become my central management for my entire network and it was becoming the bottleneck. Moving to inter vlan routing has provided significant improvement to my entire network as all pfSense does is provide access to WAN with some security (Snort, pfBlocker, SquidGuard).


  • LAYER 8 Global Moderator

    so how many of these 70+ devices are wired?  How are they distributed.. All comes down to budget if you ask me..  I have a cisco sg300 that I like.. cost me like $180 couple years back.  Current model would be sg350, it does true L3 and is very feature rich.

    There are the unfi switches, that have come long ways and are feature rich and can be managed from their controller software, etc.

    Like I said before.. depending how you lay out the access layer and the distribution layer will determine if you need a LOT of ports at your core or distribution layer or only need all the ports at the access layer, etc.  So something like a 10 ports L3 might be fine for your core or distribution..  So do you have everything wired to your current netgear or do you have some downstream switches to that.. We could prob still leverage it as access but put a L3 between it and your pfsense sense, etc.

    I see a sg350-10 at $197 on amazon currently
    https://www.amazon.com/SYSTEMS-10-Port-Gigabit-Managed-SG35010K9NA/dp/B01HYA36SG

    there is 28 for $395
    https://www.amazon.com/SYSTEMS-Sg350-28-28-Port-Gigabit-SG35028K9NA/dp/B01HYA38CA

    Here is a 8 port edgerouter for under $300
    https://www.amazon.com/Ubiquiti-Networks-ER-8-Edgerouter-Router/dp/B00IA5M2AS

    You might even take pfsense out of the equation with something like that, or you could still leverage pfsense as your edge firewall and use that as an internal router.. The Ubiquiti edgeswitch line does do layer 3, and their 24 porter starts at 215..
    https://www.ubnt.com/edgemax/edgeswitch-lite/

    You have to be careful - their unifi switches only do Layer 2, etc..

    So if you could give some more details of how all these devices are current connected and distributed throughout your house - where do you need port density?  Downstream switches? etc..  And what sort of budget you have in mind then we could work what hardware and configuration might give you the best bang for your buck!!


Log in to reply