Replacing a Cisco Router/VLAN



  • I have a client that has an office building with a number of tenants.  They have had a fiber connection with an ancient cisco router for many years.  Recently they upgraded their fiber to 100mps and discovered that the cisco router was only capable of a 10mps connection.

    The router was set up with a Wan connection and then a series of VLANS all with public IP addresses.  The WAN is aaa.zzz.xxx.1/30 and the VLANS split up a aaa.www.yyy.1/25 group of public IP addresses.  Each VLAN is assigned to one of the tenant offices.

    Over the years, all of the tenants have migrated/upgraded to PFSense boxes running their individual networks and they have been quite happy with the results

    As a result, the building owner wants to replace the Cisco router/VLANS with a pfsense box.

    There are a number of ways I could duplicate the cisco hardware, but I have a supermicro motherboard lying around with 8 LAN ports so I plan to use that for PFSense.

    I've setup dozens of PFsense installations and initially thought this would be a simple project.  However the more I think about the configuration the less sure I am of the correct approach.

    The Tenant segments which are all public route-able addresses are outside the WAN segment.  Each is defined with an /28 address block and a gateway using the first available address in the segment.  My plan has been to replicate this configuration in PFSense.  So each OPT interface will have the same configuration as one of the VLANS on the CISCO router.  Then, I would turn off outbound NAT and make a bunch of rules to pass all of the traffic.  However, I don't see how the traffic is going to get from the OPTx interface to the WAN interface and back again.

    The cisco router has a default gateway (The WAN gateway) and then an ip route 0.0.0.0.0.0.0.0.0 that then points to this gateway.  Pfsense will only let me define a route for a /1 subnet, so do I need to put in two static routes  0.0.0.0/1 and 129.0.0.0/1?

    Is there an alternative approach that I should take?


  • Netgate

    Set the pfSense WAN interface just like whatever the WAN interface is like on the Cisco.

    Create the gateway on WAN to point to wherever the cisco default gateway points to. Look in System > Routing. That is where the gateway "objects" are and you can use that to set the default gateway (automatic when you add the gateway to the first WAN interface).

    I would, personally, probably use a layer 3 switch to route to the different tenants instead of pfSense VLANs but as long as the number of VLANs is manageable it should work fine. There are advantages to traffic shaping on just one interface instead of several, should that ever become necessary. There are also advantages where multi-wan/policy routing is concerned.



  • Thanks - that is the way I currently have things set up - the WAN is set up the same as it was for the CISCO switch and its gateway is the default.  But, there are 7 other gateways for each of the OPT interfaces.  Just to be clear, I'm not using VLANS - just moving the original VLAN configuration to separate OPT interfaces.


  • Netgate

    Why are there gateways on the "inside" interfaces?



  • Yes - that was a configuration mistake on my part - I was just trying to replicate the setup from the CISCO.    In the cisco box each of the Vlans was defined as  xxx.176.12.0 with an appropriate subnet mask and a gateway set at xxx.176.12.1.  I followed that same scheme when setting up the pfsense router and of course when I plugged it in, it didn't work.  The wan gateway was set as the default, but nothing was using it.  I was letting the fact that the addresses were public confuse me - once I "adjusted" my thinking to treat them just the same as any other internal network, I realized the mistake, removed the gateways and everything started working as it should.

    The last thing I am trying to verify is the correct configuration for outbound NAT.  Each of the pfsense boxes on the various segments are doing some form of outbound NAT - routing packets to the CISCO and then distributing them appropriately upon their return.

    When I replace the cisco with pfsense, is my thinking correct that automatic outbound NAT is all that is needed at that level?  This top level pfsense box is only concerned with moving the packets to the WAN gateway and then routing them back to the single interface address that represents the correct network segment.  Once they reach the sub-level, the original NAT will continue to direct them to the correct device.


  • Rebel Alliance Global Moderator

    "When I replace the cisco with pfsense, is my thinking correct that automatic outbound NAT is all that is needed at that level?"

    Depends - you need to make sure that pfsense will nat any downstream networks you have running, etc.  You will want to verify that your outbound nats reflect any downstream networks you have in play.  And that the rules on the interfaces used to talk to those downstream networks allow traffic to them from the downstream networks.



  • OK, now I am a bit confused.  So there are downstream devices for each segment  Each has a pfsense firewall in place.  So network1 is setup with the wan port having an ipaddress of xxx.176.12.2/28 and uses a gateway of xxx.176.12.1    There are numerous servers on the lan side, so I have manual outbound nat in place so that when the server at 192.168.1.3 routes traffic out it gets translated to xxx.172.12.3 and when the server at 192.168.1.4 sends traffic the translation is to xxx.176.12.4  etc.

    On the top level PFSense box, I have the WAN interface defined with a default gateway of xxx.177.151.25  I then have 7 additional interfaces of which one is set as xxx.176.12.1/28

    My thinking has been that when the server at 192.168.1.3 routes a packet, the sublevel pfsense box will translate the address to "know" that when the packet is returned to the WAN interface (xxx.176.12.2) for xxx.176.12.3 it needs to then be directed to 192.168.1.3.

    Now when that same packet reaches the top-level pfsense box it merely needs to add a translation so that any package that falls in the subnet for network1 needs to be sent to xxx.176.12.2.

    I can't think of any reason that the top-level box will need to know anything about how the devices on network1 are defined and NATed. unless there were multiple pfsense boxes on the same network segment, then the top-level pfsense will need manual rules in place to translate the traffic from both devices.  If there is just one pfsense box on each sub-segment, then auto nat should work.  Is there something here I am missing?


  • Rebel Alliance Global Moderator

    Why would you be doing nat at downstream rfc1918 networks??  That is not normal setup..

    Unless you were not in control of these downstream networks and there was an overlap, or you were trying to work around some sort of asymmetrical routing issue why would you nat between rfc1918 space on your own network?

    Normally when you create a gateway to downstream network pfsense will create the nat rule for these downstream networks, but you would still have to address the rules.  And if you have when to hybrid or manual mode on your outbound nat you might have to create them so its always best to check!! But if you are natting the downstream networks - why would you need to create routes in the first place??

    If you are natting these downstream networks pfsense would never see IP other than the transit network that connects them. So no routes would be needed to these downstream networks - unless you had downstream networks to this downstream nat point..

    To be honest sounds like you have a MESS ;)



  • Just to summarize.  I have a client that has a large office building with a number of floors and offices.  They currently have a fiber connection with a cisco router that is set up with a series of Vlans for the different floors.  The building has a 128 public ip addresses and each floor has a subnet with some assigned portion of these addresses.    Each of the floors/tenants has a pfsense firewall in place that provides network services for whatever company is located on that floor.  Each has a separate 192.xxx.xxx.xxx network with a mix of clients and servers.    This setup which had evolved over time has worked well for almost 10 years with constant changes as companies and requirements come and go.  In essence the building owner is a mini ISP.  The fiber has been upgraded to 100mps and the cisco router is only a 10mps device.  The building owner wants me to put a pfsense box in its place.  All I am trying to do is to mimic the same configuration that the cisco router provided.  I have a box with 8 intel lan ports and I configuring it for this purpose.    Ideally I would get rid of the other pfsense boxes and move everything to this one device, however, each office/floor is managed by different organizations.  They have their own configuration and often an IT type person who manages their network.  I may have a mess, but it is the same mess that has been in place for many years.


  • Netgate

    If you are providing public IP addresses to downstream pfSense (or other) firewalls and those downstream devices have RFC1918 networks inside and are performing NAT to their public addresses on their WAN interfaces, then you want to NOT perform NAT on the upstream pfSense node.

    In fact, you want to provide very little firewalling at all. Protect the pfSense web gui, ssh, etc and pass everything else.

    I would use Hybrid outbound NAT mode for that. Place Do Not NAT rules for the public networks you are routing downstream.
    That way you can NAT something if you have to (which would not be possible if you just disabled NAT altogether.)

    You could also use Manual outbound NAT and delete/disable the rules for the networks you don't want to NAT for.

    If all of the downstream /28 networks are contained in the same /24 you can just use the /24 summary network for the NAT bypass.

    (I would still use a transit network to a Layer 3 switch for this. LACP a couple of those ports to the switch and you'd be golden.)


  • Rebel Alliance Global Moderator

    ^ exactly… Well stated!

    Depending on your needs, you wouldn't even have to nat their downstream rfc1918 address space to their public.  Does this rfc1918 space behind their downstream need internet at all?  If so prob best to nat it at the edge, this would give you more insight/control over these downstream devices go on the internet.  But if they want/need internet from these rfc1918 networks then yeah those could be natted to their public transit IP you setup.

    Did you slice up part of your public space or are you using rfc1918 as transit to these downstream that have public on them?

    "All I am trying to do is to mimic the same configuration that the cisco router provided."

    This is not always the best approach, even if its been in place for 10+ years.. The replacement of your cisco would be/is the perfect time to address your design and make any changes that provide for better control, adheres to current standards, takes advantages that are now possible vs what would of be designed 10 years ago, etc.

    It would be easier and simpler design to break it up at the edge and remove the need for any downstream routers per customer, unless they wanted to use some rfc1918 space and control its nat and access, etc.. Or breakup their /28 even more, etc.


  • Netgate

    If they have downstream routers/firewalls firewalls that are performing NAT he can give zero about what RFC1918 networks are behind them.


  • Rebel Alliance Global Moderator

    ^ very true!



  • Thanks for all the good advice and helping me work through the configuration.  My mandate is to get the new box in place quickly and have it result in no impact to the various offices, other than providing a 10 fold increase in bandwidth (hence my desire to just replace what is there).  Once I get this in place I can start the long tedious process of redoing the master design.  However as you can imagine with a dozen entities involved the process is worse then herding cats - more like herding squirrels.


  • Netgate

    Scheduling a maintenance window and doing it right the first time is often the best way to go.

    Sometimes the dog needs to wag the tail, not the other way around.