Routing but mixed local and public IPs on the LAN



  • I have been lurking reading and searching all day but as a newbie the terminology and posts I have read,  leave me a little unsure on how to tackle this problem, so please don't flame me for asking if it has been already asked and I just didn't make the connection.

    I have 128 public IPs that I need access from the outside WAN to the internal LAN. The internal LAN uses another service provider gateway for other systems and devices and work stations.  There are times this 2nd gateway provider goes down and I want to switch over to my T1's.  I've been setting up PFsense all week recreating the firewall rules and tested the workstations using the default 1:1 nat from another post so I can get out (surf, download etc..) just fine and when I check what the real world sees as my IP it is the PFSense gateway (perfect)  now I am ready to assign a server a real world IP,  plug it in to the LAN interface side but can't access it from the outside.  I think I need to configure the outbound NAT because I don't want to deal with virtual IPs or NAT on that level, but I am not sure what settings to use.  I tried setting the interface to LAN and network source (the public IP range) and destination any, and then tried the same with the WAN setting, but as soon as I saved and tested it did not work.

    So in nutshell I want to be able to give a device or computer on my LAN either an internal AND/OR and external IP set the correct subnet and enter the PFSense gateway local IP (for any local devices that I need to route out the T1's)  OR a real world IP that I set the gateway to my External router (but goes through PFsense) that I get my publicIPs from.  Kind of a mixed mode NAT (sorta)  is this possible and if so how?

    THANKS!


  • Rebel Alliance Global Moderator

    So you have this??  See attached drawing

    So your box uses the router for its default gateway, but you want to route traffic from the internet through pfsense to it via a public IP??

    If so then yes your going to have to nat that so the box thinks its coming from the pfsense IP on its local network so that it doesn't route it back out its gateway.




  • yes for the most part.  And yes I agree some level of NAT is needed to route non-public IPs back to the destination devices, but I don't think I have to use a full NAT setup since all I would have to do it setup the correct gateway.  The other ISP (Uverse) uses the 192.168.1.254 and I have the PFSense internal IP (LAN) 192.168.1.7  I'd just change the Gateway IP between the two with any of the private IP devices. From what I am readying I need to setup the Outbound NAT in a hybrid/mixed mode, however I am not sure of what settings to use.  I mainly need to route the public IPs that are on the LAN out through the WAN without any NAT.  My current firewall has this setup (AstroFlow) however it is dated and the software not maintained any longer, so I know it is possible.  I've tried setting the interface to either and both WAN and LAN and then selecting the source network with the public IP range and neither work.

    Thanks for helping


  • Rebel Alliance Global Moderator

    Sure you could route on your server. Do you know the source network where this internet traffic is coming from?  So you not want to use your router as your default gateway, and just route traffic to networks connect to that router?

    Here is the thing putting a box on what amounts to a transit network is a FAIL right out of the gate..  Any network that connects 2 routers is a transit network - putting hosts on it is BAD design..  You could get away with it if these hosts on the transit would only ever see traffic from the router that is their gateway.  But as soon as they see traffic from the other router you have problem.  And now the host needs to have routes on it to know which router to talk to get to any source IP that would talk to it.

    I would rework your network so you do not have hosts on a transit would be my advice to you.



  • I certainly won't argue the point of a bad design although I've never had a problem with the current network in this state and it's been this way for over a decade without any problems I am aware of.  I am not opposed to creating a full Nat although I've had the impression that it will require a lot of routes and rules and more maintenance. as opposed to simply changing the IP, Subnet and Gateway to the network I want to connect to. All of the public devices also have a second NIC that use the internal 192.168.1.0 network but without it's gateway configured. This allows them to talk between each other, backups and other services.

    You asked if I knew where the internet traffic is coming from,  I'm not sure I understand your question, I would normally say a resounding yes, but I;m not sure of a case where I wouldn't know where it was coming from.  Perhaps if I elaborate more on the network design, might help shed some light.

    I have three networks (for the most part, I have a few isolated iSCSI networks with a dedicated separated switch)

    192.168.1.0/24 full 255 private
    192.168.0.0/24 full 255 private
    xxx.xxx.107.0/25 128 public IPs

    10 servers/devices that have dual NIC cards with a typical network configuration:

    NIC1: 
    IP: xxx.xxx.107.5
    SN: 255.255.255.128
    Gateway (to my AT&T ISP Router on site): xxx.xxx.107.1

    NIC2:
    IP: 192.168.1.5
    SN: 255.255.255.0
    Gateway: None

    NIC2 2nd IP:
    IP: 192.168.0.5
    SN: 255.255.255.0
    Gateway: None

    Now a typical workstation on this network

    One NIC
    IP: 192.168.1.200
    SN: 255.255.255.0
    Gateway: 192.168.1.254 (Gateway of the other ISP Uverse.  Single IP to WAN just your typical non-business provider )

    Then I have this workstation that now for whatever reason the Uverse goes down.  As it stand now I would give this workstation and ADDITIONAL hypothetical IP
    IP: xxx.xxx.107.100
    SN: 255.255.255.128
    Gateway (to my AT&T ISP Router on site): xxx.xxx.107.1

    IP: 192.168.1.200
    SN: 255.255.255.0
    Gateway: NONE

    This way I can still talk to the other devices on the network and be able to get out surf download etc…  The is only temporary but I make sure the computer's internal firewall setting are setup and running.

    All of the workstations and the hosting servers share the same physical switch.

    As I said I won't argue a bad design, but it has worked without any noticeable problems for over a decade. I've been tasked right now to replace the aging firewall, hence why I'm here.  I planned to simply replace the hardware and then start looking at the network as a whole and look at a total NAT system. Perhaps another post for another day? I am looking at the task of NAT'ing the current public servers as a large undertaking since I also have physical and virtual switches as well as internal software that use the public IPs.  I have the impression that it's not going to be a super easy task as this is a production environment.  Right now I don't want to rock the boat too much.

    Thanks again for your help.


  • Rebel Alliance Global Moderator

    "Then I have this workstation that now for whatever reason the Uverse goes down.  As it stand now I would give this workstation and ADDITIONAL hypothetical IP"

    Why should you have to do anything manually if one of your connections go down… If designed correctly this would all automagically happen at your router/edge were you multiple connections come in.. You could use load balancing across them when both are working if you want.  Or you could route some of your network out 1 network, and have your inbound traffic come in the other.  And if say your internet for your users isp went down you could get them to go out your other network connection.

    You can with some help of your isp advertise your public IP space out a different provider if 1 goes down.. Depends if you own the address space or assigned to you from the isp for your use, and if you use BGP, etc..

    Another option if you can not go that route would be simple dns design where dns points to publicIPA with isp1 for normal use, but if isp1 goes down you change your public IP to point to publicIPB that goes through isp2, etc..

    Like I said you can run hosts on a transit network if they never see traffic from the device that is not their default gateway, or you have host routes on that box that point to the other gateway for source networks that would come from there.  But again its bad design to put hosts on a transit network.

    While I agree a redesign of your network might be more work than your looking to do - in the long run its better to do it right!!!  And provide for proper failover, etc..  And if production do the redesign now once and have maybe only 1 outage on the cutover than try and fit something in now, and then never get to the redesign, etc..  And still have a messed up setup..



  • ISP #2 is a "home class" 20MBps down 3Mbps up,  one dynamic IP and cheap. So it can't act as a BGP  we only have it to keep our limited 5Mbps T1 lines (ISP#1) free of extra regular surfing traffic. Having to switch over to ISP #1 is rare, but does happen and when it does go out, it is normally for more than a day. ISP #1 on the other hand a good SLA with a 24 hour report, and a 4 hour response time. We are located in a rural area so there is some location handicaps to consider.

    Heading your warnings and recommendations all noted,  what do I need to do to get the system/network so I can at least get rid of the other old router.  I intend to look at an appliance from PFSense one of the two SOHO routers with 2 support incidences so that I can use to help configure the "Ideal" network. Then I can use this one I am setting up now as a failover in due time.

    Do appreciate your time, help,  insight, suggestions and concerns.



  • I've got some more insight on the current setup, so perhaps I can persuade someone to help.

    Attached is the basic network diagram.  The Public IP/Network is not mixed with the Local Private network in the traffic sense of the word.  Each device/server has multiple NIC cars with separate networks

    1. WAN (Public IP) Gateway set to ISP#1 for all Internet traffic
    2. LAN #1 192.168.1.0/24 (Used for ActiveDirectory, Authentication,  communicating, file sharing etc..) between ALL internal devices including the servers
    3. LAN #2 192.168.0.0/24 (Used for backups and large data transfers on local network)

    The Servers that have Public IPS have another NIC to connect to LAN#1 and #2 They do not have any gateway assigned on these NICs The Private Equipment (File severs, Workstations, Domain Controllers do not have a Public IP and use the ISP#2 as the gateway for the 192.168.1.X network.

    I guess the way it is the ISP#2 router (where I want to replace with the PFSense box will be a Routing and Firewall only)  I was hoping to be able to create a NAT on it so that I could have a 2nd gateway option in the event the ISP#2 goes down, but if that is going to make it less secured or cause problems with mixing the traffic, then we can forget that option.

    Where I need help and my questions

    1. So with this setup, in order to connect to the PFSense on the local network, I would have to either have another Interface or a VIP correct?
    2. I would disable ALL NATing. What IP do I set on the Lan and WAN interface so that I can filter, block?
    3. With no NAT, does that mean I could never add it later say for example I had an IP Address change, but I still wanted to route say FTP port traffic to the old server IP, is NAT all or None?
    4. Can I ever use VPN if there is no NAT?

    ![Network Diagram.jpg](/public/imported_attachments/1/Network Diagram.jpg)
    ![Network Diagram.jpg_thumb](/public/imported_attachments/1/Network Diagram.jpg_thumb)


  • Rebel Alliance Global Moderator

    "Each device/server has multiple NIC cars with separate networks"

    Another bad design decision!!!

    Why do you not just do this.. See attached..  You can 1:1 nat for stuff that needs public access on all ports.. But why would you not just port forward in the ports you need on the devices you need, etc..  Or if you have a routed network, you can still do that and use public behind pfsense and allow that to talk to your private network if need be via pfsense routing and firewalling that traffic.

    Multihoming devices is normally really really bad idea!!  Its one thing if if these other networks are for say backend management or storage access or backup network, etc.. But multihoming that provide their own paths to internet or other networks is going to for starters administrative and security nightmare..  Now you have to put specific routing on these hosts/devices, etc..  A compromised system allows access into other networks like it was this system.. There is one thing if that network only leads to management workstations or the devices own storage, or how the backups flow to the backup server, etc..




  • You have made it clear that I am in over my head.  I totally understand the top level design and flow, but I feel lost on how to even start implementing that  and what the hardware requirements would be.  What do you suggest I do to get the ball rolling in the right direction?  hire someone?


  • Rebel Alliance Global Moderator

    If your over your head, hiring someone would be an option sure.

    So do you have full control over this network??  Or are other people/dept involved in say switch access, ip management, etc. Do you have some devices you can use to lab so you can learn, before you try and roll it out into production?  If need be fully lab it with a virtual host or even virtualbox or vmware workstation on your computer/server.

    Take a look at https://doc.pfsense.org/index.php/Multi-WAN to get started with how to setup multiple wans in pfsense.  One of these networks can be public while the other is natted if that is the only way you can do it.  But maybe your 2nd isp device can be put into bridge mode, etc.  Where you would have 2 different public networks on pfsense wan interfaces.

    Are your switches managed? What currently provides dhcp/dns services for your devices.  Do you run any sort of Active Directory - multihomed in that setup can be even more of a nightmare seen it all the time where the person that sets up the server forgets to set a nic that is use for different network not to register with AD.  Or when you use public IP space internally and have not disabled IPv6 that gets registered in the AD dns as well.

    I would work on setting up pfsense in your lab.. Play with having more than 1 wan, how to route a network behind pfsense - this can all be simulated with rfc1918 space..  Once you have a handle on that can work out how to migrate your current network to the new design with min downtime and disruption.

    You can involve pfsense support services if need be https://portal.pfsense.org/support-subscription.php

    There is nothing wrong with being in over your head.. Bring in someone and use it a learning experience for sure.



  • I appreciate the help and suggestions.  To answer your questions, yes I have full control over the network.  I do have devices to learn, but I would rather get something setup and tested on the side and then migrate over to it one device/system at a time.

    I don't know about bridge modes. the other ISP like I said before is your typical consumer/prosumer ISP. Cheap, a single Dynamic IP and lots of download. The service is AT&T Uverse. I get a router that has DHCP and basic routing and DMZ.  I currently feed that to a LinkSYS 1600AC router and take over the routing etc..  Again we only use this to alleviate the extra bandwidth for day to day activity. I wouldn't really be able to do any failover for outgoing services without some sort of DNS service, and even then that could cause more issues with the update/refresh time when the service is back up.
    I have one SMC switch (SMC8126L2)  that is manageable, however it currently acts as an unmanaged switch.  The DHCP on the local lan is just the LinkSys router, DNS is provided by an internal Microsoft DNS/Active Directory server for the local network. I have another real world public DNS on another server, however it is not for anything except servicing public requests.

    Hope that helps shed more light.


  • Rebel Alliance Global Moderator

    Your problem is going to be trying to put the pfsense connecting both your isp connections and then your multihomed setup without any sort of down time..  Since in your final setup all your machines would only have 1 network connection.

    You could prob connect the pfsense to your 2 isp routers, and then connect it to your 2 switches and run your 2 networks and your multihomed setup with min downtime by just create 2 lan networks.  And then just migrate machines off the network they don't need to be on.

    But I would really suggest you play with pfsense offline without causing any havoc to your current network until you feel more comfortable in how you can work with 2 wans coming into pfsense with policy based routing and failover to the other connection if your 2nd isp goes down, etc..  I am heading out for some beers right now…

    But sure I can draw up a way to switch pfsense in place and leaving your current lan settings the same.. And then how to migrate off of that..  Does this box your planning on using have 4 nics so you can have your 2 wan and 2 lans?  If not your going to have to vlan, which could make for more down time when you do the switch over if you do it all at once, etc..



  • Good morning John,

    I would expect a little down time for each device/computer/server, but I can't make the hop all at once. There are simply too many devices so that has to be considered. The current box I was playing with is a Nexcom NSA 1040 with 4 ports but 2 are failovers.  I am seeing if I can add another NIC with 2 ports otherwise I would just assume by one of the 4 ot 6 port SOHOs PFSense sells.  I can't see us needing any more power than that on our modest network.

    Thanks
    Jason



  • Hi John just thought I'd see if you had any more thoughts or suggestions or if you needed any more info from me.