Multi WAN with Multi LAN - Multiple questions



  • For starters I tried to attach a generic, yet thorough, PNG diagram of my network… or intended network. If it doesn't/didn't work, here's a url link to it: (http://s15.postimg.org/7enigv49l/Generic_Net_Overview_feb2015.png)

    So I have two 50 meg fiber lines from two separate ISPs coming into my building. Their hardware hands off a CAT5 connection to me on a little /30 and each ISP has allotted me a publicly route-able class C network. As it stands, the other side of the /30 gets handled by a Cisco router that has 4 fastethernet interfaces. The config on that router has basic evenly weighted routes to each ISP and the other 2 interfaces have the .1 IP on each class C. Those then connect into my pfSense box which claims the .2 on each class C. There is no BGP or complex routing happening at the border. The only "abnormal" bit in the routing is that I have requested and been allowed by each ISP to route traffic out of their lines that are sourced from my specific class C of the other ISP - this is so we can route either way in case of an outage.

    So onto the internal side. I have a User LAN and a server LAN setup. The segregation is there to keep each side protected from the other although I allow specific traffic to travel between the LANs via firewall rules (i.e. email, SNMP, etc.). I also use the dnsmasq/forwarding service of pfSense for user to server links that require FQDNs.

    This is the basic structure as it stands. Now, here are the things that are happening or need to happen:

    • Services on the server LAN need to be able to serve to the outside internet. This is handled via 1:1 NAT'ing, aliases, and DNS entries (.com) that point to said NAT's on the "primary" ISP. Our failover without BGP is that each key service also have a NAT'd address on the backup ISP's class C and DNS entries on the .net side. Our custom software knows to failover to .NET and anything that is lower priority is handled via DNS changes with low TTLs or manually entering the .NET rather than .COM address.

    • By default, clients on the user LAN should use the Time Warner ISP. By default the server LAN should use AT&T. If either ISP fails, then obviously all clients should use whichever is up and once the failure is cleared, they should go back to the defaults.

    • On the user LAN I would like to be able to set some client IPs up to use the AT&T bandwidth yet still be part of the user LAN. My gut says this should/could be handled with a subnet with some sort of firewall or group rule?

    Okay, now here are my current issues and questions:

    • Is the Cisco even needed any longer? We initially tried to remove it from the equation (hired a consultant) but never got it working. It's been a while but I think the /30 handoff was a problem as we are using multiple IPs of the /24s to actually get traffic into our servers. If the Cisco can be taken out of the picture, how can that be done while still allowing all of the above?

    • How do I make the server LAN go out AT&T and the user LAN go out TW? It's not working now… everything seems choose AT&T

    • Once we're routing out correctly (above) how can I make a select group on the user LAN actually use AT&T?

    There's more I'd like to get figured out, but these core issues are on my current goals sheet. If it matters, the netgate hardware I'm running pfSense on has 6 cat5 gig interfaces and with the current setup I'm using 4 of them. I was hoping to add in our copper T1/last resort down the line but if all 6 ports are needed to get things working as above, the last resort failover option can be handled another way.

    Any pointers appreciated. I can give current config sections if necessary.

    -dtk



  • Maybe I should break this down into multiple questions in multiple subforums?

    I know this is supposed to be the Multi-WAN specific forum so if anyone has recommendations specific to the WAN portions of this I'd appreciate it. I can reconfigure the question to only contain that portion if needed.

    Thanks,
    dtk



  • So I'm wondering of the 120+ views are people with the same or similar problems?



  • This sounds like you need policy based routing.  There is a doc for it that can get you started:  https://doc.pfsense.org/index.php/Multi-WAN_2.0#Gateway_Groups

    If you're using pfSense as the default gateway for everything in the pic you attached, the traffic is going to flow out the default gateway, sounds like that is what is  happening to you.  Although I've never configured this, it sounds like you'll need multiple gateway groups.  One for default on one ISP with backup of the other and vice-versa.  Then policy based route to the gateway groups as needed.  I suspect the .1 addresses on the  /24 networks you have on the Cisco as the upstream gateway determine your route from there.

    I'm not sure how to get the special user group to route differently unless you can guarantee the IPs they receive (DHCP reservation or static) and then make sure they show up higher in the list of policy based routes.

    Provided you can get the same connectivity as the Cisco router, I think you can get rid of it by making the a.b.c.0/24 and x.y.z.0/24 DMZs on pfSense.  If you don't have a ton of Ethernet ports on the pfSense box, VLANs are your friend.  You'll still need interfaces for the server and user LANs on pfSense to handle NAT.  There will likely be many firewall rules needed since pfSense would act as the hub for everything.  There is no way to swap out the Cisco without an outage though with /30 networks from the ISPs.


  • LAYER 8 Netgate

    @dtikev:

    The only "abnormal" bit in the routing is that I have requested and been allowed by each ISP to route traffic out of their lines that are sourced from my specific class C of the other ISP - this is so we can route either way in case of an outage.

    Without BGP how does traffic from the outside destined for the /24 swing from one circuit to the other in the event of an outage?  There's more to it than just sending.  How is it routed inbound?



  • It's going to depend upon how the upstream providers handle it too, but you could just use BGP as an add-on package:  https://doc.pfsense.org/index.php/OpenBGPD_package



  • @Derelict:

    @dtikev:

    The only "abnormal" bit in the routing is that I have requested and been allowed by each ISP to route traffic out of their lines that are sourced from my specific class C of the other ISP - this is so we can route either way in case of an outage.

    Without BGP how does traffic from the outside destined for the /24 swing from one circuit to the other in the event of an outage?  There's more to it than just sending.  How is it routed inbound?

    Inbound routing is handled via DNS. Any services that are "critical" have IPs on both public facing networks and our client software knows how to elevate or demote target IPs based on TTLs or outright outages. BGP isn't currently an option.

    So even the less than critical services have ingress points on both IP ranges and short TTLs on DNS entries allow for minimal downtime. We also have .COM and .NET for our domain pointed at the two different IP ranges so that there are ingress points to any service via names.



  • @dtikev:

    Okay, now here are my current issues and questions:

    • Is the Cisco even needed any longer? We initially tried to remove it from the equation (hired a consultant) but never got it working. It's been a while but I think the /30 handoff was a problem as we are using multiple IPs of the /24s to actually get traffic into our servers. If the Cisco can be taken out of the picture, how can that be done while still allowing all of the above?

    • How do I make the server LAN go out AT&T and the user LAN go out TW? It's not working now… everything seems choose AT&T

    • Once we're routing out correctly (above) how can I make a select group on the user LAN actually use AT&T?

    There's more I'd like to get figured out, but these core issues are on my current goals sheet. If it matters, the netgate hardware I'm running pfSense on has 6 cat5 gig interfaces and with the current setup I'm using 4 of them. I was hoping to add in our copper T1/last resort down the line but if all 6 ports are needed to get things working as above, the last resort failover option can be handled another way.

    I think theoretically you shouldn't need the Cisco anymore.  If your servers are all private IPs and you are NATing them, maybe you can create the Virtual IPs in pfSense and NAT them out the /30 connections that would come directly into the pfSense.  Just curious, why didn't you pay for pfSense support hours instead of hiring a consultant?  They should be able to tell you what is possible and help you with the config, and I doubt it would be more than paying an outside consultant.

    As others stated above, I think you need policy routing, either on the Cisco (if you keep it) or the pfSense (if you don't).  If you keep the Cisco, why do you have two links from the Cisco to pfSense?  Get rid of one, do the NAT on the Cisco, and set up policy routing on the Cisco to route the appropriate subnet or IP out the correct interface.  If you don't keep the Cisco, you probably want to set up gateway groups in the pfSense.  You can either load balance or create two gateway groups, each one sends traffic out an ISP and fails over to the other.

    I don't think you should need more interfaces on the pfSense unless some of your servers actually have public IP addresses and you're getting rid of the Cisco, then you need a physical interface on the /24 to connect the server to.  You could still use VLANs though to avoid extra interfaces.  If you have a T1 you will obviously have to connect it to a router first though.

    Hope this helps, let us know how it turns out.


Log in to reply