Looking for pointers on where to start with a *complex* setup
-
I'll preface this with a general note on me. I started as a CompSci degree holder who was hired as tech support. Over the course of 18+ years I've melded into the man of many hats and have pretty much been self trained as a network admin. We are a fairly small company and I am the IT. We've hired consultants from time to time to handle some complex stuff but I handle the great majority of IT "things". My main management is currently a lot of linux servers, a few Windows servers and a load of MS based desktops.
We had a horribly complex and temper-mental "black box" that sat at our border and handled all of the heavy ip filtering work. This was a linux box with 8 NICs running iptables. I didn't build or configure it but I learned enough to know how to add a hole when we needed a new server setup. As it became older and more temper-mental we were guided towards pfsense as a better solution.
Part of this transition has allowed us to get rid of quite a bit of complexity and so now here I am, trying to get things working as intended. We hired a consultant to get us to a base - working - setup and off of the old blackbox beast. But now as I try to move forward with our plans I am seeing odd behaviors and want to get a clear idea of how I should proceed before I build something that a future person might call… temper-mental. :)
Without getting into the nitty gritty here is what we are dealing with and where we want to go:
We have two high bandwidth fiber lines that each allot us a full class C of IP addresses. They currently are handed to us from the providers as a cat5 cable and a small /30 network to interface from my side to the ISPs side. They are currently being handled by an old Cisco router that has 4 ethernet ports so essentially the handoffs come into the cisco with their side of the /30 of each provider and then the other two ports have the .1 of each of our /24s and are connected as WAN ports on the pfsense as .2 of each /24. (NOTE: We don't run or intend to run BGP at this point).
Behind the pfsense I have 2 LANs. The intention is for one to be a USER space and the other to be a SERVER space. The main reason for the segregation is security. I don't want a random infected pc on one side to be able to effect anything on the other nor do I want a tinkerer on the USER side to be able to tinker some damage on the SERVER side. I do, however need to allow some accesses from USER to SERVER for management and even dnsmasq stuff to allow access to the mail and web servers without sending USERs out to the cloud just to get to a machine on SERVER.
On the SERVER side there are machines that are 1:1 NAT'd and need to answer to real world requests. The current setup also required Virtual IPs to work apparently so anything on SERVER that has a 1:1 NAT also has their real /24 address in the vIP table.
Let's see... ok, traffic shaping wise we are trying to get to some specific places. The basics are that we want all things on the SERVER side to default to using ISP-A for their outbound requests and all of the USERS to use ISP-B. The caveat there is that I would like the ability at least on the USER side to have a special group, or maybe a /26 range of IPs who will instead get ISP-A as their route. And on top of all of this we would want each lan to fail over to the other WAN in the case of an outage and then roll back when their intended WAN comes back.
So far I don't think any of these things should be too difficult to accomplish but nevertheless I can see that something isn't right by doing speedtests and traceroutes, etc. I am unclear on what things take precedence but it seems like no matter what I do on the firewall side of things, the default gateway setting for the WANs always trumps the firewall - yet not completely. If I set ISP-A as the default but the USER lan has a rule that says all IPs use the ISP-B gateway, traffic still goes out of ISP-A. I must be missing some fundamental thing here.
If needed I can give specifics of the config, I just wanted to start with a broad picture to see if there are any other things that are needed to get me in the correct direction. For the failover stuff we have setup two Gateway Groups. One uses ISPa as default with ISPb as tier2 and the other is the inverse. On the LANs the first rule in each is a rule that says any sourced IP on the LANs interface to any destination should use their preferred ISPs gateway group. Other than that the firewalls are full of all sorts of rules to allow traffic to and from specific places as you'd expect.
Can anyone get me started in the right direction to our intended destination?
-dtikev
p.s. I am using a Netgate 7541 so I'm limited to 6 interfaces but another goal, if it's possible, would be to take the cisco mentioned above out of the equation. We've tried twice but it always kills our NAT's.
-
Not to be pithy, but I think a diagram would be a great place to start. It would add clarity to your description as well as invite commentary. Aside from the additional detail it gives us, just creating the diagram can solidify your own conceptualization. Doesn't have to be fancy, but it should include your major goals like the server "DMZ" and split routing with as much detail as you are interested in adding.
You could include this info in the diagram, but lacking the diagram, some questions that I have based on your description are:
I'm not entirely clear on the topology. Are you saying it's something like:
(LAN1 & LAN2) –> Switch(es) --> pfSense --> Cisco --> (ISP1 & ISP 2)What is the bandwidth of the various links (internet & internal). (pfSense will become a chokepoint in your design, not sure if a Netgate 7541 is up to it? I think the Netgate 7541 includes one year of support... maybe that's a resource for you?)
What is the Cisco model number and why do you want to remove it? (Cisco's are generally fairly reliable and generally quite good at doing what they were designed to do.)
-
Not to be pithy, but I think a diagram would be a great place to start. It would add clarity to your description as well as invite commentary. Aside from the additional detail it gives us, just creating the diagram can solidify your own conceptualization. Doesn't have to be fancy, but it should include your major goals like the server "DMZ" and split routing with as much detail as you are interested in adding.
You could include this info in the diagram, but lacking the diagram, some questions that I have based on your description are:
I'm not entirely clear on the topology. Are you saying it's something like:
(LAN1 & LAN2) –> Switch(es) --> pfSense --> Cisco --> (ISP1 & ISP 2)What is the bandwidth of the various links (internet & internal). (pfSense will become a chokepoint in your design, not sure if a Netgate 7541 is up to it? I think the Netgate 7541 includes one year of support... maybe that's a resource for you?)
What is the Cisco model number and why do you want to remove it? (Cisco's are generally fairly reliable and generally quite good at doing what they were designed to do.)
Your flow diagram is correct. One ISP is currently 50Mbps and the other is 30Mbps. I will draw out a clearer diagram as soon as I can. As far as the hardware goes I was led to believe that it is more than enough for our needs so I hope that this is true. I will check with my Netgate contract… I know there is support but hadn't considered that they may actually help with the firewall setup beyond the basics.