Using pfsense with multiple WANs
-
I apologize in advance if I don't use the correct verbiage but I'll try to explain as best I can what I need to accomplish.
A primary firewall (pfsense 01) routes WAN traffic to LAN sided servers.
This server has two NICS. One public and one private. It has its own dedicated Internet connection.A second server that I labeled it pfsense 02 in the image accepts Internet connections to a small app. This server also has two NICS, one public and one private. It also has its own dedicated Internet connection. The app is being moved to one of the servers behind the LAN Switch but the incoming connections still need to be routed to that server.
All of the servers currently communicate on the LAN side of pfsense 01
As mentioned above, the app will be moved to a server behind the LAN Switch but I still need to route its traffic to the new server. That's why I was wondering about converting this server to a pfsense one.
This is where things get complicated.
I mentioned having to move a service to a server that is connected to the LAN Switch. This is new hardware being installed but it has to have its own firewall because it's a completely separate Internet connection than the other two above.In the image, I show a LAN and a LAN Switch.
I show dotted arrows where I'm not sure how to connect these things together. Am I going to connect the LAN to the pfsense 03 firewall or to the LAN Switch? I'm not sure which is what my question is.
Traffic from pfsense 01 and pfsense 02 has to reach the servers behind the LAN Switch on private IPs. Traffic is coming in from what I call 'Direct bandwidth' in the image also has to reach the servers behind the LAN Switch.
How do I make all this work? I think I'm over my head or I'm not seeing something that is much simpler than I imagine. The main problem is that the Direct bandwidth must go through a firewall otherwise it's an open connection to the Internet.
I really don't know if I'm explaining this correctly but if something is not clear, I am happy to try and clarify but please, if you answer this question, keep in mind that I am not a network engineer so need to keep the verbiage simple enough for a layman to understand :).
And, thank you in advance if you can offer any insight.
-
@lewis So, both of the LAN(s) will need to be smart switches. Do you need three pfSense box? It seems that two could work if you incorporate HAproxy package on one and the other as failover. What kind of connection is the dotted lines, Ethernet cable, fiber? To get significant help, you will need to tell us what you what you're doing and what you want to accomplish, such as is this project for a home or business?
-
Is the LAN side of this all one large subnet?
3 routers on one subnet like that is just asking for trouble IMO!
If I was doing this I would use one pfSense box (could be an HA pair) with 3 WAN connections and route/filter the traffic accordingly.I would also want to be separating those internal services with VLANs if they can be on separate subnets.
Steve
-
(I updated the image above. Maybe it can help)
So, both of the LAN(s) will need to be smart switches.
Hi. What I'm needing is a sort of software router but pfsense seems like it could handle this, I'm just not sure how and I'm not sure I have the knowledge to accomplish this.
Do you need three pfSense box?
I don't think so but I'm not 100% sure, hence this post.
I'm trying to avoid having to deal with multiple LAN and VLAN networks. I'll explain this more at the end.I have to route traffic to/from the devices I call pfsense 01/02 to the servers you see connected to the LAN Switch in the image.
you will need to tell us what you what you're doing and what
you want to accomplishThe scenario is that a company I am helping rents servers from a small data center. Every server came with its own WAN connection and IPs and are in different areas of the data center and all communicate over the LAN.
The plan is to install hardware in one single cabinet to consolidate the whole thing. Problem is, the existing servers have long-term, pre-paid contracts and their IPs cannot be moved (network policy).
I need to find a way to route the traffic from each of those existing servers to the new servers in the cabinet. However, as you can see, the new cabinet will also have its own Internet bandwidth going into a pfsense firewall (03).
The image shows that pfsense 01 is the primary firewall at this time which has a number of virtual IPs and all of that traffic has to go to the new cabinet/servers.
The image also shows a second server I called pfsense 02. This is not a pfsense box right now, it's a server running an app. That app is being moved to one of the servers in the cabinet but the IPs cannot be moved or changed so that traffic needs to be routed to the new cabinet as well.
My thought was maybe I could convert that box (02) to a pfsense firewall to forward its traffic to the new servers in the cabinet.
What kind of connection is the dotted lines, Ethernet cable, fiber?
The dotted lines represent my question in terms of my not knowing how this is getting connected.
The traffic from what I called pfsense 01 and 02 would be routed to the new hardware via the current LAN. However, putting the cabinet in place with its own firewall essentially creates it's own WAN/LAN setup too so I'm not sure what the best way of doing this part is.
So the main question is....
Method 1
Do I route the traffic from pfsense 01/02 directly to the LAN Switch and onto the servers? And if so, how? Does it mean I would have to set up a bunch of networks on each of the servers?In this case, doesn't it mean each server would have to know about the different LAN/VLAN networks? I badly want to avoid needing to get into something too complicated.
Method 2
Or do I route the traffic from pfsense 01/02 directly to the pfsense 03 box? And if so, does it mean I need to use a different network for each pfsense (01/02) and then I can use just one LAN side network for all of the servers?In this case, pfsense 01 could have 10.0.0.1/24 routing to pfsense 03. Then pfsense 02 could have 10.0.1.1/24 routing to pfsense 03.
Just like I already do, I could have different networks on pfsense 03 but its main LAN network could be say 172.16.1.1/24 so that all of the servers could be in that one network. In this method, I could also take advantage of firewall rules between networks as well which might be a big benefit.
I'm just not very clear on how to handle this so thought I would ask since there are some seriously knowledgeable people in these forums.
-
@stephenw10 said in Multi LAN networks to one pfsense:
Is the LAN side of this all one large subnet?
Yes, it's just one, private LAN. However, because the new hardware will have its own pfsense firewall, it creates a sort of a dual LAN. I can use the main LAN to route traffic to the cabinet.
The more I think about it, it seems that my Method 2 sounds like a simple solution.
3 routers on one subnet like that is just asking for trouble IMO!
I agree. It's not something I want to get into but have to solve the problem.
If I was doing this I would use one pfSense box (could be an HA pair) with 3 WAN connections and route/filter the traffic accordingly.
Yes but there are multiple existing servers in different areas of the data center, each of them with their own WAN/IPs that need to be forwarded to the new cabinet/servers.
-
I mean... it seems slight absurd that the data center can't just route those public IPs to the new hardware drop. But assuming they can't....
I would want to setup transport subnets between the existing locations and the new pfSense and then just treat it as 3 WANs there.
How is the common LAN setup? I would imagine the data center provide it as a VLAN, it's not a physical cable?
In which case you probably can't use VLANs for the transport subnets so it would have to be two subnets in one layter2. Ugly. Though you can also just use one subnet with two gateways in it. Also ugly. Not sure which is uglier! Either would work though.Steve
-
I mean... it seems slight absurd that the data center can't just route >those public IPs to the new hardware drop. But assuming they can't....
They have their reasons, nothing I can argue about so have to find a way around the problem.
I would want to setup transport subnets between the existing >locations and the new pfSense and then just treat it as 3 WANs >there.
Do you mean like my Method 2 thought? Just set up unique IPs on pfsense 01/02 and route that to pfsense 03.
Then pfsense 03 would just see that traffic like a normal setup with multiple NICs. One would be on the data center side accepting connections from pfsense 01/02, another would be for the new WAN and one for the LAN side to servers in the cab.
How is the common LAN setup? I would imagine the data center >provide it as a VLAN, it's not a physical cable?
Correct. and if I don't use the LAN side, then the cost of bandwidth doubles because it has to go over public networks.
In which case you probably can't use VLANs for the transport >subnets so it would have to be two subnets in one layter2.
Do you mean something like I mentioned?
I might have pfsense 01 (10.0.0.1/24) and pfsense 02 (10.0.1.1/24) routing to pfsense 03 over the VLAN.
If this is what you mean, you also mention this would be ugly?
I have 4 separate networks / NICS on my home pfsense.
One is the WAN to Internet.
One is the main LAN.
One is OPT1 for wireless AP.
One is OP2 for stuff I give remote access to.
All of it managed via firewall rules between networks.In this case, could I not allow pfsense 01 10.0.0.1/24 and pfsense 02 10.0.1.1/24 over the DC LAN (VLAN) to one NIC on pfsense 03?
On pfsense 03, I would have three NICS, one for the new WAN, one for the DC LAN and one on the LAN side of the switch to the servers. -
Yes. Though I would use as small a transport subnet as possible to reduce the possibility of some other device being on it. So /30 there with just two IPs.
It's ugly because there's no layer 2 separation between those subnets. Broadcasts will appear on all devices on them for example. But it should work.Steve
-
I confirmed that the DC gives each customer a VLAN which they can use for anything they want. Meaning, 10, 172, 192, all can run over the VLAN.
So method 2 then.
Now, I need to make sure I get this right as all this needs to be installed this coming weekend.
pfsense 01
The next challenge I'm going to face is the traffic coming from pfsense 01 needs to be sent to existing servers and slowly transitioned to the new cab/servers.This means I cannot make any changes on pfsense 01 until everything is in place and I start transitioning.
The pfsense 01 handles DHCP services for 10.0.0.1/24 and has incoming traffic on a couple of /29's.
pfsense 02
This one is simple enough and it's later, after everything else is in place.pfsense 03
-Use WAN for the new Internet connection.-Use LAN for the LAN Switch connection and in turn, the servers.
Maybe I should use the 192.168.1.1/24 network for this.-Use OPT1, connected to the DC LAN which accepts traffic to/from pfsense 01/02.
But, what is the best way to handle OPT1?
All of the 10.0.0.1/24 traffic needs to keep flowing to the DC LAN for now. Once everything is up and running, I'll move some services from those servers to the new ones in the cabinet.
Should I just create a new network on pfsense 01 and allow that on pfsense 03 OPT1? From there, route that to the 192.168.1.1/24 servers?
-
Yes, it should work and not present an issue because you know and control all the things connected to it. It's just ugly because there's no real separation.
-
Should I start a new question? I have one day left to get this figured out. I understand what the plan is now but not quite how to go about the procedure.
Since pfsense 01 is staying in place for now, I will need to slowly transition that traffic away from those servers using 10.0.0.x/24 on pfsense 01 to the new pfsense 03.
However, I need to keep using the 10.0.0.1/24 network on pfsense 03 otherwise, everything will become a huge mess of having to change IPs etc.
Is it possible to slowly forward 10.0.0.1/24 traffic from pfsense 01 to on pfsense 03 while keeping 10.0.0.1/24 on the LAN side of pfsense 03?
-
@lewis said in Multi LAN networks to one pfsense:
Is it possible to slowly forward 10.0.0.1/24 traffic from pfsense 01 to on pfsense 03 while keeping 10.0.0.1/24 on the LAN side of pfsense 03?
No, not at all easily as I understand your current setup.
You are going to setup a transport subnet on the VLAN that exists between pf1 and pf3 and a static route to the pf3 LAN subnet on pf1. Then change the port forwards for the servers to the new addresses on the pf3 LAN.
The static route would be invalid if the pf3 LAN subnet is also 10.0.0.0/24.One way I can see you could do this would be using multiple VIPs on the pf3 OPT interface to correspond with the /29s on pf1. Then you could use 1:1 NAT rules in pf1 to forward all the traffic to pf3 and from there forwards again the the LAN side servers.
Because port forwards take precedence over 1:1 NAT rules traffic would continue to use the servers directly behind pf1 until you disable them. So you could so that one at a time and check connectivity.
The transport subnet would need to be large enough to accommodate all the IPs you need to forward.Steve
-
Maybe I have no choice but to start using a 192.168.x.x network then. Sounds like I'm boxed in, at least in my level of knowledge here.
I very much appreciate your input. I'll look at your suggestion as well.
-
Routing to a different subnet gives you a lot more options later. That's what I would do.
You don't have to use anything from 192.168.0.0/16. You could just use a different subnet from 10.0.0.0/8 as long as you're not using it anywhere already.
However using something from 192.168 is not a bad idea because it's very obviously different. Much harder to make mistakes or misread something. Just don't use 192.168.1.0/24to avoid conflicts in the future.
Steve
-
I'm playing around trying different ideas and maybe I'm just tired but...
pfsense 03
WAN - Internet connection
DCLAN - Will handle traffic from existing 10.0.0.1/24
OPT1(LAN) 10.0.0.1/24
OPT2(not used)Again, maybe I'm not thinking right because I've been at this all weekend but here is the idea.
Routing is routing right? If I look at this as a routing issue, even if pfsense 01 is using 10.0.0.1/24, it doesn't know anything about what's behind pfsense 03.
Pfsense 03 is NAT'ing so as long as I use another IP to get the traffic from pfsense 01 to pfsense 03, this should work.
I mean, public connections don't know anything about the LAN side so that's where the trickery could come in.
As I move servers from the LAN side of pfsense 01, couldn't I put something like 172.16.1.x on the DCLAN of pfsense 03 and forward incoming NAT'ed connections to that network where pfsense 03 would NAT it to its LAN side, 10.0.0.1/24?
LOL, sorry, my brain is fried. Maybe it's something, maybe it's not.
-
Yes, you could do that. That's what I was suggesting above with the 1:1 NAT rules.
Since you have multiple public IPs on pf1 though you're going to need the same number of IPs on pf3 DCLAN in order to be able to forward the incoming connections to the right servers. Using 1:1NAT on pf1 just makes that easier.Steve
-
This is where my limited knowledge is not helping.
I'm not understanding the part where we're talking about adding VIP's or using a transport network.
I have to clarify to make sure I didn't add complexity that doesn't need to be in the mix.
On pfsense 01, I have only two networks, WAN and LAN.
I have two /29 public IPs set in the Virtual IPs.
I've whittled the use down to one /29 and three IPs being used as part of this change. Two of them are each being used with the HAproxy server routing to a few load balanced servers.On the LAN side, there is only 10.0.0.1/24 and it's DHCP server. Nothing more complex.
One way I can see you could do this would be using multiple VIPs on the pf3 OPT interface to correspond with the /29s on pf1. Then you could use 1:1 NAT rules in pf1 to forward all the traffic to pf3 and from there forwards again the the LAN side servers.
All of this has to be on the DCLAN side as there isn't enough bandwidth to handle the traffic if I sent it to the WAN side of pfsense 03.
Pfsense 01 has to send the traffic to the DCLAN interface of pfsense 03 which would have a private IP.
This is where it's not clear what we're talking about, a transport, a full network or one VIP for each server being redirected?
The public /29's on pfsense 01 and pfsense 03 are completely different of course.
To try and visualize what you're proposing,...
One to one;
pfsense 01
Add 172.16.1.x/24 IPs as needed (VIPs?) per server being moved from this firewalls NAT to pfsense 03.Meaning, a LAN server on pfsense 01 that is 10.0.0.50 for example, would have its rule changed to point to 172.16.1.50 which in turn would be sent to pfsense 03's DCLAN interface and then routed to the new 10.0.0.50 server on the LAN side?
How would DCLAN be configured?Or transport;
Add a new network on pfsense 01, say 172.16.1.1/30 that would be used to forward traffic to DCLAN on pfsense 03?
This means pfsense 03 would need a 172.16.x.x/? network on it's DCLAN NIC?
In this method, similar to above, I would change rules to point from 10.0.0.x servers to point to 172.16.x.x servers but use just one transport to get those to pfsense 03?I feel like I'm complicating all this but maybe once I understand the smaller details, it will all suddenly make sense.
-
If you need to use 10.0.0.0/24 for the LAN side of pf3 then you cannot route the traffic over a transport subnet. You must use NAT.
That means you must have a VIP on pf3 on it's DCLAN interface for each server you are forwarding to.
pf1 only needs a single VIP in the new subnet so it can send traffic on it.It's an ugly setup as we have discussed. But it should work.
Steve
-
I wanted to report back that it's all working but I'm stuck already :(.
Before I bring this to the DC, I need to test it by isolating it on its own.
I have pfsense 03 with its own Internet connection.
I set one IP, 192.168.1.3/24 on DCLAN to simulate an external network connection trying to get to the LAN network servers.I set this IP because I already have 192.168.1.1/24 and even 10.0.0.1/24 used on my network so wanted just a single IP on that interface.
Since I only have one IP set on the DCLAN interface, I'm wanting to set up some port forwards.
I can get to the pfsense firewall but I cannot get to anything behind the LAN switch. For example, I have something at port 8006 so I port forwarded 192.168.1.3, port 7071 to 10.0.0.74, port 8006.
I can see the hit but no luck from PC 192.168.1.205.
DCLAN tcp 192.168.1.205:32491 -> 10.0.0.71:8006 (192.168.1.3:7071) FIN_WAIT_2:FIN_WAIT_2 5 / 4 654 B / 172 B
-
You see a state on the LAN showing it leave?
It looks like it's allowing the traffic in on DCLAN and opening the correct state with NAT there.
The server may not allow that traffic if it's new and hasn't been configured for that.
You will need a gateway on DCLAN for traffic from external sources so incoming states are tagged reply-to correctly. Otherwise it will use the default route which will be the pf3 WAN (I expect).
Steve
