Using pfsense with multiple WANs
-
Should I start a new question? I have one day left to get this figured out. I understand what the plan is now but not quite how to go about the procedure.
Since pfsense 01 is staying in place for now, I will need to slowly transition that traffic away from those servers using 10.0.0.x/24 on pfsense 01 to the new pfsense 03.
However, I need to keep using the 10.0.0.1/24 network on pfsense 03 otherwise, everything will become a huge mess of having to change IPs etc.
Is it possible to slowly forward 10.0.0.1/24 traffic from pfsense 01 to on pfsense 03 while keeping 10.0.0.1/24 on the LAN side of pfsense 03?
-
@lewis said in Multi LAN networks to one pfsense:
Is it possible to slowly forward 10.0.0.1/24 traffic from pfsense 01 to on pfsense 03 while keeping 10.0.0.1/24 on the LAN side of pfsense 03?
No, not at all easily as I understand your current setup.
You are going to setup a transport subnet on the VLAN that exists between pf1 and pf3 and a static route to the pf3 LAN subnet on pf1. Then change the port forwards for the servers to the new addresses on the pf3 LAN.
The static route would be invalid if the pf3 LAN subnet is also 10.0.0.0/24.One way I can see you could do this would be using multiple VIPs on the pf3 OPT interface to correspond with the /29s on pf1. Then you could use 1:1 NAT rules in pf1 to forward all the traffic to pf3 and from there forwards again the the LAN side servers.
Because port forwards take precedence over 1:1 NAT rules traffic would continue to use the servers directly behind pf1 until you disable them. So you could so that one at a time and check connectivity.
The transport subnet would need to be large enough to accommodate all the IPs you need to forward.Steve
-
Maybe I have no choice but to start using a 192.168.x.x network then. Sounds like I'm boxed in, at least in my level of knowledge here.
I very much appreciate your input. I'll look at your suggestion as well.
-
Routing to a different subnet gives you a lot more options later. That's what I would do.
You don't have to use anything from 192.168.0.0/16. You could just use a different subnet from 10.0.0.0/8 as long as you're not using it anywhere already.
However using something from 192.168 is not a bad idea because it's very obviously different. Much harder to make mistakes or misread something. Just don't use 192.168.1.0/24to avoid conflicts in the future.
Steve
-
I'm playing around trying different ideas and maybe I'm just tired but...
pfsense 03
WAN - Internet connection
DCLAN - Will handle traffic from existing 10.0.0.1/24
OPT1(LAN) 10.0.0.1/24
OPT2(not used)Again, maybe I'm not thinking right because I've been at this all weekend but here is the idea.
Routing is routing right? If I look at this as a routing issue, even if pfsense 01 is using 10.0.0.1/24, it doesn't know anything about what's behind pfsense 03.
Pfsense 03 is NAT'ing so as long as I use another IP to get the traffic from pfsense 01 to pfsense 03, this should work.
I mean, public connections don't know anything about the LAN side so that's where the trickery could come in.
As I move servers from the LAN side of pfsense 01, couldn't I put something like 172.16.1.x on the DCLAN of pfsense 03 and forward incoming NAT'ed connections to that network where pfsense 03 would NAT it to its LAN side, 10.0.0.1/24?
LOL, sorry, my brain is fried. Maybe it's something, maybe it's not.
-
Yes, you could do that. That's what I was suggesting above with the 1:1 NAT rules.
Since you have multiple public IPs on pf1 though you're going to need the same number of IPs on pf3 DCLAN in order to be able to forward the incoming connections to the right servers. Using 1:1NAT on pf1 just makes that easier.Steve
-
This is where my limited knowledge is not helping.
I'm not understanding the part where we're talking about adding VIP's or using a transport network.
I have to clarify to make sure I didn't add complexity that doesn't need to be in the mix.
On pfsense 01, I have only two networks, WAN and LAN.
I have two /29 public IPs set in the Virtual IPs.
I've whittled the use down to one /29 and three IPs being used as part of this change. Two of them are each being used with the HAproxy server routing to a few load balanced servers.On the LAN side, there is only 10.0.0.1/24 and it's DHCP server. Nothing more complex.
One way I can see you could do this would be using multiple VIPs on the pf3 OPT interface to correspond with the /29s on pf1. Then you could use 1:1 NAT rules in pf1 to forward all the traffic to pf3 and from there forwards again the the LAN side servers.
All of this has to be on the DCLAN side as there isn't enough bandwidth to handle the traffic if I sent it to the WAN side of pfsense 03.
Pfsense 01 has to send the traffic to the DCLAN interface of pfsense 03 which would have a private IP.
This is where it's not clear what we're talking about, a transport, a full network or one VIP for each server being redirected?
The public /29's on pfsense 01 and pfsense 03 are completely different of course.
To try and visualize what you're proposing,...
One to one;
pfsense 01
Add 172.16.1.x/24 IPs as needed (VIPs?) per server being moved from this firewalls NAT to pfsense 03.Meaning, a LAN server on pfsense 01 that is 10.0.0.50 for example, would have its rule changed to point to 172.16.1.50 which in turn would be sent to pfsense 03's DCLAN interface and then routed to the new 10.0.0.50 server on the LAN side?
How would DCLAN be configured?Or transport;
Add a new network on pfsense 01, say 172.16.1.1/30 that would be used to forward traffic to DCLAN on pfsense 03?
This means pfsense 03 would need a 172.16.x.x/? network on it's DCLAN NIC?
In this method, similar to above, I would change rules to point from 10.0.0.x servers to point to 172.16.x.x servers but use just one transport to get those to pfsense 03?I feel like I'm complicating all this but maybe once I understand the smaller details, it will all suddenly make sense.
-
If you need to use 10.0.0.0/24 for the LAN side of pf3 then you cannot route the traffic over a transport subnet. You must use NAT.
That means you must have a VIP on pf3 on it's DCLAN interface for each server you are forwarding to.
pf1 only needs a single VIP in the new subnet so it can send traffic on it.It's an ugly setup as we have discussed. But it should work.
Steve
-
I wanted to report back that it's all working but I'm stuck already :(.
Before I bring this to the DC, I need to test it by isolating it on its own.
I have pfsense 03 with its own Internet connection.
I set one IP, 192.168.1.3/24 on DCLAN to simulate an external network connection trying to get to the LAN network servers.I set this IP because I already have 192.168.1.1/24 and even 10.0.0.1/24 used on my network so wanted just a single IP on that interface.
Since I only have one IP set on the DCLAN interface, I'm wanting to set up some port forwards.
I can get to the pfsense firewall but I cannot get to anything behind the LAN switch. For example, I have something at port 8006 so I port forwarded 192.168.1.3, port 7071 to 10.0.0.74, port 8006.
I can see the hit but no luck from PC 192.168.1.205.
DCLAN tcp 192.168.1.205:32491 -> 10.0.0.71:8006 (192.168.1.3:7071) FIN_WAIT_2:FIN_WAIT_2 5 / 4 654 B / 172 B
-
You see a state on the LAN showing it leave?
It looks like it's allowing the traffic in on DCLAN and opening the correct state with NAT there.
The server may not allow that traffic if it's new and hasn't been configured for that.
You will need a gateway on DCLAN for traffic from external sources so incoming states are tagged reply-to correctly. Otherwise it will use the default route which will be the pf3 WAN (I expect).
Steve
-
Ah, darn, then I'm out of luck. I'm already using those networks and wanted to set something up that's isolated yet gives me access from the existing 192.168.1.1/24 network so I can transfer files, set up the servers etc.
The servers on the LAN Switch side are accessible. I have a laptop connected to the LAN Switch and it's able to see everything on that 10.0.0.1/24 network. -
Hmm, not sure what you're saying there. Those subnets look fine.
I'm just saying that pf3 needs to treat the DCLAN connection as a WAN so that it can receive the incoming connections from pf1 and send the replies back that way. In order to do that the DCLAN interface must have pf1 as a gateway.
Steve
-
Basically, I have a simulated condition before it goes to the data center.
I have a pfsense here at my home which has the LAN of 192.168.1.1/24. That pfsense also has two other networks on different interfaces, OPT1 10.0.0.1/24 and OPT2 192.168.254.1.
I've isolated the hardware onto it's own so is using pfsense 03.
It has WAN to Internet.
It has DCLAN with just 192.168.1.3/24 set on it so I can reach it from my network so I have a cable connected to one of the LAN Switch ports and my LAN switch.
It has 10g1LAN which is connected to the LAN Switch and 10g2OPT2 which is not in use.
All devices on the LAN Switch have Internet access and all devices/servers can communicate together.
This is where I wanted to simulate the DCLAN connection and is what I'm playing around with to understand before this goes to the DC.
My options seem to be, set up a transport link between my own pfsense and pfsense 03. However, since I'm not coming in from the Internet, this seems to be a waste of time. I just need to gain access to the LAN Switch side hardware so I can configure all of the hardware. Hence why I just assigned it a fixed IP that wasn't in use, 192.168.1.3. However, that allows me access to the pfsense 03 but nothing on the LAN Switch which is what I need.
I hope I'm explaining all this well. It's amazing how much text there is in writing things where if you are with someone and they are showing you something face to face, you pick it up in minutes.
-
So how are you testing? What's not working?
What you are describing sounds like it should work.
Set up a port forward on your home pfSense (substituting for pf1) from it';s public IP to the pf3 DCLAN IP.
Set up a forward in pf3 from there to the server on it's WAN.If you connect externally that should work as long as it's configured correctly.
Steve
-
Well, I don't know what's not working :). Probably something small I'm not seeing.
I can connect to 192.168.1.3 from any client on the 192.168.1.1/24 network because the IP is basically just another device on the LAN.
I can only reach the pfsense 03 device however, nothing behind it.For example, trying to reach a proxmox server which is at 10.0.0.71:8006. I added this rule to allow 192.168.1.x to reach it.
-
Ok so same as before; that should work from another host in the DCLAN subnet however pf3 is configured because only routing between local subnets is required.
For other sources the DCLAN interface in pf3 needs to be configured correctly (with a gateway).Check the state table in pf3 for a LAN state when you are trying to connect. If it's open and shows traffic it must be the server either refusing the connection or replying incorrectly.
Steve
-
So far, everything works in terms of giving DCLAN the IP of 192.168.1.3. I'm using port forwarding to reach everything.
I have some hardware problems I need to solve then the last step will be trying to set up a transport and simulate the setup once installed at the DC.
-
So it's all working as expected in your home test?
-
Hi, not done yet, things are still weird.
Something keeps happening with the firewall that keeps logging me out. When I connect directly to the pf03 10.0.0.1/24 network using a laptop everything works fine.
When I connect from the 192.168.1.1/24 network, things get weird. Every time I log into something, I get logged out of everything else in the other tabs.
I know it's because I'm forwarding non standard ports all over the place but it's making it really hard to get anything done. Today, I'll set up a PC directly connected so I don't have to use this crazy port forward stuff, then I can move on to the last step which is creating this transport network. -
Yeah, I've hit that a few times if you have several tabs open to the same IP or hostname where only the destination port is different. Conflicting sessions override each other. I've just used separate browsers to prevent that before.
Steve
