Using pfsense with multiple WANs
-
I don't care if I use gre or simply find a way to move servers/services one at a time using just one or two rules and later forward from pf02 to where they would then be, pf01. I wondered about using a virtual IP or set of VIP's on pf02 to accomplish this but I can't figure out how to set up rules for that sort of scenario.
-
@lewis said in Multi LAN networks to one pfsense:
LOL, this is confusing. That's been the point of this post all along. What happened?
Not sure what the point of this thread is, your drawings don't make that much sense, and you have confusing vips and lans and overlaps.
So want to bridge over a GRE tunnel - not even sure that is possible with pfsense.. I haven't done any sort of deep dive into the thread - but you sure @stephenw10 understands your trying to do a L2 bridge across your GRE?
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.03 | Lab VMs 2.7.2, 24.03 -
Hmm, that's not the direction I expect that to work in.
Currently you have a bunch of servers behind pf02 that are accessed using public IPs via port forwards in pf02.
You need to move those servers to be behind pf01 instead as I understand it?
So what you need to do is change the port forwards in pf02 to point to the pf01 GRE tunnel IP instead. And then on pf01 add port forwards from the GRE tunnel IP to the servers in the pf01 LAN.
I said before that you won't be able to connect directly from pf02 LAN subnet to the servers behind pf01 because the source IPs would conflict. The same would be true the other way.
If you really need to do that you would need to use outboubnd NAT to mask the source IP and use the GRE tunnel IP directly as the destination.Also just to reiterate; you don't actually need the GRE tunnel for just two pfSense instances like that but as soon as you add another (or more) you would so it make sense to do them all the same.
Steve
-
@johnpoz said in Multi LAN networks to one pfsense:
@lewis said in Multi LAN networks to one pfsense:
LOL, this is confusing. That's been the point of this post all along. What happened?
Not sure what the point of this thread is, your drawings don't make that much sense, and you have confusing vips and lans and overlaps.
So want to bridge over a GRE tunnel - not even sure that is possible with pfsense.. I haven't done any sort of deep dive into the thread - but you sure @stephenw10 understands your trying to do a L2 bridge across your GRE?
I don't think it's that confusing other than you've not been part of the entire and yes, very long thread.
Steve suggested using gre to accomplish what I need but I had a couple of mistakes in there that I've since fixed.
There is no point in re-explaining the whole thing since its all up above and just adds to a very long thread.
The short of it is, I have two pf firewalls, both are using a LAN of 10.0.0.1/24. I cannot change the LAN ips, I have to migrate servers/services from the first pf to the second one and only have a VLAN to do it with.
-
@johnpoz
The problem we are trying to address is this:Muliple pfSense instances with servers behind them all configured to port forward public IPs to those servers. All instances share a common LAN subnet.
Those servers are being physically relocated to all be behind a new pfSense install.
The servers much remain in the same private subnet.
Because of reasons the public IPs cannot be routed to the new pfSense instance so traffic to each public IP has to go through the existing installs. -
I think at this point, the simplest way to do this would be as follows.
Use the public interface of pf02 to get at the servers/services on its LAN. Once moved to the new network, forward the traffic from pf02 to pf01 over the VLAN and to 10.100.0.1/24.
So, a rule on pf02 that is say port forwarding ports 80/443 to 10.0.0.120 on its LAN would be changed to 10.100.0.1.
On pf01, I would just have rules for each service as I do now for LAN on pf02.
In my mind, without the amount of knowledge you guys have, that sounds like it would work.
-
Yup, that will work.
But only for pf02 because it's acting as the gateway for the pf01 DCLAN interface. If you try to do the same for pf03 it will fail. -
@stephenw10 said in Multi LAN networks to one pfsense:
Yup, that will work.
But only for pf02 because it's acting as the gateway for the pf01 DCLAN interface. If you try to do the same for pf03 it will fail.Yes, I'll have to figure out how to do the same with the other server that will become pf03 sending traffic to ph01.
At least that would allow me to start working on this asap since I've kinda lost a week now.
-
@stephenw10 said in Multi LAN networks to one pfsense:
Because of reasons the public IPs cannot be routed to the new pfSense instance so traffic to each public IP has to go through the existing installs.
That just seems like a MESS... If your going to move servers, then take the down time, move them and bring them up on new public IPs if you can not route the old IP network to the new location..
The servers much remain in the same private subnet.
Why? That makes no sense either..
-
Mmm, the situation is.... sub-optimal!
-
If you read the entire thread where I mentioned all of the reasons why I have to do this crazy stuff, then you would stop saying 'why do it this way'. It certainly isn't something I wanted to have to do but am stuck having to.
It is a mess, a huge one and only one person to get it all done.
Just got to my desk to try to work on this again.
I had to add a public IP to the esx host then allow the pf01 IP access to the ports I need to import vms to the new network.Then I can finally have all the traffic that's supposed to go to LAN on pf02 going to DCLAN>LAN on pf01.
When I get to having moved all of those vms to their new servers, the last thing is figuring out how pf03 but really not worried about that just yet. pf02 is the most important.
I have a severe deadline to move everything off of a bunch of hosts on pf02 so that is my priority.
Anyhow, now that I have a copy of 10.0.0.120 onto the pf01 network, I have to re-route that from pf02.
-
So you need the VMs behind each to be able to connect directly? During production, not just in order to move them?
I think I've said several times that's not possible with this scheme. They are using the same subnet so require NATing between them. The only way to do it would be to outbound NAT that traffic from pf02 so the servers in pf01 see a different source IP. The servers in the pf02 LAN would need to use a different target IP though.
Reviewing this whole ticket though it looks like we went off down the path of routing the traffic between the pfSense instances quite early.
What was the reason for not simply keeping the existing LAN side subnet and using it directly behind the new pfSense instance. One large layer 2 still?
Obviously the IP addresses would need to remain and not be duplicated. And you have multiple routers on the same subnet so asymmetry is highly likely. Each server VM would need to keep it's existing gateway and could only accept connections from that gateway.
I suspect that was the show stopper here?Steve
-
******** LETS START FROM HERE ********
While this might be too much of a post for a forum, it's also an interesting challenge. it seems. Anyone could be handed this mess and told what they can and cannot change. You'd have to work withing those constraints. It sucks, a lot. I've been at this for countless hours now.
I'm not sure how to answer your question since the thread has kinda gone all over the place. Mainly it went to using GRE which got me really sideways. The gre tunnel seems to be completed and ready to use though.
And while the stuff is in a DC, it's not really their fault. They cannot change their methods/policies for just one customer and I know them well enough to know they care about their customers and would help if they could.
The problem is the new hardware is in a different part of the DC. The only way to communicate between them is either public IPs or the VLAN that I've got across both networks.
When installing the new hardware, I had to use the 10.0.0.1/24 LAN side. This simply cannot change as it would mess up a ton of custom programming and processes talking between servers. It would have been super easy to just go with a 192.168.1.1/24 for example but it cannot be done.At this point in this question, we can forget about my needing access from pf01 LAN to pf02 LAN in order to move all of the vms from one network to the other over the VLAN. I've solved that by simply using a public IP on each host I'll need access to.
Most of the vms are redundant web servers so I can easily move them without disrupting services. The ones that aren't will suffer a little down time, no issues.
I'm left with two main things to solve
1: Everything on pf02 LAN uses this one main DB server at 10.0.0.10. It has to be up always. There won't be any way to split that traffic meaning I cannot take it down but I've prepared a new server on the pf01 LAN. No public access is required for this DB but all internal servers need access to it.
The plan is to export everything from the current DB server to the new server. During that time, we'll have to shut down the live DB server to prevent anything from changing between the two databases.
But that's a problem. The servers on the pf02 LAN need access to the DB and so will the ones I've moved to pf01. I cannot have data being read/written between two servers as that will split the data.
What ever way I use, it has to keep the servers on pf02 seeing the DB at 10.0.0.10 and as they migrate over to pf01, they will see it as 10.0.0.10 on that LAN. Eventually, all servers get migrated and that's not a problem.
Since I cannot route LAN to LAN, it seems my only option will be to set up an outgoing rule that will take 10.0.010 DB traffic over the WAN to the new network and onto the new 10.0.0.10 DB Server. Ugly. Can this be done using the gre or VIP or something along those lines?
2: Final part is to redirect all traffic from public IPs on pf02 on to pf01. Later, do the same on another server I've tried not to talk about to prevent confusion, pf03. The difference with pf03 is that it doesn't have any LAN, it's just a firewall with public IPs to one service behind it which just needs to be re-directed to pf01. In this case, because the public IPs cannot be moved from that location to the location where the new servers are.
I hope this helps to bring this back to something more reasonable :).
-
Here is an updated version of the image without gre.
NOTE that pf02 traffic goes to pf01 DCLAN, not LAN. Therefore, there should be a way to do this since it's not 10.0.0.x/24 going to 10.0.0.x/24 directly.
-
Dude you still have the same network on these servers.
So 10.0.0.10 wants to talk to 10.0.0.11 - that is on the other pfsense.. Why would it send traffic anywhere? .11 is on its local network.
For you to do what you want is an extended vlan, this is normally done over dark fiber between DC or Building. Where its actually the same L2.
Extending the L2 is normally how you would do what your wanting. So the L2 is actually the same L2.
If they are going to be different L2s with the same L3 on them - then you have to do some natting so say when .10 wants to talk to .11 (this is really a vip) on pfX, and sends it to what it thinks is say 10.0.1.11 on pfY.. When then forwards it to its lan 10.0.0.11
It would be a real freaking mess doing that.. You want to extend the L2..
You clearly have some connection between pf1 and pf2 - why can this not be a bridge that bridges this L2 into just 1 extended L2 network? Your 10.0.0/24 network..
-
@johnpoz said in Multi LAN networks to one pfsense:
Dude you still have the same network on these servers.
So 10.0.0.10 wants to talk to 10.0.0.11 - that is on the other >pfsense.. Why would it send traffic anywhere? .11 is on its
local network.Not sure what the 'dude' comment is about. I already know I have a problem. It's why I posted the question, because it's beyond my knowledge of how this could be accomplished and thought some higher level network folks might have some suggestions.
For you to do what you want is an extended vlan, this is normally >done over dark fiber between DC or Building. Where its actually >the same L2.
Again, this is in a data center. I don't know own it, I cannot change how they do things. All I can do is try to find a way to solve the problem. What I do have is a VLAN between the networks. I cannot get anything else, that's what I have.
You clearly have some connection between pf1 and pf2 - why can >this not be a bridge that bridges this L2 into just 1 extended L2 >network? Your 10.0.0/24 network..
It can be anything that will work. I'm asking for ideas, thoughts on how I can solve this and I have limited knowledge since I've never had to get into this kind of situation before.
Again... with LIMITED knowledge...
I have a VLAN so there is a way to communicate between the two networks. What ever can be done with a VLAN is all I have.I have two interfaces on pf02, WAN and LAN, nothing else.
I have three interfaces on pf02, WAN, DCLAN (connected to the DC VLAN) and LAN, connected to a switch and all servers behind that are 10.0.0.1/24.I cannot change the IPs from 10.0.0.x/24 otherwise, this would be a snap. I have to migrate servers over to this new network but I also have to keep routing the public traffic from pf02 to pf01 now.
Asking me why I'm doing anything doesn't make sense. I'm not doing anything, I'm here asking you guys, the pros, for ideas, suggestions and even some help on how this could be accomplished.
Happy to and have tried what is being suggested but I'm out of my league on this so please be patient with me :).
-
@lewis said in Multi LAN networks to one pfsense:
I have a VLAN so there is a way to communicate between the two networks
So you have a L2 connection between them - then just join them together so you have the same L2.. Problem solved..
https://docs.netgate.com/pfsense/en/latest/interfaces/qinq.html
Why can it not just be the same L2? This is the same DC and your moving to a different rack.. They can connect whatever L2 you want to any Rack..
The plan is to install hardware in one single cabinet to consolidate the whole thing
Your in a DC... They will connect vlan X, to whatever cabinet you want... So your L2 network you have in isle 3, rack 2 can also exist in isle 6, rack 7 building 2, etc.. Just ask them!
This something that happens every single day around the globe in every DC...
Seems more like you have a simple layer 1 problem - where the wires go, then anything else. Be it vlans you use behind pfsense, or your wan connections. Ask the DC to move your connections around so you can put them all in 1 cabinet, etc. Or same cage or same room, etc..
-
I'm not sure how to answer because some of this is simply beyond my knowledge, isn't something I've had to deal with before and is why I posted here looking for input.
The main problem is that I have two firewalls, one on each network and both handle the 10.0.0.1/24 LAN.
When all my gear was in one place, all of it communicated between each other via the VLAN I was given and on 10.0.0.1/24.
I was given rack space in another section of the DC which is not normally connected to the first area. I got them to give me an Ethernet drop between the new section and the old section so I could have the VLAN between both. It seemed to take a lot of talking just to get that and I know they will not change their normal methods just for me. Not sure what else I could ask them for.
However, since both firewalls handle a 10.0.0.1/24, that's where the problem comes in.
It seemed like on the VLAN side, I could simply move machines from one network to another since the VLAN acts kind of like a LAN. However, that's not the case. Servers aren't seeing each other because they each have their own gateway on their own network.
-
Anyhow, the point is yes, there is a VLAN so yes, there should be a way to do this but I've not found a way and so far, this thread has not either as I'm happy to try anything at this point.
I still have the gre setup configured, should I get rid of all that?
Also, from the start, even with the VLAN, I'm not able to communicate between the two networks. I thought it might be as simple as needing some rules to allow pf01 LAN devices to communicate through the DCLAN (VLAN) interface but I've yet to find documentation to explains how I can create rules for that and all the ones I've tried never worked.
-
@lewis said in Multi LAN networks to one pfsense:
The main problem is that I have two firewalls, one on each network and both handle the 10.0.0.1/24 LAN.
Yeah so you make 10.0.0/24 a common L2 network.. And use .1 on 1 and .2 on the other and on your hosts set the gateway to which gateway you want them to use.
But to be honest, why do you have multiple firewalls.. You only need 1..
I would suggest you call the DC, tell them you want to consolidate everything and you want your connections moved to where ever your moving everything too.
Your trying to solve a problem that your creating because you just haven't asked them to move connection X over to different port on the patch panel where your moving everything too..
