Using pfsense with multiple WANs
-
From cli on both pf, Still cannot ping between 10.102.0.1 and 10.102.0.2 but I assume the next thing is rules.
My first planned test is like this.
From 10.0.0.71 on the pf01 LAN side, I want to reach 10.0.0.120 behind the LAN on pf02. Any port.
-
Yup, the tunnel looks good now, states both ways on both sides.
If you have rules on the GRE interfaces to allow it you should be able to ping between them.
Steve
-
@stephenw10 said in Multi LAN networks to one pfsense:
Yup, the tunnel looks good now, states both ways on both sides.
If you have rules on the GRE interfaces to allow it you should be able to ping between them.
Steve
To save 10 messages and to make sure I don't accidentally take something down, can you tell me what I would do at this point.
For my test, from 10.0.0.71 on the pf01 LAN side, I want to reach 10.0.0.120 behind the LAN on pf02. Any port.
What rule do I need on pfxx and is the rule on the LAN or GRE side? With this one example, I should be able to start adding more rules safely.
-
@lewis said in Multi LAN networks to one pfsense:
10.0.0.71 on the pf01 LAN side, I want to reach 10.0.0.120 behind the LAN on pf02.
I just don't see how you think that could happen? Those are the same network..
-
@johnpoz said in Multi LAN networks to one pfsense:
@lewis said in Multi LAN networks to one pfsense:
10.0.0.71 on the pf01 LAN side, I want to reach 10.0.0.120 behind the LAN on pf02.
I just don't see how you think that could happen? Those are the same network..
LOL, yes, if you just walked into this without reading the thread, then you will be confused.
-
I don't care if I use gre or simply find a way to move servers/services one at a time using just one or two rules and later forward from pf02 to where they would then be, pf01. I wondered about using a virtual IP or set of VIP's on pf02 to accomplish this but I can't figure out how to set up rules for that sort of scenario.
-
@lewis said in Multi LAN networks to one pfsense:
LOL, this is confusing. That's been the point of this post all along. What happened?
Not sure what the point of this thread is, your drawings don't make that much sense, and you have confusing vips and lans and overlaps.
So want to bridge over a GRE tunnel - not even sure that is possible with pfsense.. I haven't done any sort of deep dive into the thread - but you sure @stephenw10 understands your trying to do a L2 bridge across your GRE?
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.03 | Lab VMs 2.7.2, 24.03 -
Hmm, that's not the direction I expect that to work in.
Currently you have a bunch of servers behind pf02 that are accessed using public IPs via port forwards in pf02.
You need to move those servers to be behind pf01 instead as I understand it?
So what you need to do is change the port forwards in pf02 to point to the pf01 GRE tunnel IP instead. And then on pf01 add port forwards from the GRE tunnel IP to the servers in the pf01 LAN.
I said before that you won't be able to connect directly from pf02 LAN subnet to the servers behind pf01 because the source IPs would conflict. The same would be true the other way.
If you really need to do that you would need to use outboubnd NAT to mask the source IP and use the GRE tunnel IP directly as the destination.Also just to reiterate; you don't actually need the GRE tunnel for just two pfSense instances like that but as soon as you add another (or more) you would so it make sense to do them all the same.
Steve
-
@johnpoz said in Multi LAN networks to one pfsense:
@lewis said in Multi LAN networks to one pfsense:
LOL, this is confusing. That's been the point of this post all along. What happened?
Not sure what the point of this thread is, your drawings don't make that much sense, and you have confusing vips and lans and overlaps.
So want to bridge over a GRE tunnel - not even sure that is possible with pfsense.. I haven't done any sort of deep dive into the thread - but you sure @stephenw10 understands your trying to do a L2 bridge across your GRE?
I don't think it's that confusing other than you've not been part of the entire and yes, very long thread.
Steve suggested using gre to accomplish what I need but I had a couple of mistakes in there that I've since fixed.
There is no point in re-explaining the whole thing since its all up above and just adds to a very long thread.
The short of it is, I have two pf firewalls, both are using a LAN of 10.0.0.1/24. I cannot change the LAN ips, I have to migrate servers/services from the first pf to the second one and only have a VLAN to do it with.
-
@johnpoz
The problem we are trying to address is this:Muliple pfSense instances with servers behind them all configured to port forward public IPs to those servers. All instances share a common LAN subnet.
Those servers are being physically relocated to all be behind a new pfSense install.
The servers much remain in the same private subnet.
Because of reasons the public IPs cannot be routed to the new pfSense instance so traffic to each public IP has to go through the existing installs. -
I think at this point, the simplest way to do this would be as follows.
Use the public interface of pf02 to get at the servers/services on its LAN. Once moved to the new network, forward the traffic from pf02 to pf01 over the VLAN and to 10.100.0.1/24.
So, a rule on pf02 that is say port forwarding ports 80/443 to 10.0.0.120 on its LAN would be changed to 10.100.0.1.
On pf01, I would just have rules for each service as I do now for LAN on pf02.
In my mind, without the amount of knowledge you guys have, that sounds like it would work.
-
Yup, that will work.
But only for pf02 because it's acting as the gateway for the pf01 DCLAN interface. If you try to do the same for pf03 it will fail. -
@stephenw10 said in Multi LAN networks to one pfsense:
Yup, that will work.
But only for pf02 because it's acting as the gateway for the pf01 DCLAN interface. If you try to do the same for pf03 it will fail.Yes, I'll have to figure out how to do the same with the other server that will become pf03 sending traffic to ph01.
At least that would allow me to start working on this asap since I've kinda lost a week now.
-
@stephenw10 said in Multi LAN networks to one pfsense:
Because of reasons the public IPs cannot be routed to the new pfSense instance so traffic to each public IP has to go through the existing installs.
That just seems like a MESS... If your going to move servers, then take the down time, move them and bring them up on new public IPs if you can not route the old IP network to the new location..
The servers much remain in the same private subnet.
Why? That makes no sense either..
-
Mmm, the situation is.... sub-optimal!
-
If you read the entire thread where I mentioned all of the reasons why I have to do this crazy stuff, then you would stop saying 'why do it this way'. It certainly isn't something I wanted to have to do but am stuck having to.
It is a mess, a huge one and only one person to get it all done.
Just got to my desk to try to work on this again.
I had to add a public IP to the esx host then allow the pf01 IP access to the ports I need to import vms to the new network.Then I can finally have all the traffic that's supposed to go to LAN on pf02 going to DCLAN>LAN on pf01.
When I get to having moved all of those vms to their new servers, the last thing is figuring out how pf03 but really not worried about that just yet. pf02 is the most important.
I have a severe deadline to move everything off of a bunch of hosts on pf02 so that is my priority.
Anyhow, now that I have a copy of 10.0.0.120 onto the pf01 network, I have to re-route that from pf02.
-
So you need the VMs behind each to be able to connect directly? During production, not just in order to move them?
I think I've said several times that's not possible with this scheme. They are using the same subnet so require NATing between them. The only way to do it would be to outbound NAT that traffic from pf02 so the servers in pf01 see a different source IP. The servers in the pf02 LAN would need to use a different target IP though.
Reviewing this whole ticket though it looks like we went off down the path of routing the traffic between the pfSense instances quite early.
What was the reason for not simply keeping the existing LAN side subnet and using it directly behind the new pfSense instance. One large layer 2 still?
Obviously the IP addresses would need to remain and not be duplicated. And you have multiple routers on the same subnet so asymmetry is highly likely. Each server VM would need to keep it's existing gateway and could only accept connections from that gateway.
I suspect that was the show stopper here?Steve
-
******** LETS START FROM HERE ********
While this might be too much of a post for a forum, it's also an interesting challenge. it seems. Anyone could be handed this mess and told what they can and cannot change. You'd have to work withing those constraints. It sucks, a lot. I've been at this for countless hours now.
I'm not sure how to answer your question since the thread has kinda gone all over the place. Mainly it went to using GRE which got me really sideways. The gre tunnel seems to be completed and ready to use though.
And while the stuff is in a DC, it's not really their fault. They cannot change their methods/policies for just one customer and I know them well enough to know they care about their customers and would help if they could.
The problem is the new hardware is in a different part of the DC. The only way to communicate between them is either public IPs or the VLAN that I've got across both networks.
When installing the new hardware, I had to use the 10.0.0.1/24 LAN side. This simply cannot change as it would mess up a ton of custom programming and processes talking between servers. It would have been super easy to just go with a 192.168.1.1/24 for example but it cannot be done.At this point in this question, we can forget about my needing access from pf01 LAN to pf02 LAN in order to move all of the vms from one network to the other over the VLAN. I've solved that by simply using a public IP on each host I'll need access to.
Most of the vms are redundant web servers so I can easily move them without disrupting services. The ones that aren't will suffer a little down time, no issues.
I'm left with two main things to solve
1: Everything on pf02 LAN uses this one main DB server at 10.0.0.10. It has to be up always. There won't be any way to split that traffic meaning I cannot take it down but I've prepared a new server on the pf01 LAN. No public access is required for this DB but all internal servers need access to it.
The plan is to export everything from the current DB server to the new server. During that time, we'll have to shut down the live DB server to prevent anything from changing between the two databases.
But that's a problem. The servers on the pf02 LAN need access to the DB and so will the ones I've moved to pf01. I cannot have data being read/written between two servers as that will split the data.
What ever way I use, it has to keep the servers on pf02 seeing the DB at 10.0.0.10 and as they migrate over to pf01, they will see it as 10.0.0.10 on that LAN. Eventually, all servers get migrated and that's not a problem.
Since I cannot route LAN to LAN, it seems my only option will be to set up an outgoing rule that will take 10.0.010 DB traffic over the WAN to the new network and onto the new 10.0.0.10 DB Server. Ugly. Can this be done using the gre or VIP or something along those lines?
2: Final part is to redirect all traffic from public IPs on pf02 on to pf01. Later, do the same on another server I've tried not to talk about to prevent confusion, pf03. The difference with pf03 is that it doesn't have any LAN, it's just a firewall with public IPs to one service behind it which just needs to be re-directed to pf01. In this case, because the public IPs cannot be moved from that location to the location where the new servers are.
I hope this helps to bring this back to something more reasonable :).
-
Here is an updated version of the image without gre.
NOTE that pf02 traffic goes to pf01 DCLAN, not LAN. Therefore, there should be a way to do this since it's not 10.0.0.x/24 going to 10.0.0.x/24 directly.
-
Dude you still have the same network on these servers.
So 10.0.0.10 wants to talk to 10.0.0.11 - that is on the other pfsense.. Why would it send traffic anywhere? .11 is on its local network.
For you to do what you want is an extended vlan, this is normally done over dark fiber between DC or Building. Where its actually the same L2.
Extending the L2 is normally how you would do what your wanting. So the L2 is actually the same L2.
If they are going to be different L2s with the same L3 on them - then you have to do some natting so say when .10 wants to talk to .11 (this is really a vip) on pfX, and sends it to what it thinks is say 10.0.1.11 on pfY.. When then forwards it to its lan 10.0.0.11
It would be a real freaking mess doing that.. You want to extend the L2..
You clearly have some connection between pf1 and pf2 - why can this not be a bridge that bridges this L2 into just 1 extended L2 network? Your 10.0.0/24 network..
