Using pfsense with multiple WANs
-
If you read the entire thread where I mentioned all of the reasons why I have to do this crazy stuff, then you would stop saying 'why do it this way'. It certainly isn't something I wanted to have to do but am stuck having to.
It is a mess, a huge one and only one person to get it all done.
Just got to my desk to try to work on this again.
I had to add a public IP to the esx host then allow the pf01 IP access to the ports I need to import vms to the new network.Then I can finally have all the traffic that's supposed to go to LAN on pf02 going to DCLAN>LAN on pf01.
When I get to having moved all of those vms to their new servers, the last thing is figuring out how pf03 but really not worried about that just yet. pf02 is the most important.
I have a severe deadline to move everything off of a bunch of hosts on pf02 so that is my priority.
Anyhow, now that I have a copy of 10.0.0.120 onto the pf01 network, I have to re-route that from pf02.
-
So you need the VMs behind each to be able to connect directly? During production, not just in order to move them?
I think I've said several times that's not possible with this scheme. They are using the same subnet so require NATing between them. The only way to do it would be to outbound NAT that traffic from pf02 so the servers in pf01 see a different source IP. The servers in the pf02 LAN would need to use a different target IP though.
Reviewing this whole ticket though it looks like we went off down the path of routing the traffic between the pfSense instances quite early.
What was the reason for not simply keeping the existing LAN side subnet and using it directly behind the new pfSense instance. One large layer 2 still?
Obviously the IP addresses would need to remain and not be duplicated. And you have multiple routers on the same subnet so asymmetry is highly likely. Each server VM would need to keep it's existing gateway and could only accept connections from that gateway.
I suspect that was the show stopper here?Steve
-
******** LETS START FROM HERE ********
While this might be too much of a post for a forum, it's also an interesting challenge. it seems. Anyone could be handed this mess and told what they can and cannot change. You'd have to work withing those constraints. It sucks, a lot. I've been at this for countless hours now.
I'm not sure how to answer your question since the thread has kinda gone all over the place. Mainly it went to using GRE which got me really sideways. The gre tunnel seems to be completed and ready to use though.
And while the stuff is in a DC, it's not really their fault. They cannot change their methods/policies for just one customer and I know them well enough to know they care about their customers and would help if they could.
The problem is the new hardware is in a different part of the DC. The only way to communicate between them is either public IPs or the VLAN that I've got across both networks.
When installing the new hardware, I had to use the 10.0.0.1/24 LAN side. This simply cannot change as it would mess up a ton of custom programming and processes talking between servers. It would have been super easy to just go with a 192.168.1.1/24 for example but it cannot be done.At this point in this question, we can forget about my needing access from pf01 LAN to pf02 LAN in order to move all of the vms from one network to the other over the VLAN. I've solved that by simply using a public IP on each host I'll need access to.
Most of the vms are redundant web servers so I can easily move them without disrupting services. The ones that aren't will suffer a little down time, no issues.
I'm left with two main things to solve
1: Everything on pf02 LAN uses this one main DB server at 10.0.0.10. It has to be up always. There won't be any way to split that traffic meaning I cannot take it down but I've prepared a new server on the pf01 LAN. No public access is required for this DB but all internal servers need access to it.
The plan is to export everything from the current DB server to the new server. During that time, we'll have to shut down the live DB server to prevent anything from changing between the two databases.
But that's a problem. The servers on the pf02 LAN need access to the DB and so will the ones I've moved to pf01. I cannot have data being read/written between two servers as that will split the data.
What ever way I use, it has to keep the servers on pf02 seeing the DB at 10.0.0.10 and as they migrate over to pf01, they will see it as 10.0.0.10 on that LAN. Eventually, all servers get migrated and that's not a problem.
Since I cannot route LAN to LAN, it seems my only option will be to set up an outgoing rule that will take 10.0.010 DB traffic over the WAN to the new network and onto the new 10.0.0.10 DB Server. Ugly. Can this be done using the gre or VIP or something along those lines?
2: Final part is to redirect all traffic from public IPs on pf02 on to pf01. Later, do the same on another server I've tried not to talk about to prevent confusion, pf03. The difference with pf03 is that it doesn't have any LAN, it's just a firewall with public IPs to one service behind it which just needs to be re-directed to pf01. In this case, because the public IPs cannot be moved from that location to the location where the new servers are.
I hope this helps to bring this back to something more reasonable :).
-
Here is an updated version of the image without gre.
NOTE that pf02 traffic goes to pf01 DCLAN, not LAN. Therefore, there should be a way to do this since it's not 10.0.0.x/24 going to 10.0.0.x/24 directly.
-
Dude you still have the same network on these servers.
So 10.0.0.10 wants to talk to 10.0.0.11 - that is on the other pfsense.. Why would it send traffic anywhere? .11 is on its local network.
For you to do what you want is an extended vlan, this is normally done over dark fiber between DC or Building. Where its actually the same L2.
Extending the L2 is normally how you would do what your wanting. So the L2 is actually the same L2.
If they are going to be different L2s with the same L3 on them - then you have to do some natting so say when .10 wants to talk to .11 (this is really a vip) on pfX, and sends it to what it thinks is say 10.0.1.11 on pfY.. When then forwards it to its lan 10.0.0.11
It would be a real freaking mess doing that.. You want to extend the L2..
You clearly have some connection between pf1 and pf2 - why can this not be a bridge that bridges this L2 into just 1 extended L2 network? Your 10.0.0/24 network..
-
@johnpoz said in Multi LAN networks to one pfsense:
Dude you still have the same network on these servers.
So 10.0.0.10 wants to talk to 10.0.0.11 - that is on the other >pfsense.. Why would it send traffic anywhere? .11 is on its
local network.Not sure what the 'dude' comment is about. I already know I have a problem. It's why I posted the question, because it's beyond my knowledge of how this could be accomplished and thought some higher level network folks might have some suggestions.
For you to do what you want is an extended vlan, this is normally >done over dark fiber between DC or Building. Where its actually >the same L2.
Again, this is in a data center. I don't know own it, I cannot change how they do things. All I can do is try to find a way to solve the problem. What I do have is a VLAN between the networks. I cannot get anything else, that's what I have.
You clearly have some connection between pf1 and pf2 - why can >this not be a bridge that bridges this L2 into just 1 extended L2 >network? Your 10.0.0/24 network..
It can be anything that will work. I'm asking for ideas, thoughts on how I can solve this and I have limited knowledge since I've never had to get into this kind of situation before.
Again... with LIMITED knowledge...
I have a VLAN so there is a way to communicate between the two networks. What ever can be done with a VLAN is all I have.I have two interfaces on pf02, WAN and LAN, nothing else.
I have three interfaces on pf02, WAN, DCLAN (connected to the DC VLAN) and LAN, connected to a switch and all servers behind that are 10.0.0.1/24.I cannot change the IPs from 10.0.0.x/24 otherwise, this would be a snap. I have to migrate servers over to this new network but I also have to keep routing the public traffic from pf02 to pf01 now.
Asking me why I'm doing anything doesn't make sense. I'm not doing anything, I'm here asking you guys, the pros, for ideas, suggestions and even some help on how this could be accomplished.
Happy to and have tried what is being suggested but I'm out of my league on this so please be patient with me :).
-
@lewis said in Multi LAN networks to one pfsense:
I have a VLAN so there is a way to communicate between the two networks
So you have a L2 connection between them - then just join them together so you have the same L2.. Problem solved..
https://docs.netgate.com/pfsense/en/latest/interfaces/qinq.html
Why can it not just be the same L2? This is the same DC and your moving to a different rack.. They can connect whatever L2 you want to any Rack..
The plan is to install hardware in one single cabinet to consolidate the whole thing
Your in a DC... They will connect vlan X, to whatever cabinet you want... So your L2 network you have in isle 3, rack 2 can also exist in isle 6, rack 7 building 2, etc.. Just ask them!
This something that happens every single day around the globe in every DC...
Seems more like you have a simple layer 1 problem - where the wires go, then anything else. Be it vlans you use behind pfsense, or your wan connections. Ask the DC to move your connections around so you can put them all in 1 cabinet, etc. Or same cage or same room, etc..
-
I'm not sure how to answer because some of this is simply beyond my knowledge, isn't something I've had to deal with before and is why I posted here looking for input.
The main problem is that I have two firewalls, one on each network and both handle the 10.0.0.1/24 LAN.
When all my gear was in one place, all of it communicated between each other via the VLAN I was given and on 10.0.0.1/24.
I was given rack space in another section of the DC which is not normally connected to the first area. I got them to give me an Ethernet drop between the new section and the old section so I could have the VLAN between both. It seemed to take a lot of talking just to get that and I know they will not change their normal methods just for me. Not sure what else I could ask them for.
However, since both firewalls handle a 10.0.0.1/24, that's where the problem comes in.
It seemed like on the VLAN side, I could simply move machines from one network to another since the VLAN acts kind of like a LAN. However, that's not the case. Servers aren't seeing each other because they each have their own gateway on their own network.
-
Anyhow, the point is yes, there is a VLAN so yes, there should be a way to do this but I've not found a way and so far, this thread has not either as I'm happy to try anything at this point.
I still have the gre setup configured, should I get rid of all that?
Also, from the start, even with the VLAN, I'm not able to communicate between the two networks. I thought it might be as simple as needing some rules to allow pf01 LAN devices to communicate through the DCLAN (VLAN) interface but I've yet to find documentation to explains how I can create rules for that and all the ones I've tried never worked.
-
@lewis said in Multi LAN networks to one pfsense:
The main problem is that I have two firewalls, one on each network and both handle the 10.0.0.1/24 LAN.
Yeah so you make 10.0.0/24 a common L2 network.. And use .1 on 1 and .2 on the other and on your hosts set the gateway to which gateway you want them to use.
But to be honest, why do you have multiple firewalls.. You only need 1..
I would suggest you call the DC, tell them you want to consolidate everything and you want your connections moved to where ever your moving everything too.
Your trying to solve a problem that your creating because you just haven't asked them to move connection X over to different port on the patch panel where your moving everything too..
-
Your trying to solve a problem that your creating because you just >haven't asked them to move connection X over to different port on >the patch panel where your moving everything too
John I assume... this has been explained, repeatedly. There is no point in focusing on the DC, they gave me what they can. I have a VLAN connection between the two networks. There is nothing else to give me.
I'm not creating this problem, I'm working withing the constraints I've been given as explained throughout this thread.
For me, this is also an interesting challenge and by solving it, I'll learn something new about using pfsense in yet another way.
The thread itself explains how I've ended up with multiple firewalls.
It is not as simple as moving something from one place to another. It's a live production environment. All of this is explained above.
It is also explained that the public IPs cannot be moved from one network to another and changing those is also not an option. However, one reason is because the devices in the field are hard coded to specific IPs and impossible to get at.
Add to that, there are more than one pfsense firewall, each used to be its own network, all have to now forward their traffic to the new pf01 network.
In the end, pf02 will not have any servers/services on it's LAN but it needs to forward public connections to the LAN behind pf01. And, there will be at least one other pf03 that also doesn't have any LAN but its public facing connections have to be forwarded to the new network.
BTW, should I remove all of the gre configuration at this point?
With my limited knowledge, it does seem that since I have this VLAN connecting everything, all I would need to do is to move a server from one network to the other and give it the new gateway.
If that's the case, then what I need to understand is how I can route that traffic from pf02 public to pf01 after making this change.
pf01 has a decicated port, DCLAN connected to CD VLAN.
-
This is what I need to end up with.
All servers/services will slowly be moved to the LAN on pf01.
The biggest challenge is I have a DB server on LAN pf02 that everything needs to talk with.That's about the only machine that would need some sort of direct connection from both networks.
Pf02 will end up acting as a router/switch. This is why I asked early on if maybe I was better off looking at a software router to replace pf02 etc with. However, pfsense is so powerful that I had to believe there is a nice solution for doing this.
-
@lewis could you not just replace one of the pfsense units with a level 2 switch and use that to route the second WAN and remote VLANs all to a single pfsense router?
-
To try and get back to a nice starting point, I got rid of all the gre stuff again.
I cannot get into more VLAN's because I don't control any switches in between unless there is a way to do it pf to pf directly.
I have a VIP of 10.100.0.2 on LAN of pf02 and I have 10.100.0.1 on pf01 DCLAN.
Of course, I can ping 10.100.0.2 from anything on the pf02 LAN and tried to set up rules to allow vms on that LAN to ping 10.100.0.1. No go.
How can I accomplish that as a first step?
-
@patch said in Multi LAN networks to one pfsense:
@lewis could you not just replace one of the pfsense units with a level 2 switch and use that to route the second WAN and remote VLANs all to a single pfsense router?
I wish it was that easy but it was a server turned into a firewall using pfsense. Nothing can be changed in that network, meaning, one cannot change hardware to something else, it's all servers in there.
I had mentioned that it could be possible to convert pfsense to a software router but that will only solve a problem after everything is moved over.
I've been thinking about this a lot and the main server/service that is a problem seems to be only the database server. It seems that if I move the DB server to the new network then find a way to re-route all connections to that to the new network, everything else could fall into place.
Then later, I could solve the issue of either using pf or a sw router for the rest.
-
@lewis said in Multi LAN networks to one pfsense:
they gave me what they can
No they didn't - you just don't know what to ask them for or what to do.
If you have L2 that is connected between location A and location B... Then use that to connect to switch at location A and then Location B and you have 1 extended vlan..
Do this sort of thing every single day in multiple DC across the freaking globe.. Your thinking its some quantum physics problem when its 2+2..
If they have some "dclan" vlan that connects your locations.. There you go you have an L2 between your locations - connect this to your current "lan" at each location and you have 1 LAN...
-
@johnpoz said in Multi LAN networks to one pfsense:
@lewis said in Multi LAN networks to one pfsense:
they gave me what they can
No they didn't - you just don't know what to ask them for or what to do.
If you have L2 that is connected between location A and location B... Then use that to connect to switch at location A and then Location B and you have 1 extended vlan..
Do this sort of thing every single day in multiple DC across the freaking globe.. Your thinking its some quantum physics problem when its 2+2..
It's odd that you keep blaming me like I planned things this way. There is no reason for you to act like you're frustrated with me and it's certainly not helping the tone of was a very friendly thread so far.
No, I don't think it's that complicated, I already told you I simply have never had to deal with something like this so asked for help in these forums.
It isn't that simple. The servers are in different areas of the DC and the new hardware is partly to consolidate everything into one place.
They didn't even seem to like the idea of giving me a VLAN connection between the two locations. There is no direct connection between one area to the other, they simply gave me a separate Ethernet connection to their 'LAN' network where my VLAN exists within. I have no control over the connection or switches that traffic goes through.
If you have L2 that is connected between location A and location >B... Then use that to connect to switch at location A and then >Location B and you have 1 extended vlan..
If you have some ideas, feel free to the steps I should take since as I've told you several times, this is not something I've done before.
-
@lewis said in Multi LAN networks to one pfsense:
It isn't that simple. The servers are in different areas of the DC
Yes it is.. Dude I have worked with DCs for like 20 years.. Maybe just don't understand how they work - its not magic, its a bunch of switches and wires that connect different racks together. Any rack can talk to any other rack - since its just a vlan on the switches that have to be configure, or some wires that have to be moved from this patch panel to that patch panel.
I know its not "your" fault - but its like watching a monkey figure out how to start a fire with 2 sticks.. When there is a box of matches right there - he just doesn't know what "matches" are..
Connect your "lans" together.. Ask the DC to do that for you! They might charge you a cross connect fee at the worse case.. Depending where your location A is and where B is in their DC..
"internet" connection can be brought into any location in the DC.. So why can you not just move the public IPs that your currently using in location A to B? Who exactly are you talking at this so called DC? What DC is it exactly - PM me the name if you don't want to post it.
-
Monkey? Wow, thanks for that.
I did get a cross connect installed. The 'lans' are 'connected' for lack of better words.
There is just a VLAN across their own LAN which apparently doesn't route across all sections of the DC.
It's not just one LAN connecting to another LAN otherwise, this would be simple. The servers off of pf02 aren't just in one place, they are in different areas of the DC.
I already explained what the public IPs have to be routed this way.
I posted here because I don't have that level of knowledge but you're more interested in telling me how much of a monkey I am than actually helping.
It's fine, this thread is obviously going to devolve into drama so there's no point in continuing. Thanks to everyone who tried helping. I'm sure I'll find a solution and will even be sure to come back and share it so this whole thing doesn't go to waste.
-
@lewis said in Multi LAN networks to one pfsense:
it was a server turned into a firewall using pfsense. Nothing can be changed in that network, meaning, one cannot change hardware to something else, it's all servers in there
@lewis said in Multi LAN networks to one pfsense:
They didn't even seem to like the idea of giving me a VLAN connection between the two locations. There is no direct connection between one area to the other, they simply gave me a separate Ethernet connection to their 'LAN' network where my VLAN exists within
@lewis said in Multi LAN networks to one pfsense:
I did get a cross connect installed. The 'lans' are 'connected' for lack of better words.
There is just a VLAN across their own LAN which apparently doesn't route across all sections of the DC.
It's not just one LAN connecting to another LAN otherwise, this would be simple.I still find what resources are actually available at each location a little confusing.
-
Each has a server. Is this your hardware with a set number of physical NIC or do you have a virtual server with virtual NIC
-
WAN is available at each rack location. Is that a physical cable or logical connection
-
Cross connect is that a physical connection. I appreciate your data centre is connecting to your other rack using their VLAN but on your cable is do you see their VLAN setting or is your connection isolated so you only see a LAN. If so it is possible their VLAN supports a VLAN over a VLAN which would mean you could use your own VLAN configuration to acheive the required network configuration.
-
