Using pfsense with multiple WANs
-
@johnpoz said in Multi LAN networks to one pfsense:
Dude you still have the same network on these servers.
So 10.0.0.10 wants to talk to 10.0.0.11 - that is on the other >pfsense.. Why would it send traffic anywhere? .11 is on its
local network.Not sure what the 'dude' comment is about. I already know I have a problem. It's why I posted the question, because it's beyond my knowledge of how this could be accomplished and thought some higher level network folks might have some suggestions.
For you to do what you want is an extended vlan, this is normally >done over dark fiber between DC or Building. Where its actually >the same L2.
Again, this is in a data center. I don't know own it, I cannot change how they do things. All I can do is try to find a way to solve the problem. What I do have is a VLAN between the networks. I cannot get anything else, that's what I have.
You clearly have some connection between pf1 and pf2 - why can >this not be a bridge that bridges this L2 into just 1 extended L2 >network? Your 10.0.0/24 network..
It can be anything that will work. I'm asking for ideas, thoughts on how I can solve this and I have limited knowledge since I've never had to get into this kind of situation before.
Again... with LIMITED knowledge...
I have a VLAN so there is a way to communicate between the two networks. What ever can be done with a VLAN is all I have.I have two interfaces on pf02, WAN and LAN, nothing else.
I have three interfaces on pf02, WAN, DCLAN (connected to the DC VLAN) and LAN, connected to a switch and all servers behind that are 10.0.0.1/24.I cannot change the IPs from 10.0.0.x/24 otherwise, this would be a snap. I have to migrate servers over to this new network but I also have to keep routing the public traffic from pf02 to pf01 now.
Asking me why I'm doing anything doesn't make sense. I'm not doing anything, I'm here asking you guys, the pros, for ideas, suggestions and even some help on how this could be accomplished.
Happy to and have tried what is being suggested but I'm out of my league on this so please be patient with me :).
-
@lewis said in Multi LAN networks to one pfsense:
I have a VLAN so there is a way to communicate between the two networks
So you have a L2 connection between them - then just join them together so you have the same L2.. Problem solved..
https://docs.netgate.com/pfsense/en/latest/interfaces/qinq.html
Why can it not just be the same L2? This is the same DC and your moving to a different rack.. They can connect whatever L2 you want to any Rack..
The plan is to install hardware in one single cabinet to consolidate the whole thing
Your in a DC... They will connect vlan X, to whatever cabinet you want... So your L2 network you have in isle 3, rack 2 can also exist in isle 6, rack 7 building 2, etc.. Just ask them!
This something that happens every single day around the globe in every DC...
Seems more like you have a simple layer 1 problem - where the wires go, then anything else. Be it vlans you use behind pfsense, or your wan connections. Ask the DC to move your connections around so you can put them all in 1 cabinet, etc. Or same cage or same room, etc..
-
I'm not sure how to answer because some of this is simply beyond my knowledge, isn't something I've had to deal with before and is why I posted here looking for input.
The main problem is that I have two firewalls, one on each network and both handle the 10.0.0.1/24 LAN.
When all my gear was in one place, all of it communicated between each other via the VLAN I was given and on 10.0.0.1/24.
I was given rack space in another section of the DC which is not normally connected to the first area. I got them to give me an Ethernet drop between the new section and the old section so I could have the VLAN between both. It seemed to take a lot of talking just to get that and I know they will not change their normal methods just for me. Not sure what else I could ask them for.
However, since both firewalls handle a 10.0.0.1/24, that's where the problem comes in.
It seemed like on the VLAN side, I could simply move machines from one network to another since the VLAN acts kind of like a LAN. However, that's not the case. Servers aren't seeing each other because they each have their own gateway on their own network.
-
Anyhow, the point is yes, there is a VLAN so yes, there should be a way to do this but I've not found a way and so far, this thread has not either as I'm happy to try anything at this point.
I still have the gre setup configured, should I get rid of all that?
Also, from the start, even with the VLAN, I'm not able to communicate between the two networks. I thought it might be as simple as needing some rules to allow pf01 LAN devices to communicate through the DCLAN (VLAN) interface but I've yet to find documentation to explains how I can create rules for that and all the ones I've tried never worked.
-
@lewis said in Multi LAN networks to one pfsense:
The main problem is that I have two firewalls, one on each network and both handle the 10.0.0.1/24 LAN.
Yeah so you make 10.0.0/24 a common L2 network.. And use .1 on 1 and .2 on the other and on your hosts set the gateway to which gateway you want them to use.
But to be honest, why do you have multiple firewalls.. You only need 1..
I would suggest you call the DC, tell them you want to consolidate everything and you want your connections moved to where ever your moving everything too.
Your trying to solve a problem that your creating because you just haven't asked them to move connection X over to different port on the patch panel where your moving everything too..
-
Your trying to solve a problem that your creating because you just >haven't asked them to move connection X over to different port on >the patch panel where your moving everything too
John I assume... this has been explained, repeatedly. There is no point in focusing on the DC, they gave me what they can. I have a VLAN connection between the two networks. There is nothing else to give me.
I'm not creating this problem, I'm working withing the constraints I've been given as explained throughout this thread.
For me, this is also an interesting challenge and by solving it, I'll learn something new about using pfsense in yet another way.
The thread itself explains how I've ended up with multiple firewalls.
It is not as simple as moving something from one place to another. It's a live production environment. All of this is explained above.
It is also explained that the public IPs cannot be moved from one network to another and changing those is also not an option. However, one reason is because the devices in the field are hard coded to specific IPs and impossible to get at.
Add to that, there are more than one pfsense firewall, each used to be its own network, all have to now forward their traffic to the new pf01 network.
In the end, pf02 will not have any servers/services on it's LAN but it needs to forward public connections to the LAN behind pf01. And, there will be at least one other pf03 that also doesn't have any LAN but its public facing connections have to be forwarded to the new network.
BTW, should I remove all of the gre configuration at this point?
With my limited knowledge, it does seem that since I have this VLAN connecting everything, all I would need to do is to move a server from one network to the other and give it the new gateway.
If that's the case, then what I need to understand is how I can route that traffic from pf02 public to pf01 after making this change.
pf01 has a decicated port, DCLAN connected to CD VLAN.
-
This is what I need to end up with.
All servers/services will slowly be moved to the LAN on pf01.
The biggest challenge is I have a DB server on LAN pf02 that everything needs to talk with.That's about the only machine that would need some sort of direct connection from both networks.
Pf02 will end up acting as a router/switch. This is why I asked early on if maybe I was better off looking at a software router to replace pf02 etc with. However, pfsense is so powerful that I had to believe there is a nice solution for doing this.
-
@lewis could you not just replace one of the pfsense units with a level 2 switch and use that to route the second WAN and remote VLANs all to a single pfsense router?
-
To try and get back to a nice starting point, I got rid of all the gre stuff again.
I cannot get into more VLAN's because I don't control any switches in between unless there is a way to do it pf to pf directly.
I have a VIP of 10.100.0.2 on LAN of pf02 and I have 10.100.0.1 on pf01 DCLAN.
Of course, I can ping 10.100.0.2 from anything on the pf02 LAN and tried to set up rules to allow vms on that LAN to ping 10.100.0.1. No go.
How can I accomplish that as a first step?
-
@patch said in Multi LAN networks to one pfsense:
@lewis could you not just replace one of the pfsense units with a level 2 switch and use that to route the second WAN and remote VLANs all to a single pfsense router?
I wish it was that easy but it was a server turned into a firewall using pfsense. Nothing can be changed in that network, meaning, one cannot change hardware to something else, it's all servers in there.
I had mentioned that it could be possible to convert pfsense to a software router but that will only solve a problem after everything is moved over.
I've been thinking about this a lot and the main server/service that is a problem seems to be only the database server. It seems that if I move the DB server to the new network then find a way to re-route all connections to that to the new network, everything else could fall into place.
Then later, I could solve the issue of either using pf or a sw router for the rest.
-
@lewis said in Multi LAN networks to one pfsense:
they gave me what they can
No they didn't - you just don't know what to ask them for or what to do.
If you have L2 that is connected between location A and location B... Then use that to connect to switch at location A and then Location B and you have 1 extended vlan..
Do this sort of thing every single day in multiple DC across the freaking globe.. Your thinking its some quantum physics problem when its 2+2..
If they have some "dclan" vlan that connects your locations.. There you go you have an L2 between your locations - connect this to your current "lan" at each location and you have 1 LAN...
-
@johnpoz said in Multi LAN networks to one pfsense:
@lewis said in Multi LAN networks to one pfsense:
they gave me what they can
No they didn't - you just don't know what to ask them for or what to do.
If you have L2 that is connected between location A and location B... Then use that to connect to switch at location A and then Location B and you have 1 extended vlan..
Do this sort of thing every single day in multiple DC across the freaking globe.. Your thinking its some quantum physics problem when its 2+2..
It's odd that you keep blaming me like I planned things this way. There is no reason for you to act like you're frustrated with me and it's certainly not helping the tone of was a very friendly thread so far.
No, I don't think it's that complicated, I already told you I simply have never had to deal with something like this so asked for help in these forums.
It isn't that simple. The servers are in different areas of the DC and the new hardware is partly to consolidate everything into one place.
They didn't even seem to like the idea of giving me a VLAN connection between the two locations. There is no direct connection between one area to the other, they simply gave me a separate Ethernet connection to their 'LAN' network where my VLAN exists within. I have no control over the connection or switches that traffic goes through.
If you have L2 that is connected between location A and location >B... Then use that to connect to switch at location A and then >Location B and you have 1 extended vlan..
If you have some ideas, feel free to the steps I should take since as I've told you several times, this is not something I've done before.
-
@lewis said in Multi LAN networks to one pfsense:
It isn't that simple. The servers are in different areas of the DC
Yes it is.. Dude I have worked with DCs for like 20 years.. Maybe just don't understand how they work - its not magic, its a bunch of switches and wires that connect different racks together. Any rack can talk to any other rack - since its just a vlan on the switches that have to be configure, or some wires that have to be moved from this patch panel to that patch panel.
I know its not "your" fault - but its like watching a monkey figure out how to start a fire with 2 sticks.. When there is a box of matches right there - he just doesn't know what "matches" are..
Connect your "lans" together.. Ask the DC to do that for you! They might charge you a cross connect fee at the worse case.. Depending where your location A is and where B is in their DC..
"internet" connection can be brought into any location in the DC.. So why can you not just move the public IPs that your currently using in location A to B? Who exactly are you talking at this so called DC? What DC is it exactly - PM me the name if you don't want to post it.
-
Monkey? Wow, thanks for that.
I did get a cross connect installed. The 'lans' are 'connected' for lack of better words.
There is just a VLAN across their own LAN which apparently doesn't route across all sections of the DC.
It's not just one LAN connecting to another LAN otherwise, this would be simple. The servers off of pf02 aren't just in one place, they are in different areas of the DC.
I already explained what the public IPs have to be routed this way.
I posted here because I don't have that level of knowledge but you're more interested in telling me how much of a monkey I am than actually helping.
It's fine, this thread is obviously going to devolve into drama so there's no point in continuing. Thanks to everyone who tried helping. I'm sure I'll find a solution and will even be sure to come back and share it so this whole thing doesn't go to waste.
-
@lewis said in Multi LAN networks to one pfsense:
it was a server turned into a firewall using pfsense. Nothing can be changed in that network, meaning, one cannot change hardware to something else, it's all servers in there
@lewis said in Multi LAN networks to one pfsense:
They didn't even seem to like the idea of giving me a VLAN connection between the two locations. There is no direct connection between one area to the other, they simply gave me a separate Ethernet connection to their 'LAN' network where my VLAN exists within
@lewis said in Multi LAN networks to one pfsense:
I did get a cross connect installed. The 'lans' are 'connected' for lack of better words.
There is just a VLAN across their own LAN which apparently doesn't route across all sections of the DC.
It's not just one LAN connecting to another LAN otherwise, this would be simple.I still find what resources are actually available at each location a little confusing.
-
Each has a server. Is this your hardware with a set number of physical NIC or do you have a virtual server with virtual NIC
-
WAN is available at each rack location. Is that a physical cable or logical connection
-
Cross connect is that a physical connection. I appreciate your data centre is connecting to your other rack using their VLAN but on your cable is do you see their VLAN setting or is your connection isolated so you only see a LAN. If so it is possible their VLAN supports a VLAN over a VLAN which would mean you could use your own VLAN configuration to acheive the required network configuration.
-
-
Ok, there's confusion all round here!
But to clarify:
You do already have a L2 (VLAN) that is common to all the physical locations.
Given the constraint that the servers have hard coded IPs and need to talk to one another using those none of the NAT'd/routed solutions we previously discussed will work.
The only thing you can do is to use 10.0.0.0/24 as the common L2 subnet.So remove the GRE tunnels and the VIPs and the DCLAN interface from pf01.
The servers 'behind' will need to be on the same LAN L2.Now you can move the servers to the new location and nothing will have changed.
However as I mentioned before the problem you may see with that solution is that each server will need to be configured to use a particular gateway (pf01, pf02 etc) and will not be able to receive external traffic from another gateway. That would create an asymmetric route. Thus if you move an existing server to the new host and it is receiving connections from the public IPs on pf02 it will not be able to accept connections from the new public IPs on pf01.
The only way around that would be to outbound NAT the connections on one or both firewalls. But doing that would hide the source IP from the server and the applications running on it which may be a show-stopper?
Steve
-
I thought I should come back and update this now that I have part of my solution. Doubt anyone will ever read to the end but if they do, they will find this.
It's not completed but it's allowed me to migrate everything as I needed and the last step will be a final configuration.
First, the DCLAN was disconnected and not used after all.
Both pfsense LAN interface were connected to the same DC LAN / VLAN.Before doing this, I made sure all devices had unique IPs otherwise this would break things.
I then migrated all of the servers/services I needed to for now over to the new network (where you see the servers).Almost nothing had to change other than putting the gateway into the DHCP config for servers that get DHCP IPs and hard coding the gw on those that don't. Almost all machines start off by getting a DHCP IP so I can get their MAC then usually just assign a permanent DHCP so I can keep track of assets.
What's nice about this is that I can just change the gw on a device and it then uses the pfx that I need it to.
Still have some problems but I think most of it is ARP caching that will eventually clear up.Right now, this works for what I need. The final config will be along the same lines but working out any problems that would be found during this migration period.
I do not need all of the servers to be in the same network so even if some cannot reach others on the LAN side, it's fine.
-
Nice.
Did you see any issues with servers needing to accept connections forwarded from both pfSense installs?
-
Hi, no issues what so ever so far. It just ends up being maybe a bit non standard but it works perfectly well and is quite flexible in that I can add more networks in the future as well.
-
Ah it dawns on me there is one thing I should mention so someone else doesn't get caught.
ARP cache played a huge part in this and an accidentally left over rule in one of the firewalls as well.
The firewalls were basically competing to be the gateway so things would get weird like a vm would boot up with a gw then a while later change to another.
Once ARP cleared up, everything was fine.
