Using pfsense with multiple WANs
-
This is what I need to end up with.
All servers/services will slowly be moved to the LAN on pf01.
The biggest challenge is I have a DB server on LAN pf02 that everything needs to talk with.That's about the only machine that would need some sort of direct connection from both networks.
Pf02 will end up acting as a router/switch. This is why I asked early on if maybe I was better off looking at a software router to replace pf02 etc with. However, pfsense is so powerful that I had to believe there is a nice solution for doing this.
-
@lewis could you not just replace one of the pfsense units with a level 2 switch and use that to route the second WAN and remote VLANs all to a single pfsense router?
-
To try and get back to a nice starting point, I got rid of all the gre stuff again.
I cannot get into more VLAN's because I don't control any switches in between unless there is a way to do it pf to pf directly.
I have a VIP of 10.100.0.2 on LAN of pf02 and I have 10.100.0.1 on pf01 DCLAN.
Of course, I can ping 10.100.0.2 from anything on the pf02 LAN and tried to set up rules to allow vms on that LAN to ping 10.100.0.1. No go.
How can I accomplish that as a first step?
-
@patch said in Multi LAN networks to one pfsense:
@lewis could you not just replace one of the pfsense units with a level 2 switch and use that to route the second WAN and remote VLANs all to a single pfsense router?
I wish it was that easy but it was a server turned into a firewall using pfsense. Nothing can be changed in that network, meaning, one cannot change hardware to something else, it's all servers in there.
I had mentioned that it could be possible to convert pfsense to a software router but that will only solve a problem after everything is moved over.
I've been thinking about this a lot and the main server/service that is a problem seems to be only the database server. It seems that if I move the DB server to the new network then find a way to re-route all connections to that to the new network, everything else could fall into place.
Then later, I could solve the issue of either using pf or a sw router for the rest.
-
@lewis said in Multi LAN networks to one pfsense:
they gave me what they can
No they didn't - you just don't know what to ask them for or what to do.
If you have L2 that is connected between location A and location B... Then use that to connect to switch at location A and then Location B and you have 1 extended vlan..
Do this sort of thing every single day in multiple DC across the freaking globe.. Your thinking its some quantum physics problem when its 2+2..
If they have some "dclan" vlan that connects your locations.. There you go you have an L2 between your locations - connect this to your current "lan" at each location and you have 1 LAN...
-
@johnpoz said in Multi LAN networks to one pfsense:
@lewis said in Multi LAN networks to one pfsense:
they gave me what they can
No they didn't - you just don't know what to ask them for or what to do.
If you have L2 that is connected between location A and location B... Then use that to connect to switch at location A and then Location B and you have 1 extended vlan..
Do this sort of thing every single day in multiple DC across the freaking globe.. Your thinking its some quantum physics problem when its 2+2..
It's odd that you keep blaming me like I planned things this way. There is no reason for you to act like you're frustrated with me and it's certainly not helping the tone of was a very friendly thread so far.
No, I don't think it's that complicated, I already told you I simply have never had to deal with something like this so asked for help in these forums.
It isn't that simple. The servers are in different areas of the DC and the new hardware is partly to consolidate everything into one place.
They didn't even seem to like the idea of giving me a VLAN connection between the two locations. There is no direct connection between one area to the other, they simply gave me a separate Ethernet connection to their 'LAN' network where my VLAN exists within. I have no control over the connection or switches that traffic goes through.
If you have L2 that is connected between location A and location >B... Then use that to connect to switch at location A and then >Location B and you have 1 extended vlan..
If you have some ideas, feel free to the steps I should take since as I've told you several times, this is not something I've done before.
-
@lewis said in Multi LAN networks to one pfsense:
It isn't that simple. The servers are in different areas of the DC
Yes it is.. Dude I have worked with DCs for like 20 years.. Maybe just don't understand how they work - its not magic, its a bunch of switches and wires that connect different racks together. Any rack can talk to any other rack - since its just a vlan on the switches that have to be configure, or some wires that have to be moved from this patch panel to that patch panel.
I know its not "your" fault - but its like watching a monkey figure out how to start a fire with 2 sticks.. When there is a box of matches right there - he just doesn't know what "matches" are..
Connect your "lans" together.. Ask the DC to do that for you! They might charge you a cross connect fee at the worse case.. Depending where your location A is and where B is in their DC..
"internet" connection can be brought into any location in the DC.. So why can you not just move the public IPs that your currently using in location A to B? Who exactly are you talking at this so called DC? What DC is it exactly - PM me the name if you don't want to post it.
-
Monkey? Wow, thanks for that.
I did get a cross connect installed. The 'lans' are 'connected' for lack of better words.
There is just a VLAN across their own LAN which apparently doesn't route across all sections of the DC.
It's not just one LAN connecting to another LAN otherwise, this would be simple. The servers off of pf02 aren't just in one place, they are in different areas of the DC.
I already explained what the public IPs have to be routed this way.
I posted here because I don't have that level of knowledge but you're more interested in telling me how much of a monkey I am than actually helping.
It's fine, this thread is obviously going to devolve into drama so there's no point in continuing. Thanks to everyone who tried helping. I'm sure I'll find a solution and will even be sure to come back and share it so this whole thing doesn't go to waste.
-
@lewis said in Multi LAN networks to one pfsense:
it was a server turned into a firewall using pfsense. Nothing can be changed in that network, meaning, one cannot change hardware to something else, it's all servers in there
@lewis said in Multi LAN networks to one pfsense:
They didn't even seem to like the idea of giving me a VLAN connection between the two locations. There is no direct connection between one area to the other, they simply gave me a separate Ethernet connection to their 'LAN' network where my VLAN exists within
@lewis said in Multi LAN networks to one pfsense:
I did get a cross connect installed. The 'lans' are 'connected' for lack of better words.
There is just a VLAN across their own LAN which apparently doesn't route across all sections of the DC.
It's not just one LAN connecting to another LAN otherwise, this would be simple.I still find what resources are actually available at each location a little confusing.
-
Each has a server. Is this your hardware with a set number of physical NIC or do you have a virtual server with virtual NIC
-
WAN is available at each rack location. Is that a physical cable or logical connection
-
Cross connect is that a physical connection. I appreciate your data centre is connecting to your other rack using their VLAN but on your cable is do you see their VLAN setting or is your connection isolated so you only see a LAN. If so it is possible their VLAN supports a VLAN over a VLAN which would mean you could use your own VLAN configuration to acheive the required network configuration.
-
-
Ok, there's confusion all round here!
But to clarify:
You do already have a L2 (VLAN) that is common to all the physical locations.
Given the constraint that the servers have hard coded IPs and need to talk to one another using those none of the NAT'd/routed solutions we previously discussed will work.
The only thing you can do is to use 10.0.0.0/24 as the common L2 subnet.So remove the GRE tunnels and the VIPs and the DCLAN interface from pf01.
The servers 'behind' will need to be on the same LAN L2.Now you can move the servers to the new location and nothing will have changed.
However as I mentioned before the problem you may see with that solution is that each server will need to be configured to use a particular gateway (pf01, pf02 etc) and will not be able to receive external traffic from another gateway. That would create an asymmetric route. Thus if you move an existing server to the new host and it is receiving connections from the public IPs on pf02 it will not be able to accept connections from the new public IPs on pf01.
The only way around that would be to outbound NAT the connections on one or both firewalls. But doing that would hide the source IP from the server and the applications running on it which may be a show-stopper?
Steve
-
I thought I should come back and update this now that I have part of my solution. Doubt anyone will ever read to the end but if they do, they will find this.
It's not completed but it's allowed me to migrate everything as I needed and the last step will be a final configuration.
First, the DCLAN was disconnected and not used after all.
Both pfsense LAN interface were connected to the same DC LAN / VLAN.Before doing this, I made sure all devices had unique IPs otherwise this would break things.
I then migrated all of the servers/services I needed to for now over to the new network (where you see the servers).Almost nothing had to change other than putting the gateway into the DHCP config for servers that get DHCP IPs and hard coding the gw on those that don't. Almost all machines start off by getting a DHCP IP so I can get their MAC then usually just assign a permanent DHCP so I can keep track of assets.
What's nice about this is that I can just change the gw on a device and it then uses the pfx that I need it to.
Still have some problems but I think most of it is ARP caching that will eventually clear up.Right now, this works for what I need. The final config will be along the same lines but working out any problems that would be found during this migration period.
I do not need all of the servers to be in the same network so even if some cannot reach others on the LAN side, it's fine.
-
Nice.
Did you see any issues with servers needing to accept connections forwarded from both pfSense installs?
-
Hi, no issues what so ever so far. It just ends up being maybe a bit non standard but it works perfectly well and is quite flexible in that I can add more networks in the future as well.
-
Ah it dawns on me there is one thing I should mention so someone else doesn't get caught.
ARP cache played a huge part in this and an accidentally left over rule in one of the firewalls as well.
The firewalls were basically competing to be the gateway so things would get weird like a vm would boot up with a gw then a while later change to another.
Once ARP cleared up, everything was fine.
