Using pfsense with multiple WANs
-
Ok, so pfSense01 cannot have an IP address on it's interface in the DCLAN in the 10.0.0.0/24 subnet because it's already using that as it's LAN.
So you need to create a new subnet there that it can use and assign the pfSense01 DCLAN interface an IP from it. Then you need to add a VIP to pfSense02 in that subnet on it's DCLAN interface so it can route traffic to pf01.
Then you can simply change the port forwards of pf02 to point to pf01 in the new subnet as you move them across.What won't work is servers in the existing DCLAN will be unable to connect to servers behind pf01. The source IP will conflict with the pf01 LAN.
The reply-to tags on traffic coming into the DCLAN interface on pf01 will allow replies to go back out of the gateway defined on that interface. However that means if you have multiple routers forwarding traffic to pf01 on the interface only the one defined as the gateway will get replies as expected.
If you have multiple pfSense boxes doing that you might need to use tunneling of some sort to avoid asymmetric routing.Steve
-
This post is deleted! -
This post is deleted! -
I had a couple of replies but I deleted them to prevent confusion so I can continue from your last comment.
What won't work is servers in the existing DCLAN will be unable to >connect to servers behind pf01. The source IP will conflict with the >pf01 LAN.
I think the answer to importing vms will be to do it over the WAN. This way, I can simply open firewall ports on pfsense 02 that allow me to reach the vmware hosts I need to import from. Once imported, I can then change the forwarding on pfsense 02.
So you need to create a new subnet there that it can use and assign >the pfSense01 DCLAN interface an IP from it. Then you need to add >a VIP to pfSense02 in that subnet on it's DCLAN interface so it can >route traffic to pf01.
Alright, we're at the point of this transport then :).
So pf01 will become the main firewall and this is where things get interesting. I have to read and re-read what you say above because it's over my head at the moment.
If I understand where I'm at, I'm just wanting to create a network inside my own network using private IPs. The pf01 would know about a whole class C for example and pf02 would be assigned one IP or a /30 subnet from that class C.
Sorry, brain twisting for me since this is probably fairly simple but typing it all out is much more complex sounding maybe.
-
It just needs to be any currently unused subnet. So, no, it can't be a /30 from inside the existing DCLAN /24 because that will still conflict with the pf01 LAN which is using that same subnet.
So it could be 10.100.0.0/24 for example. With pf01 using 10.100.0.10/24 is it's DCLAN IP and pf02 using 10.100.0.1/24 as a VIP on the DCLAN.Steve
-
Yes, I understood that and was thinking of using the 192.168.x.x or 172.16.x.x network but your suggestion works too.
You said create a network like 10.100.0.0/24 on pf01 DCLAN, use 10.100.0.10/24 on the interface and a VIP on pf02 of 10.100.0.1/24.
Trying to understand this, I'm not really creating a network where there is typically a gateway? In other words, pf01 is not going to become the gateway for pf02, pf03, etc. Instead, the pf01 DCLAN is simply in the same network space as the other pfxx are?
Hope I'm explaining this right.
-
The VIP on pf02 will be the gateway for pf01 on the DCLAN interface.
pf01 mist treat the DCLAN interface like a second WAN. It needs to be able to send replies to forwarded traffic back via pf02.Steve
-
This post is deleted! -
The VIP on pf02 will be the gateway for pf01 on the DCLAN interface.
pf01 mist treat the DCLAN interface like a second WAN. It needs to be >able to send replies to forwarded traffic back via pf02.Oh gee, I'm seeing all this backwards. I see the pf01 as being the gateway for external pfxx once this is set up.
Once I understand this, it's going to be a powerful thing.
-
Yeah, you can't set a network address on an interface. It needs to be an IP address within it.
And, yeah, since pf02 will be the gateway here I would use .1 as the VIP there and some other IP in the subnet on pf01. I suggested .10 above. It needs to be in the subnet though so if you use /30 it can only be .2.Steve
-
This is kinda what I thought we were heading to. I'm not sure about the /30 however since those are network range while pf01 has a single network configured.
I added a VIP alias on pf02 of 10.100.0.1/30 on the LAN interface .
I added 10.100.0.10/24 on pf01 DCLAN.
I pinged across devices and no reply.
To test, I changed pf02 to 10.100.0.1/24 and I can ping between pf servers.
Obviously I'm missing something.This update is showing all of the servers on the old LAN have been moved to the new LAN, only the firewalls remain.
The reason behind this is because some of the IP blocks cannot be moved so the only way to prevent having to make massive changes all at once is to re-route the traffic to where the hosts have been moved.
-
Sometimes, it just takes seeing something to understand it immediately. What should I change in my image that could make this clear?
-
Yeah you need use the same subnet mask everywhere and if it needs to have all 4 pfSense instances in it it needs to be at least /29.
10.100.0.0/30 only contains 10.100.0.1 and 10.100.0.2 as usable IPs. When you set that on pf02 it could not have seen pf01 at 10.100.0.10 because it's outside the /30. If you just use /24 it should be fine.What will probably be a problem though, as I said, is that you can only define one gateway on the DCLAN interface on pf01. That means all traffic coming into it will be tagged reply-to with that gateway and replies will go back to it. Traffic from the other pfSense instances will likely fail because of the asymmetric route.
The only way I can think to get around that would be to use tunnelling between the different pfSense instances so each could be a separate interface/gateway on pf01.I will just say that it still seems absurd to me that the data center can't just reroute those public IPs to the new pf01 WAN.
Steve
-
Yeah you need use the same subnet mask everywhere and if it needs >to have all 4 pfSense instances in it it needs to be at least /29.
At the moment, there is only pf01 and pf02 but yes, once all this is working, there are a couple others that will be used in the same way.
What will probably be a problem though, as I said, is that you can >only define one gateway on the DCLAN interface on pf01.
Right now, the two pfxx are simply in the same network, /24 which is why they can communicate.
That means all traffic coming into it will be tagged reply-to with that >gateway and replies will go back to it. Traffic from the other pfSense >instances will likely fail because of the asymmetric route.
I wondered at one point if I should just convert those pfsense machines into a software router but pf kinda does just that.
Is there a way to assign multiple VIP's to the same interface without a gateway? What about if I used a separate network for each pfsense?
No gateways, just VIP's in the same network.Let's say two remote pf machines talking to pf01.
pf01 would have VIP 10.100.0.10 and 10.101.0.10.
pf02 would have VIP 10.100.0,11 so it could talk with pf01.
pf03 would have VIP 10.101.0.11 so it could talk with pf01.Would that not eliminate the need for a gateway and the potential problem you are concerned about or am I missing some basic routing/firewall knowledge there?
-
In the meantime, I can ping between the two pf01 and pf02 but I've never done a rule where something internal can get to something else internal from one firewall to another.
Meaning, there's no NAT'ing involved here, it's just DCLAN/LAN pf01 to LAN 02. However, there's a default rule that prevents private IP's from coming into the firewall too but I think that's on the WAN side only.
I also notice that pfsense automatically created an anti lockout rule on the DCLAN but not the LAN of pf01.
What kind of crazy rules do I need to set up to allow this to work? So far, nothing I've tried has allowed me to access anything on the pf02 LAN from a windows machine on the LAN side of pf01.
-
Hmm, well I just typed out this big reply about how that won't work but decided to test it to be sure and found something I've never tried before. If you add a gateway you can apply it in the firewall rule and it does change the reply-to tagging which should work for you here. You just have to add gateways for each upstream pfSense instance in pf01 and make sure the rules passing the port forwards are set with that.
You can use separate subnets for each instance but you don't have to. You can just put all the pfSense instances in 10.100.0.0/24 for example.
This is a very unusual config but it does look like it will work.
-
@stephenw10 said in Multi LAN networks to one pfsense:
Hmm, well I just typed out this big reply about how that won't work but decided to test it to be sure and found something I've never tried before. If you add a gateway you can apply it in the firewall rule and it does change the reply-to tagging which should work for you here. You just have to add gateways for each upstream pfSense instance in pf01 and make sure the rules passing the port forwards are set with that.
You can use separate subnets for each instance but you don't have to. You can just put all the pfSense instances in 10.100.0.0/24 for example.
This is a very unusual config but it does look like it will work.
Can you give me an example. This is really getting over my head at this point, probably because I'm just not able to visualize it since I've never done anything like this.
Eventually, once all the servers are moved over to the new network, maybe then it can be simpler. I was wondering if the pf01 could just handle a network and be the gateway for the other pfsense firewalls. Then the other firewalls would simply be assigned an IP in the same network.
For example, pf01 would handle 10.100.0.1/24 and pf02 would have 10.100.0.2, pf03 would have 10.100.0.3.
-
Urgh, nope I spoke too soon. My first instinct was right. You can't apply reply-to tagging like that, the policy routing breaks the port forward.
So back to where we were, traffic coming into the DCLAN interface will only go back via the gateway defined on it.
So to do this you need separate interfaces that can have different gateways.
I think we already discussed VLANs and decided they could not be used?In which case to make this work you will probably need to use tunneling of some sort. So for example a GRE tunnel between each each pfX and pf01 you can forward the traffic across.
Steve
-
There is one free interface on the firewall, another 10gb NIC but there's only one Ethernet cable from the DC to connect to the LAN.
I seem to recall that VLANs cannot exist inside of another VLAN right?
What ever way this could work, it's got to be within the constraints of the DC provided VLAN.
-
You can use QinQ to move VLANs within a VLAN but you would normally expect to have the switches carrying that configured for it. The DC would likely have to be involved.
Otherwise I would suggest using GRE tunnels:
https://docs.netgate.com/pfsense/en/latest/interfaces/gre.htmlSteve
