Using pfsense with multiple WANs
-
Yeah, you can't set a network address on an interface. It needs to be an IP address within it.
And, yeah, since pf02 will be the gateway here I would use .1 as the VIP there and some other IP in the subnet on pf01. I suggested .10 above. It needs to be in the subnet though so if you use /30 it can only be .2.Steve
-
This is kinda what I thought we were heading to. I'm not sure about the /30 however since those are network range while pf01 has a single network configured.
I added a VIP alias on pf02 of 10.100.0.1/30 on the LAN interface .
I added 10.100.0.10/24 on pf01 DCLAN.
I pinged across devices and no reply.
To test, I changed pf02 to 10.100.0.1/24 and I can ping between pf servers.
Obviously I'm missing something.This update is showing all of the servers on the old LAN have been moved to the new LAN, only the firewalls remain.
The reason behind this is because some of the IP blocks cannot be moved so the only way to prevent having to make massive changes all at once is to re-route the traffic to where the hosts have been moved.
-
Sometimes, it just takes seeing something to understand it immediately. What should I change in my image that could make this clear?
-
Yeah you need use the same subnet mask everywhere and if it needs to have all 4 pfSense instances in it it needs to be at least /29.
10.100.0.0/30 only contains 10.100.0.1 and 10.100.0.2 as usable IPs. When you set that on pf02 it could not have seen pf01 at 10.100.0.10 because it's outside the /30. If you just use /24 it should be fine.What will probably be a problem though, as I said, is that you can only define one gateway on the DCLAN interface on pf01. That means all traffic coming into it will be tagged reply-to with that gateway and replies will go back to it. Traffic from the other pfSense instances will likely fail because of the asymmetric route.
The only way I can think to get around that would be to use tunnelling between the different pfSense instances so each could be a separate interface/gateway on pf01.I will just say that it still seems absurd to me that the data center can't just reroute those public IPs to the new pf01 WAN.
Steve
-
Yeah you need use the same subnet mask everywhere and if it needs >to have all 4 pfSense instances in it it needs to be at least /29.
At the moment, there is only pf01 and pf02 but yes, once all this is working, there are a couple others that will be used in the same way.
What will probably be a problem though, as I said, is that you can >only define one gateway on the DCLAN interface on pf01.
Right now, the two pfxx are simply in the same network, /24 which is why they can communicate.
That means all traffic coming into it will be tagged reply-to with that >gateway and replies will go back to it. Traffic from the other pfSense >instances will likely fail because of the asymmetric route.
I wondered at one point if I should just convert those pfsense machines into a software router but pf kinda does just that.
Is there a way to assign multiple VIP's to the same interface without a gateway? What about if I used a separate network for each pfsense?
No gateways, just VIP's in the same network.Let's say two remote pf machines talking to pf01.
pf01 would have VIP 10.100.0.10 and 10.101.0.10.
pf02 would have VIP 10.100.0,11 so it could talk with pf01.
pf03 would have VIP 10.101.0.11 so it could talk with pf01.Would that not eliminate the need for a gateway and the potential problem you are concerned about or am I missing some basic routing/firewall knowledge there?
-
In the meantime, I can ping between the two pf01 and pf02 but I've never done a rule where something internal can get to something else internal from one firewall to another.
Meaning, there's no NAT'ing involved here, it's just DCLAN/LAN pf01 to LAN 02. However, there's a default rule that prevents private IP's from coming into the firewall too but I think that's on the WAN side only.
I also notice that pfsense automatically created an anti lockout rule on the DCLAN but not the LAN of pf01.
What kind of crazy rules do I need to set up to allow this to work? So far, nothing I've tried has allowed me to access anything on the pf02 LAN from a windows machine on the LAN side of pf01.
-
Hmm, well I just typed out this big reply about how that won't work but decided to test it to be sure and found something I've never tried before. If you add a gateway you can apply it in the firewall rule and it does change the reply-to tagging which should work for you here. You just have to add gateways for each upstream pfSense instance in pf01 and make sure the rules passing the port forwards are set with that.
You can use separate subnets for each instance but you don't have to. You can just put all the pfSense instances in 10.100.0.0/24 for example.
This is a very unusual config but it does look like it will work.
-
@stephenw10 said in Multi LAN networks to one pfsense:
Hmm, well I just typed out this big reply about how that won't work but decided to test it to be sure and found something I've never tried before. If you add a gateway you can apply it in the firewall rule and it does change the reply-to tagging which should work for you here. You just have to add gateways for each upstream pfSense instance in pf01 and make sure the rules passing the port forwards are set with that.
You can use separate subnets for each instance but you don't have to. You can just put all the pfSense instances in 10.100.0.0/24 for example.
This is a very unusual config but it does look like it will work.
Can you give me an example. This is really getting over my head at this point, probably because I'm just not able to visualize it since I've never done anything like this.
Eventually, once all the servers are moved over to the new network, maybe then it can be simpler. I was wondering if the pf01 could just handle a network and be the gateway for the other pfsense firewalls. Then the other firewalls would simply be assigned an IP in the same network.
For example, pf01 would handle 10.100.0.1/24 and pf02 would have 10.100.0.2, pf03 would have 10.100.0.3.
-
Urgh, nope I spoke too soon. My first instinct was right. You can't apply reply-to tagging like that, the policy routing breaks the port forward.
So back to where we were, traffic coming into the DCLAN interface will only go back via the gateway defined on it.
So to do this you need separate interfaces that can have different gateways.
I think we already discussed VLANs and decided they could not be used?In which case to make this work you will probably need to use tunneling of some sort. So for example a GRE tunnel between each each pfX and pf01 you can forward the traffic across.
Steve
-
There is one free interface on the firewall, another 10gb NIC but there's only one Ethernet cable from the DC to connect to the LAN.
I seem to recall that VLANs cannot exist inside of another VLAN right?
What ever way this could work, it's got to be within the constraints of the DC provided VLAN.
-
You can use QinQ to move VLANs within a VLAN but you would normally expect to have the switches carrying that configured for it. The DC would likely have to be involved.
Otherwise I would suggest using GRE tunnels:
https://docs.netgate.com/pfsense/en/latest/interfaces/gre.htmlSteve
-
I'm looking at the GRE page.
It mentions routing packets between two locations that are not directly connected, which do not require encryption.Since it's my own LAN, I don't need encryption so that's good but the traffic itself, a lot of it would be encrypted such as https. I assume this just means that it would pass that along as is.
I'll give this a try shortly.
-
There are a lot of articles out there but I can't find one that is simply pf to pf. Most are pf to openwrt and others and most are with IPSEC too. I can't find enough information to know if I need to set this up on the pf01 or pf02.
Based on that page, you create a GRE tunnel on one end.
This is what I've done so far.I have a VIP of 10.100.0.3 on pf02.
I have DCLAN set to 10.100.0.1/24 on pf01.
I created the tunnel as per the images and enabled the interface.
From the CLI of each pf, I cannot ping the other.
pf02: ping 10.100.0.1 or 2, nothing.
pf01: ping 10.100.0.3, nothing.I assume this is because I have to set up rules now.
-
The remote address is the other end of the tunnel outside the tunnel so it that is on pf01 it should be the pf02 VIP, 10.100.0.3.
The local and remote tunnel addresses are the IPs inside the GRE tunnel. So that should be a new subnet for example 10.102.0.1 and 10.102.0.2 using /30 subnet.Create the tunnel on both pfSense instances and assign the generated interfaces. You can then forward or route traffic across it.
Steve
-
pf02 is the old firewall with 10.0.0.1/24 on it.
pf01 is to become the main firewall with DCLAN and 10.0.0.1/24 LAN.Like this on pf01?
I don't need GRE set up on pf02?
-
Yes, you need to setup GRE at both ends so pf02 also.
I would try to use a numbering scheme that is logical because it much easier to9 avoid mistakes that way.
So to start I would probably change the IP pf02 is using in the DCLAN to 10.100.0.2. Use 10.100.0.3 for pf03 etc.
Otherwise, yes that looks good. Youy cna use 10.103.0.0/30 for the tunnel to pf03 etc.
Steve
-
You have no idea how frustrated I am having to keep asking questions and how much I appreciate that you've helped me all this time. I really should have opened a new question for the GRE.
Something about this is either confusing or I've already got it right.
First, we want to create a tunnel between two firewalls using GRE. The GRE tunnel needs to communicate between the two firewalls so IPs pf01 10.100.0.1/24 (24 since I'll have other pf on the same tunnel/network in different subnets) and pf02 10.0.0.1/24.
The tunnel then has its own internal IPs so pf01 10.102.0.1/30 and pf02 10.102.0.2/30 to create one net.
Adding additional pfxx will be simpler since there are no hosts behind that network, we're just using pf as a means to get IP traffic from one network firewall to another.
So if I add pf03, then pf03 can have a LAN 10.3.0.1/24 connecting to DCLAN 10.100.0.1/24. Then a GRE tunnel, pf03 10.103.0.2/30 and pf02 will have 10.103.0.1/30.
IF I have this, then the following which is how I have it all set up should be ok.
IF this is ok, then the only thing I'm missing is one or more rules.
IF this is correct, I just need to start pointing hosts behind pf02 from 10.0.0.x/24 to 10.102.0.1/30.This is a bit simple and mind boggling at the same time because it's all text. Seeing one done would really simplify everything for me and anyone else wanting to understand this. I suspect I'm still screwing something up because from pf02 cli, I cannot ping 10.102.0.1 and from pf01 cli, I cannot ping 10.102.0.2 but it could be because I don't have any rule/s yet. Truly not sure what kind of rule I'll need anyhow.
-
The parent interface for the GRE tunnel on pf02 must be the VIP
10.100.0.2. The LAN interface there as you have it set there is not in the same subnet.On pf03 the LAN (is that's what's on the DCLAN VLAN) should be 10.100.0.3. Or you can add that as a VIP on top of some other subnet like pf02.
The GRE tunnel there will then be between pf03 and pf01.Otherwise that looks right. Can you ping across the tunnels?
Steve
-
Sorry, I only brought up pf3 in terms of your comment about trying to use a numbering scheme. Right now, there are only pf01 and pf02 at the moment to keep things simpler.
On pf02, I have a VIP of 10.100.0.2/30 on the LAN.
Do I need a VIP on pf01 also? I thought the DCLAN. I only have the DCLAN static set which is 10.100.0.1/24.I'm probably not getting this yet but it seems that the DCLAN interface just gets the one static IP.
However, GRE tunnels get their own end to end IPs and then another set for communications inside the tunnel.
Could I just hire you for an hour to get this done over a phone call?
It would save us 100 messages :).I was hoping it could be something useful for anyone else that is wanting to understand how to do this. There's a lot of information out there but it's pretty vague when you've never done it before.
-
I forgot to add...
I can ping from pf01 to 10.100.0.2
I can ping from pf02 to 10.100.0.1
