Multi ISP without failover
-
Maybe this sounds weird, but I have the need for using two different ISPs on my network for some time.
I already have a /29 on the WAN side from ISP1, and a public static range /24 on the LAN side locally. Works great (only webservers).
I will now get a new/additional /29 from an ISP2 with a different assigned /24 that I want to use on LAN side. I need to use both these two different /24 ranges at the same time on LAN-side during transition period for a year or so.
I will move one and one site on the LAN to the same last octet (XX.XX.XX.233 for instance).
I will get two new redundant lines from ISP2 and I do have two free slots on pfSense. One possibility if I can't get a good/easy solution here is of course to fire up one more pfSense physical unit with same config as today and just modify that 2nd one for the new ISP and connect it to the same switch LAN side. Then I can setup each server on network to use either the IP (gw) of pfSense1 or pfSense2 and it will automatically end up on the correct pfSense. But I guess there is a way to do this on the existing pfSense?
Any idea how I can configure this? I would basically have two gateways to the Internet, the old and the new that both has to operate at the same time. I can't configure two gw on the same interface, so I assume there is some kind of routing rules I must do on the pfsense? So that it routes the 2nd ISP /24 over to the ISP2-network based on the /24.
There are no co-operation between the two different ISPs, I can't ask them to change anything (or maybe just one of them).
-
@fireix said in Multi ISP without failover:
One possibility if I can't get a good/easy solution here is of course to fire up one more pfSense physical unit with same config as today and just modify that 2nd one for the new ISP and connect it to the same switch LAN side.
Doing this requires that you configure different default gateways on the respective LAN devices.
But I guess there is a way to do this on the existing pfSense?
Yes. Just configure the second public subnet on an unused interface and state the proper gateway.
In System > Routing > Gateways > Default gateway you can set the default gateway.
This is used by pfSense and by default for outbound traffic.
For the local subnet, which should go out on the other gateway, you have to configure Policy Routing rules. Which means you have to state the respective gateway in the advanced options in rules allowing outbound traffic.
This rule must only be match outbound traffic, since it redirects any matching packet to the stated gateway.
To ensure this, create an alias for local subnets. Best to add all RFC 1918 networks to it. Then you can use this as destination with "invert match" checked to catch only traffic, which is destined to public IP addresses.
E.g. looks like this:
-
@viragomann said in Multi ISP without failover:
Yes. Just configure the second public subnet on an unused interface and state the proper gateway.
Would that be the interface I connect the 2nd ISPs fiber into, so that is named "WAN2" for instance and set interface to an /29 IP and put the ISPs upstream gateway there? Or do you mean to add the local /24 range to a available interface. I was a bit confused :)
I understood the policy routing part, that I define the /24 network of the 2nd ISP so that all traffic goes over that ISPs #2 WAN interface on the way out.
Note: the local subnet in this case would all be public static IP-addresses (both the /29 and the "local" /24 IPs), I only have that. So I have basically a kind of transparent fw where each server on the LAN has static GW IP of the pfSense and I guess the policy routing will route it correctly with that instructions you gave (based on source IP).
-
@fireix said in Multi ISP without failover:
Would that be the interface I connect the 2nd ISPs fiber into, so that is named "WAN2" for instance and set interface to an /29 IP and put the ISPs upstream gateway there?
Yes, correct.
Note: the local subnet in this case would all be public static IP-addresses (both the /29 and the "local" /24 IPs)
I see. But that makes no difference.
I understood the policy routing part, that I define the /24 network of the 2nd ISP so that all traffic goes over that ISPs #2 WAN interface on the way out.
You have two /29 WANs from two ISPs. Let's call them WAN1 and WAN2.
Further you have two /24 LANs (public). These should be connected to different interfaces as well, but that's not really needed. I call them LAN1 and LAN2 subnet.
So you have to add the policy routing on the LAN interface, where you want to route outbound connections to the non-default gateway. If you have both subnets connected to a single interface, just limit the source to the desired subnet. -
@viragomann said in Multi ISP without failover:
Further you have two /24 LANs (public). These should be connected to different interfaces as well, but that's not really needed. I call them LAN1 and LAN2 subnet.
Here I got a bit lost. So I do need to define the /24 at an available interface (port) also, so I will end up with both WAN2 and LAN2. In total min. two ports. This would basically be exactly same setup I already have for WAN1/LAN1. Wouldn't this remove the need for manual policy routing? Since I state the GW on each server inside LAN1/LAN2, the server would contact the correct LAN1 or LAN2 and find its way automatically since both WAN1 and WAN2 would have upstream gw setting to each ISP?
Or is your point that I have to use the available ports, but with using policy routing, I can somehow re-use the LAN1 connection and leave LAN2 disconnected physically (but still defined with an /24 IP from the ISP2 that would be my servers GW2)? How would I in this case make sure that incoming traffic from Internet through the the ISP2 WAN2 connection gets to the IP-addresses that belong on the /24 from ISP2? If that LAN2 port is not connected physically port, only defined interface. I assume the traffic will not be routed from WAN2 to the LAN1 port at all unless some kind of routing for incoming traffic.
-
@fireix said in Multi ISP without failover:
So I do need to define the /24 at an available interface (port) also, so I will end up with both WAN2 and LAN2.
As I mentioned above, that's not required, but recommended.
If both LANs are on the same L2 pfSense is not able to firewall traffic between them.This would basically be exactly same setup I already have for WAN1/LAN1.
Yes, pretty similar.
Wouldn't this remove the need for manual policy routing? Since I state the GW on each server inside LAN1/LAN2, the server would contact the correct LAN1 or LAN2 and find its way automatically since both WAN1 and WAN2 would have upstream gw setting to each ISP?
How should your LAN servers find the correct way without different routing for both LAN subnets?
There is no hard-link between LAN2 and WAN2.
As mentioned above, without policy routing, upstream packets go out to the default gateway. And there is only one, or maybe a gateway group. But the latter isn't, what you try to achieve.I assume the traffic will not be routed from WAN2 to the LAN1 port at all unless some kind of routing for incoming traffic.
pfSense routes incoming traffic just to the destination IP. If the packet is destined to a LAN2 IP it will be routed to it, no matter if both LANs are defined on the same NIC or on different ones, and no matter, on which WAN NIC the packet as entered.
-
@viragomann said in Multi ISP without failover:
pfSense routes incoming traffic just to the destination IP. If the packet is destined to a LAN2 IP it will be routed to it, no matter if both LANs are defined on the same NIC or on different ones, and no matter, on which WAN NIC the packet as entered.
Ah, now I understand. Thanks :)